71 Commits

Author SHA1 Message Date
H1K0 528b7004e3 fix(frontend): open original only when tapping the preview image
deploy / deploy (push) Successful in 1m24s
The preview area was one big link, so a click anywhere in it — including
the black letterbox margins — opened the original. Make the link
pointer-events:none and re-enable pointer-events on the image itself, so
"open original" fires only on the image, not the surrounding margins.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 22:36:16 +03:00
H1K0 41c4667eb4 feat(frontend): enlarge viewer prev/next into full-height tap zones
Paging previously required hitting a small arrow circle pinned near each
edge. Replace them with full-height left/right tap zones (30%, capped at
220px) so a tap anywhere on that side pages. The arrow stays as a chip
near the edge with a hover-only hint gradient; the zones sit above the
image, and the "Replacing…" overlay is lifted above them so an in-flight
content replace still blocks paging.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 22:35:47 +03:00
H1K0 c3ed36f178 fix(frontend): fit PWA icons into maskable safe zone
deploy / deploy (push) Successful in 1m20s
Android's adaptive-icon mask clipped the bamboo artwork and showed a
white backing on install. Fix the icon set so it survives masking.

- maskable icons: artwork fitted inside the 0.40W safe-zone circle,
  opaque full-bleed background (no transparency, so no white fill)
- any icons: rounded-corner tiles with transparent corners
- drop legacy transparent android-icon-* (RealFaviconGenerator) that
  Android picked up as adaptive icons and backed with white
- rename to icon-tile-*/icon-maskable-* so the manifest src changes
  and installed PWAs re-fetch the launcher icon

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 19:50:57 +03:00
H1K0 93e55e8e37 fix(frontend): show content date in local time, not UTC
deploy / deploy (push) Successful in 1m19s
The datetime-local input was populated by slicing the raw UTC ISO
string, so a saved time appeared shifted by the local offset (e.g.
12:00 Moscow rendered as 09:00). Convert the UTC instant to local
wall-clock on load via isoToLocalInput; the save path already
round-trips local -> UTC.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 18:41:17 +03:00
H1K0 3f12b60261 fix(frontend): generate API types before svelte-check
deploy / deploy (push) Successful in 2m13s
schema.ts is gitignored and generated by `generate:types`, which only
`dev` and `build` ran as a prestep — not `check`. The deploy runs
`npm run check` before `npm run build`, so on a clean checkout (where
schema.ts is absent) check fails with "Cannot find module './schema'"
before build ever regenerates it. Give check (and check:watch) the same
generate:types prestep so it stands on its own.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 18:22:48 +03:00
H1K0 6d13fa425a fix(backend): swap db name in URL-style test DSN
deploy / deploy (push) Failing after 54s
The integration suite creates a per-run database and rewrites the admin
DSN to point at it via replaceDSNDatabase. The URL-style branch was a
stub that returned the DSN unchanged, so with a URL DSN every test ran
against the shared `postgres` database instead of its own — no isolation.
The first test to create a given user/object won; the rest collided on
unique constraints, surfacing as cascades of 500s (and ImportFromFolder
seeing accumulated rows).

CI passes TANABATA_TEST_ADMIN_DSN in URL form, so this only broke in the
deploy pipeline, not local key=value runs. Parse the URL and swap its
path to the per-run database.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 18:13:40 +03:00
H1K0 a0a987d561 ci(project): run the test suite before deploying
deploy / deploy (push) Failing after 1m35s
Gate the deploy on tests. After pulling master and before building the
image, run the full Go suite (incl. integration tests, against an ephemeral
Postgres) and the frontend type-check + build. Everything runs inside
throwaway toolchain containers, so the host still needs only docker and a
red build never reaches production. Update DEPLOY.md to reflect the gate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 10:57:16 +03:00
H1K0 3e04da3926 docs(project): clarify API path vs product version in openapi
deploy / deploy (push) Successful in 1m0s
Add a short Versioning note: info.version tracks the 3.0.0 product release
while the /api/v1 path is the API compatibility version, and the two are
independent. Pre-empts the "why v1 under 3.0.0?" confusion.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 10:30:55 +03:00
H1K0 87d3c27b65 style: apply prettier and gofmt
Formatting only: run prettier across the frontend and gofmt over the
backend. Fixes struct-field alignment in domain/pool.go and pre-existing
prettier drift in a few frontend files.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 10:13:56 +03:00
H1K0 36546c46c9 fix(frontend): clear svelte-check a11y and reactivity warnings
- TagBadge: derive color/style so they track a changing tag prop.
- FilterBar: seed tokens from the value prop via untrack (the effect keeps
  it in sync), silencing state_referenced_locally.
- FileCard: give the interactive card role="button".
- admin user toggles: add aria-labels to the switch buttons.

svelte-check is now clean (0 errors, 0 warnings).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 10:13:43 +03:00
H1K0 ac27ea3176 chore(project): license the project under AGPL-3.0
Add the GNU AGPL-3.0 license text, point the openapi.yaml license metadata
at AGPL-3.0-or-later (was the "Proprietary" placeholder), and note the
license in the README.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 10:13:29 +03:00
H1K0 eb3cf72969 feat(frontend): richer pool multi-select and clearer date label
Give the in-pool file selection the same action set as the files grid.
SelectionBar is now driven by props (count + callbacks) instead of reading
the selection store directly, with an optional "Remove from pool" action.
The pool page uses it to offer Edit tags, Add to pool, Mark reviewed,
Remove from pool and Delete (to trash), replacing the previous
add/remove-only bar.

Also rename the content_datetime label from the photo-centric "Date taken"
to "Content date" across the files sort, the file viewer field and the pool
sort options (and align the pool's "created" option with the files grid).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 08:50:21 +03:00
H1K0 4b6db84afc feat: per-pool automatic file sorting
deploy / deploy (push) Successful in 2s
Pools now store a sort setting (sort_key + sort_order). "manual" keeps the
user-arranged order in file_pool.position and allows drag-to-reorder; any
other key (content_datetime, created, original_name) sorts the pool's files
automatically server-side, in which case reordering is rejected. Manual
order is always ascending by position — direction does not apply.

Backend: add the columns (sort_key defaults to 'manual'), generalise the
pool-files keyset cursor to page by the active sort, persist the setting on
create/update, and guard reorder against non-manual pools.

Frontend: a sort dropdown (+ direction toggle) on the pool page that PATCHes
the pool and reloads; drag-to-reorder is disabled unless the pool is manual.

Closes the "automatic sorting in pools" requirement (REQUIREMENTS.md §4.6.5).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 08:36:12 +03:00
H1K0 28f5b5d150 docs(project): sync docs with code and bump to 3.0.0
Prepare the 3.0.0 release:
- Bump the version in openapi.yaml and frontend/package.json to 3.0.0.
- Document the existing GET /health endpoint in openapi.yaml (served at the
  root, outside /api/v1) and refine the auth note.
- Add docs/REQUIREMENTS.md (product requirements, in English) and
  docs/ARCHITECTURE.md (system overview); remove the old Russian
  docs/Описание.md.
- Rewrite GO_PROJECT_STRUCTURE.md and FRONTEND_STRUCTURE.md to match the
  current code (dedup CLI, imagehash, real components/stores, Tailwind v4).
- Fix stale counts and references in CLAUDE.md and link the new docs from
  README.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 08:35:16 +03:00
H1K0 4883820f25 test(backend): encode the JPEG test fixture so it decodes
minimalJPEG() returned a hand-written byte literal that passed MIME
sniffing but failed image.Decode ("missing SOS marker"), so the upload
path left phash NULL and TestDuplicateDetection found no duplicate
clusters. Build the fixture with image/jpeg.Encode instead, so it decodes
through the same path the server uses to hash uploads. Encoding is
deterministic, so two uploads still hash identically.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 08:34:59 +03:00
H1K0 a6a46af12e feat(backend): graceful shutdown on SIGTERM/SIGINT
Run the HTTP server in a goroutine and, on SIGINT/SIGTERM, call
srv.Shutdown so it stops accepting connections and lets in-flight
requests finish before exiting (ErrServerClosed is a clean exit). This
stops uploads/streams being cut when the container is stopped or
recreated on deploy.

The drain deadline is configurable via SHUTDOWN_TIMEOUT (default 15s).
docker-compose.yml feeds the same variable into the app's
stop_grace_period, so Docker won't SIGKILL mid-drain and the two values
can't drift.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 08:12:10 +03:00
H1K0 bf7fa49a16 feat(frontend): support nested objects in the metadata editor
deploy / deploy (push) Successful in 22s
Replace the flat key/value metadata editor with a recursive tree editor.
Each entry is a leaf (scalar edited as text) or a nested object with its own
children; a per-row toggle switches between the two, expanding pasted JSON
into rows and collapsing groups back to JSON text without losing data. Leaf
values still round-trip as JSON where possible, so numbers and booleans keep
their type.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 00:09:11 +03:00
H1K0 76d9c35363 feat(frontend): show and edit file metadata
deploy / deploy (push) Successful in 1m1s
Add a key/value metadata editor to the file viewer (display, add, edit and
remove fields; values round-trip as JSON where possible, otherwise as plain
strings) and a compact side-by-side metadata preview to the duplicate merge
dialog so each side's keys and values are visible while choosing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-23 00:37:41 +03:00
H1K0 71cf79eeaa fix(backend): preserve omitted fields on file update
FileRepo.Update rewrites every editable column, so a PATCH that omitted a
field cleared it. Seed original_name and metadata — which have no always-
present input in the file editor — from the current file so a partial
update leaves them untouched instead of nulling them. The merge path builds
its own complete patch and calls the repo directly, so it is unaffected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-23 00:37:31 +03:00
H1K0 a16accf443 feat(frontend): show keep-to-other distance in the duplicates view
deploy / deploy (push) Successful in 1m1s
Each action row now shows the perceptual distance (Δn) between the kept file
and that file, read from the new per-cluster distances; recomputed live when the
survivor pick changes. A transitively-linked pair with no stored distance shows
a muted Δ—.
2026-06-22 22:42:14 +03:00
H1K0 78b5e86dd6 feat(backend): report stored pairwise distances in the duplicates response
ListVisible already loads each pair's Hamming distance, but clusterPairs threw
it away. Thread it through: Clusters now returns a Cluster carrying the stored
pairwise distances (indexed once per page), and the list endpoint emits them as
{a, b, distance}. Pairs linked only transitively have no stored distance and are
omitted. Lets the UI show how close each file is to the kept one without a
client-side hash compare (phash exceeds JS's safe-integer range).
2026-06-22 22:42:14 +03:00
H1K0 281358ff04 feat(frontend): open cluster files in the full viewer for comparison
deploy / deploy (push) Successful in 18s
The lightbox only showed the preview image, but dedup decisions need the
metadata — date, tags, EXIF — to choose a survivor. Replace it with the same
FileViewer the files page uses, mounted as a full-screen overlay with prev/
next wired across the cluster's files. A review toggle made in the viewer is
mirrored back into the cluster list. Drops the now-superseded PreviewLightbox.
2026-06-22 22:26:08 +03:00
H1K0 d8cccbb9e0 fix(frontend): resolve duplicate actions in place without a full reload
Every action button (Not a dup, Delete, Merge) called reload(), which wiped
the whole cluster list and refetched from offset 0 — re-running the costly
server-side clustering and jumping the scroll to the top on each tap. Give
each cluster a stable local key and edit it in place: drop the resolved file
from its cluster, removing the cluster when fewer than two files remain. The
server view is live, so subsequent pages reconcile. A separate monotonic page
cursor keeps "Load more" from repeating clusters after local removals.
2026-06-22 22:20:39 +03:00
H1K0 d7bfe6b596 feat(frontend): enlarge cluster images in a preview lightbox
deploy / deploy (push) Successful in 36s
The duplicate cluster view only showed 96px thumbnails, too small to tell
near-duplicates apart. Add a zoom button per tile that opens a lightbox
showing the full preview — the same image the single-file viewer uses — and
pages across the cluster with arrows/Esc. The zoom button stops propagation
so it doesn't also reassign the cluster's "keep" selection.
2026-06-22 22:13:38 +03:00
H1K0 e5a731eb29 perf(frontend): lazy-load thumbnails via IntersectionObserver
Thumb and FileCard fetched their auth-gated thumbnail on mount, so a view
that rendered many tiles at once — notably a duplicate cluster, which can
hold hundreds of files — fired thousands of requests immediately and hung
the tab. Gate each fetch behind an IntersectionObserver that loads only when
the tile nears the viewport (200px rootMargin), then stops observing.
2026-06-22 22:10:12 +03:00
H1K0 6fe6e4cd55 chore(project): lower duplicate hash threshold to 4 in example env
The distance histogram over a real ~65k-pair table is flat-to-rising from
distance 1 and nearly doubles at 10 — the 8-10 band is dominated by
coincidental matches, not duplicates. Default the example to 4 (the valley
minimum) to keep the genuine-duplicate signal and cut the pairs to cluster.
2026-06-22 22:09:04 +03:00
H1K0 6834b916cd feat(frontend): replace file content from the viewer
deploy / deploy (push) Successful in 29s
The backend has had PUT /files/:id/content for a while, but nothing in the
UI exposed it. Add a "Replace file content" action to the file viewer's top
bar: it opens a file picker, confirms before overwriting (the original bytes
are replaced irreversibly under the same id; tags/pools/metadata are kept),
then uploads via a new api.uploadPut helper.

The file id is unchanged, so the preview URL stays the same while the bytes
behind it don't — refetch the preview past the browser cache (the server
cache is already invalidated by the replace) and re-mint the content token so
"open original" serves the new content. A spinner overlay covers the preview
during the upload, and the viewer's own shortcuts yield while the confirm is
open or a replace is in flight.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 17:33:22 +03:00
H1K0 745fe38e63 chore(project): drop the docs/reference Flask snapshot
The previous Python/Flask app under docs/reference/ was kept as a visual
design reference while bootstrapping the new frontend. That's well past
done, and 85 files of dead code just add noise to search and exploration.
Remove it (recoverable from history if needed) and update CLAUDE.md: keep
the design tokens, drop the now-dead pointer at the folder.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 17:20:49 +03:00
H1K0 43c5c12fb9 feat(frontend): add selected pool files to another pool
The pool view's selection bar could only remove files from the current
pool. Add an "Add to pool" action beside it that opens the existing file
picker with the selected files (in selection order), so a multi-select can
be copied into another pool in one step. On success the picker closes and
the selection clears.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 17:19:02 +03:00
H1K0 c9b7f0701b feat(frontend): keyboard control for the add-to-pool dialog
The file viewer could only open the pool picker via its top-right button —
there was no `p` shortcut there (only the grid had one), so pressing `p`
on the view page did nothing. Add `p` to open the picker from the viewer,
and give the picker itself full keyboard control: `/` focuses the search
box, arrows move a highlight through the pool list, Enter adds to the
highlighted pool, and Escape clears the search first, then closes.

Both the viewer and the grid now yield the keyboard entirely to the open
picker (the picker owns Escape via its own window handler) so the
clear-then-close behaviour isn't pre-empted by the host's own Escape.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 17:15:12 +03:00
H1K0 b8d08925a2 feat(backend): take video thumbnails from the middle frame
deploy / deploy (push) Successful in 1m1s
Video thumbnails and previews were extracted ~1s in, which lands on
shared intros, title cards or black lead-in frames. Take the frame from
the middle (duration/2) instead — the same frame used for the perceptual
hash — so the thumbnail/preview reflects what dedup compared. Fold the
midpoint logic into a shared extractVideoFrameMiddle helper reused by
both the thumbnail/preview path and VideoFrameMiddle.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 15:14:04 +03:00
H1K0 dc40729646 fix(frontend): shift-select in gesture direction, not grid order
Shift range-select normalized the range with Math.min/Math.max and
always iterated ascending, so the selection's insertion order (which the
Set preserves and which carries through to e.g. pool add order) ignored
the gesture direction. Iterate anchor → target instead via a shared
selectRange helper, so selecting first→last and last→first yield
correspondingly ordered selections.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 14:54:32 +03:00
H1K0 432b2d5b1e feat(backend): audit event for pool file reordering
deploy / deploy (push) Successful in 59s
Reorder was the only pool-file operation that didn't record an audit
entry, unlike AddFiles (file_pool_add) and RemoveFiles
(file_pool_remove). Log file_pool_reorder on success and seed the new
action type.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 14:04:41 +03:00
H1K0 47d9cae15b feat(backend): progress bar for the dedup CLI
deploy / deploy (push) Successful in 53s
Both phases now share a small, dependency-free progress indicator: an in-place
bar on a TTY (e.g. `docker compose run`), and a line every 10% when stdout is
piped (cron/CI) so logs don't fill with carriage returns. Also fixes the pairs
phase, which mislabelled its progress as "hashed" — it now reads "matching".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 13:27:14 +03:00
H1K0 595eb5e06a docs(project): document DUPLICATE_HASH_THRESHOLD in .env.example
deploy / deploy (push) Successful in 1m3s
The dedup pairs rebuild reads this tunable (default 10/64); it was added to the
backend config but never documented for operators. No other new env vars were
introduced by duplicate detection — the dedup compose service reuses the
existing PUID/PGID/FILES_DIR/THUMBS_DIR.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 13:21:23 +03:00
H1K0 19bdd3faa9 build(project): add one-shot dedup compose service
Adds a `dedup` task service under the "tools" profile so it's kept out of
`docker compose up` and run on demand:

  docker compose run --rm dedup            # hashes, then rebuild pairs
  docker compose run --rm dedup -pairs     # only rebuild pairs
  docker compose run --rm dedup -hashes    # only backfill hashes

It reuses the app image, .env, volumes and networks, overriding only the
entrypoint to /app/dedup. Unlike `docker exec` on the live server, this runs in
its own container and is self-documented for cron/CI use.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 13:20:27 +03:00
H1K0 16e68236a0 build(project): ship the dedup CLI in the runtime image
The backend build stage compiled only ./cmd/server, so the dedup maintenance
tool was never available on deploy. Build it alongside the server and copy
/out/dedup to /app/dedup in the runtime image (which already has ffmpeg/ffprobe
for video frames and the /data volume). Run it with `docker exec`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 13:13:53 +03:00
H1K0 dcbe640fae feat(frontend): duplicates view, field-by-field merge dialog, api module
Adds the duplicate-detection UI:

- api/duplicates.ts: getDuplicates / dismissDuplicate / resolveDuplicate, plus
  the cluster and merge-field types.
- /files/duplicates: an offset-paginated list of clusters. Each cluster shows its
  files (auth-loaded thumbnails via a reusable Thumb component); the user clicks a
  file to mark it the survivor, then per other file: Merge, Delete, or "Not a dup"
  (dismiss). The list reloads after each action so it stays consistent with the
  rescan-gated server state.
- DuplicateMergeDialog: a bottom sheet to merge two files field-by-field — each
  scalar from the kept or other file, metadata keep/other/merge, tags & pools
  keep-or-union, with a swap-survivor toggle and an optional trash-the-other box.
- Entry point: a Duplicates action in the files Header next to Trash.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 13:08:46 +03:00
H1K0 96a903aaff docs(project): document duplicate detection endpoints
Adds GET /files/duplicates, POST /files/duplicates/dismiss and POST
/files/duplicates/resolve to the OpenAPI spec, plus the DuplicateCluster,
DuplicateClusterPage and DuplicateResolve (with MergeScalarChoice /
MergeRelationChoice) schemas describing the field-by-field merge contract.

Also fills a pre-existing gap in the File schema: it now documents the `tags`
array (always returned by the API) and marks the always-present fields required,
so generated clients type these as non-optional.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 13:07:37 +03:00
H1K0 6e3e6a4194 feat(backend): dedup CLI (hash backfill + pairs rescan)
cmd/dedup is the offline maintenance tool for duplicate detection. It reuses the
server's config and runs two phases (both by default; -hashes / -pairs to pick):

- hashes: compute the perceptual hash of every live image/video missing one —
  images from their bytes, videos from a middle frame via DiskStorage.
  VideoFrameMiddle. Per-file failures are reported and counted, not fatal.
- pairs: rebuild data.duplicate_pairs from all current hashes (DuplicateService.
  Rescan).

Idempotent and safe to re-run: hashing only touches NULL phashes, the pairs
rebuild is a full replace. This is how video phashes and any backlog get
computed, and how newly uploaded duplicates become visible.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 12:46:40 +03:00
H1K0 9216a8687f feat(backend): duplicate pairs, dismissals, and merge resolution
Adds the duplicate-detection backend on top of perceptual hashing:

- Two tables (edited into the original migrations): data.duplicate_pairs holds
  precomputed near-duplicate candidates (rebuilt wholesale by the rescan), and
  data.duplicate_dismissals is a global "not a duplicate" overlay that survives
  rescans. New audit actions file_merge / duplicate_dismiss.
- DuplicateService:
  - Rescan builds every pair within DUPLICATE_HASH_THRESHOLD via a BK-tree over
    the perceptual hashes and replaces the pairs table. This is the only thing
    that populates pairs, so GET never compares all-vs-all (scales to 110k+).
  - Clusters reads the precomputed pairs (ACL-filtered, non-trashed, non-
    dismissed), groups them into connected components via union-find, and
    paginates whole clusters.
  - Resolve merges a pair field-by-field: each scalar from keep or discard,
    metadata keep/discard/shallow-merge, tags/pools keep or union; then trashes
    the discarded file. Enforces edit ACL on both.
  - Dismiss records a canonical pair (view ACL on both).
- Endpoints under /files: GET /files/duplicates, POST /files/duplicates/dismiss,
  POST /files/duplicates/resolve (registered before /:id to avoid collision).
  Plain delete reuses /files/bulk/delete.
- Repo support: ListMissingPHash, ListAllPHashes, CopyPoolMemberships, plus the
  DuplicatePairRepo (ReplaceAll via COPY, ListVisible) and DismissalRepo.

Unit tests cover the BK-tree pairing, union-find clustering, metadata merge and
field validation; an integration test covers rescan -> list -> merge -> dismiss
(including that a dismissal survives a re-rescan).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 12:42:37 +03:00
H1K0 88849cc16b feat(backend): perceptual hashing for images and video
Adds a 64-bit dHash perceptual hash (internal/imagehash, built on the existing
disintegration/imaging — no new dependency) and starts populating the long-unused
data.files.phash column:

- Upload sets phash inline for images (cheap, from the in-memory bytes).
- Replace recomputes it from new content for images and clears it for anything
  else, so a stale hash never survives a content swap.
- FileRepo.SetPHash sets/clears the hash (used by Replace and, later, the dedup
  backfill).
- DiskStorage.VideoFrameMiddle extracts a frame from the middle of a clip
  (ffprobe duration -> ffmpeg -ss duration/2), avoiding the shared-intro collision
  a fixed early offset causes. It is a concrete method, not part of the storage
  port: only the dedup CLI needs it, keeping ffmpeg off the upload path. Video
  phashes are therefore computed by that CLI, not at upload time.
- DUPLICATE_HASH_THRESHOLD config (default 10/64) for the later pair rescan.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 12:20:52 +03:00
H1K0 58cea88f52 fix(project): drop external dockerfile frontend directive
deploy / deploy (push) Successful in 3s
The `# syntax=docker/dockerfile:1` line made BuildKit fetch its frontend
image from Docker Hub on every build, even when all base images and layers
were already cached. On a host that briefly can't resolve registry-1.docker.io
this is the first and only mandatory network round-trip, so the build fails at
"resolve image config for docker-image://docker.io/docker/dockerfile:1" before
any stage runs.

This Dockerfile uses no frontend-specific syntax (no heredocs, no RUN --mount,
no COPY --link) — only multi-stage, COPY --from/--chown, RUN, ENV, etc., all
handled by the engine's built-in frontend. Dropping the directive removes the
Docker Hub dependency and lets a fully cached build complete offline.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 12:15:23 +03:00
H1K0 b25085bcd7 feat(backend): set application_name on database connections
deploy / deploy (push) Successful in 1m38s
Tag pooled connections with application_name="tanabata-backend" via the
parsed pgxpool config, so the backend's sessions are identifiable in
pg_stat_activity and server logs. An application_name supplied in the
DSN (or PGAPPNAME) still takes precedence.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 10:51:37 +03:00
H1K0 cf6c312e04 fix(frontend): don't treat empty 2xx bodies as request failures
The API client only skipped JSON parsing for 204, so a success with an
empty body (e.g. 201 from POST /pools/:id/files) hit res.json() on an
empty stream, threw, and surfaced as "Failed to add to pool" even though
the add had committed. Read the body as text and parse only when present.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 10:43:23 +03:00
H1K0 da4ce37aff fix(frontend): use opaque background icons for PWA install
The manifest's largest, maskable icon (ms-icon-310x310) was transparent,
so installing the PWA produced a transparent app icon. Generate opaque
192/512 "any" + maskable icons from favicon-bg.png (solid #524B6B
background, maskable variants padded into the inner 80% safe zone) and
point the manifest at them instead of the transparent ms-icon entry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 10:39:41 +03:00
H1K0 70d12615b8 feat(frontend): keyboard navigation for tag and category grids
deploy / deploy (push) Successful in 23s
Mirror the Files grid's roving keyboard focus on the tag and category
lists (and the tags shown on a category page): arrows move a focus ring,
Enter opens the focused item, "/" jumps to search, Escape drops the ring.
Extracts the model into a reusable createRovingGrid controller; vertical
movement is geometric since the pills wrap at variable widths. The
tag/category edit pages gain Escape-to-leave parity with the file viewer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 23:36:36 +03:00
H1K0 97d6daaa13 fix(frontend): stop trimming the file grid while scrolling
The grid windowed to ~4 viewports, dropping off-screen rows as it grew.
That broke large multi-selects: range/drag selection could not span past
trimmed cards, which silently vanished mid-scroll. Accumulate all loaded
rows for the visit instead; the grid is still cleared on sort/filter
change and on leaving the page (reset effect + section cache). Removes
the now-dead trim/anchor-refill fallbacks in loadMore/loadPrev.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 23:27:19 +03:00
H1K0 95db88388b feat(frontend): add MIME filter controls to FilterBar
deploy / deploy (push) Successful in 24s
The DSL already supported m~/m= tokens but the filter UI had no way to
add them. Add Images/Video quick buttons and a free-text MIME input that
append m~<pattern> tokens (LIKE on the type name), plus friendly token
labels in dsl.ts.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 22:52:12 +03:00
H1K0 a864ca4f7b feat(frontend): review-status filter, badge, and review toggles
deploy / deploy (push) Successful in 1m28s
FilterBar gains an Any/Needs review/Reviewed segment (r=1/r=0 token);
FileCard shows a "needs review" dot; FileViewer gets a header toggle that
propagates back to the grid; SelectionBar gains a bulk "Mark reviewed"
action. Adds a --color-success theme token.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 21:17:11 +03:00
H1K0 5bb53d7f9d docs(project): document review status (filter token + bulk/review)
Add File.needs_review, the POST /files/bulk/review path, and the r=1/r=0
filter tokens to the DSL description.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 21:17:02 +03:00
H1K0 48e901cac1 feat(backend): per-file review status with DSL filter and bulk endpoint
Replaces the old "untagged" sentinel tag with a proper per-file workflow
status: needs_review starts true on upload/import and is cleared by an
explicit action (no auto-clear on tagging). Surfaced as a filter token
(r=1 needs review, r=0 done) so it combines with tag/MIME conditions, and
toggled via POST /files/bulk/review (single id or many, edit-ACL enforced,
audit-logged as file_review).

needs_review lives on data.files (column added to the original 003 migration,
partial index in 006, action type seeded in 007).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 21:16:47 +03:00
H1K0 4d11beb296 feat(backend): import server-folder files oldest-first by mtime
os.ReadDir returns entries in name order; sort them by ascending mtime
before importing so each Upload's created_at reflects the files'
chronological order. mtimes are cached once for the sort and reused as the
content_datetime fallback, dropping the redundant per-file Info() call.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 18:16:24 +03:00
H1K0 57192a49f9 fix(backend): apply auto-tag rule to existing files on creation
CreateRule accepted apply_to_existing but ignored it, so enabling the
checkbox while creating a rule never retroactively tagged files already
carrying the when-tag — only activating an existing rule did. Extract the
retroactive expansion into TagRuleRepo.ApplyToExisting (reused by SetActive)
and call it from CreateRule when the rule is active, inside one transaction
so a file is never left half-tagged. Mirrors SetRuleActive semantics,
including following only active downstream rules.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 18:09:31 +03:00
H1K0 9937984a5a feat(backend): reject non-positive token TTLs at config load
Every duration in the config is a token TTL (access, refresh, content). A zero
or negative value mints already-expired tokens — no login, no media playback —
and previously loaded silently. parseDuration now rejects <= 0 with a clear
error, so misconfiguration fails fast at startup instead of mysteriously at
runtime. The AuthService itself stays permissive (it's constructed directly in
tests with arbitrary TTLs); config load is the gate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 17:56:25 +03:00
H1K0 6fba04cd00 docs(project): document the content-token endpoint and CONTENT_TOKEN_TTL
Add POST /files/{file_id}/content-token to the spec, note that the content
GET's access_token parameter also accepts a content token, and document the
CONTENT_TOKEN_TTL knob (default 6h) and its leak/revocation trade-off.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 17:53:30 +03:00
H1K0 35b1b2e6d5 feat(frontend): use a content token for the open-original link
Mint a content token on file load (POST /files/:id/content-token) and put it in
the original-content URL instead of the access token, so opening an original —
especially a long video — in a new tab keeps working past the 15-minute access
token expiry. Falls back to the access token until the content token arrives,
and re-mints when paging to another file.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 17:53:20 +03:00
H1K0 98de298e5b feat(backend): file-scoped content tokens for media URLs
Opening an original by URL (?access_token=) baked in the 15-minute access
token, so a long video opened in a new tab stopped streaming once that token
expired mid-playback: the access token can't be refreshed in an already-opened
tab, and its next Range request 401'd.

Add a content token: a signed, single-file capability (typ=content, fid claim)
with its own longer TTL (CONTENT_TOKEN_TTL, default 6h) and — crucially — no
session id, so it survives refresh rotation and outlives the short access TTL.
POST /files/:id/content-token mints one after the same view-ACL check content
serving does; GET /files/:id/content now runs under content-aware auth that
accepts either a normal access token or a content token scoped to that file.
View permission is still enforced against the token's user, so the token only
changes when a file may be read by URL, never which files. It's a bearer
capability for that one file until expiry, hence the bounded, configurable TTL.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 17:53:10 +03:00
H1K0 b470782e97 docs(project): add README with reverse-proxy and quick-start guide
Root README covering the stack, quick start, and dev commands, plus a Reverse
proxy (nginx) section: client_max_body_size for large uploads, forwarded headers
feeding the rate limiter, and buffering-off for streaming large media.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:52:09 +03:00
H1K0 5a05bb86e1 build(project): publish app on loopback and segment Docker networks
Bind the published port to 127.0.0.1 so the app is reachable only through the
host reverse proxy, not on the LAN/WAN — a 0.0.0.0 publish would also bypass
ufw/firewalld, since Docker's DNAT rules sit ahead of the host firewall.

Split the stack onto two networks with deterministic bridge names: `web`
(dk-tanabata) for the public-facing side, and `backend` (dk-tanabata-bnd,
internal:true) for the private app↔DB tier. The DB sits only on `backend`, which
has no gateway, so it has no route off-host.

Document TRUSTED_PROXIES and the loopback publish in .env.example.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:51:57 +03:00
H1K0 99668ec0d8 feat(backend): trust reverse-proxy X-Forwarded-For for the client IP
The auth rate limiter keys on c.ClientIP(), but the router was built with
gin.New() and never called SetTrustedProxies — so Gin trusted all proxies by
default. Behind a host reverse proxy that meant the limiter either bucketed
every request under the proxy's IP, or (with the port reachable directly) could
be bypassed by a forged X-Forwarded-For.

NewRouter now takes a trusted-proxy list and configures SetTrustedProxies,
returning an error on an invalid list so misconfiguration fails fast at startup.
The list comes from a new TRUSTED_PROXIES config (CSV of CIDRs/IPs), defaulting
to loopback plus the Docker bridge ranges a host proxy reaches the container
through.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:51:44 +03:00
H1K0 88e07f0723 build(project): install vips-tools for image thumbnailing
deploy / deploy (push) Successful in 57s
Add vips-tools (vipsthumbnail) to the runtime image alongside ffmpeg and
exiftool, and document THUMB_MAX_PIXELS as the pure-Go fallback guard.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 01:25:55 +03:00
H1K0 384386a34e feat(backend): generate thumbnails/previews via vipsthumbnail
Use vipsthumbnail as the primary still-image path for both thumbnails
and previews (shared serveGenerated), falling back to the pure-Go
imaging pipeline when vips isn't on PATH. vips shrinks on load (e.g.
JPEG DCT scaling), so a 200+ Mpx photo is resized in a fraction of the
memory and CPU of a full in-process decode and no longer exceeds the
decode cap — the source that previously got only a placeholder now gets
a real thumbnail and preview. The output JPEG is written straight to the
cache (atomic temp→rename), fit within the target box and never upscaled.

The in-process pixel cap now guards only the pure-Go fallback.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 01:25:55 +03:00
H1K0 e694f17be9 docs(project): document THUMB_MAX_PIXELS and THUMB_CONCURRENCY
deploy / deploy (push) Successful in 53s
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 01:07:30 +03:00
H1K0 2d2a42d523 fix(backend): bound thumbnail generation and decode larger images
Thumbnails/previews are generated lazily per request with no concurrency
limit, and the imaging resize already fans out across every core — so
scrolling to a handful of large images spawned that many all-core,
hundreds-of-MB decodes at once and pegged the server. Add a generation
semaphore (THUMB_CONCURRENCY, default = half the CPUs) so only a bounded
number run at a time; queued requests wait and re-check the cache.

Also raise the decode cap from 64 Mpx to a configurable ~300 Mpx default
(THUMB_MAX_PIXELS) so genuinely large photos (e.g. 13000×17000 ≈ 221
Mpx) get a real thumbnail instead of falling back to a placeholder.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 01:07:30 +03:00
H1K0 a371045b41 fix(frontend): keep auth tokens in sync across browser tabs
deploy / deploy (push) Successful in 1m4s
Refresh tokens rotate on every use and each refresh deletes the old
session server-side, so when one tab refreshed, other open tabs were
left holding a dead access token and a rotated-away refresh token —
their next request 401'd and bounced them to the login screen.

Sync the auth store across tabs via the storage event (propagating
logins, refreshes, and logouts), and make refresh race-resilient: if a
refresh fails but a newer token has meanwhile synced in from another
tab, adopt it and retry instead of ending a still-valid session.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 00:13:13 +03:00
H1K0 da867406e2 fix(frontend): let a tag be created/edited without a colour
A native <input type="color"> always holds a value, so the form always
sent the input's default colour and a tag could never be colourless. Add
a "Color" checkbox gating the swatch: off by default on the new-tag form
(so tags are colourless unless opted in) and initialised from the tag on
the edit form, which can now clear a colour. Sends color: null when off.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 23:31:59 +03:00
H1K0 21c7aa31ea fix(backend): store an empty tag colour as NULL
The PATCH "clear colour" path sent an empty string, which violates the
hex CHECK constraint and never falls back to the category colour. Map ''
to NULL via NULLIF in the tag insert/update so a cleared or omitted
colour is stored as NULL.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 23:31:59 +03:00
H1K0 4def59c86d feat(frontend): log a pool view when the pool page opens
Fire POST /pools/{id}/views fire-and-forget after the pool loads, the
same way the file viewer logs file views.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 23:20:46 +03:00
H1K0 d345839634 docs(project): document the pool view endpoint
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 23:20:46 +03:00
H1K0 7d0ea4e388 feat(backend): record pool views to activity.pool_views
Add POST /pools/{id}/views, mirroring the file-view endpoint: it
enforces view ACL and appends a row to activity.pool_views (viewed_at
defaults to statement_timestamp(), so each view is its own history row).
The table existed but nothing wrote to it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 23:20:46 +03:00
174 changed files with 8407 additions and 7144 deletions
+50 -1
View File
@@ -14,7 +14,10 @@
# DATABASE_URL at host.docker.internal (see the Database section below). # DATABASE_URL at host.docker.internal (see the Database section below).
COMPOSE_PROFILES=with-db COMPOSE_PROFILES=with-db
# Host port the app is published on. The container always listens on 42776. # Host port the app is published on, bound to 127.0.0.1 (loopback) — a reverse
# proxy on the host fronts it (see README → Reverse proxy). The container always
# listens on 42776. To expose the app directly without a proxy, drop the
# "127.0.0.1:" prefix on the ports line in docker-compose.yml.
APP_PORT=42776 APP_PORT=42776
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -51,6 +54,30 @@ JWT_SECRET=change-me-to-a-random-32-byte-secret
JWT_ACCESS_TTL=15m JWT_ACCESS_TTL=15m
JWT_REFRESH_TTL=720h JWT_REFRESH_TTL=720h
# How long a content token is valid. It's a single-file capability the client
# puts in a media URL to open/stream an original by link (e.g. a long video in a
# new tab), so playback survives the short access-token expiry and session
# rotation. Longer = fewer interruptions but a wider window in which a leaked URL
# can read that one file; it can't be revoked before expiry. Keep it roughly as
# long as a viewing session lasts.
CONTENT_TOKEN_TTL=6h
# How long a graceful shutdown waits for in-flight requests to finish after the
# app receives SIGTERM/SIGINT (e.g. on `docker compose up --build`, which
# recreates the container). This SAME value is fed to the container's
# `stop_grace_period` in docker-compose.yml, so Docker won't SIGKILL the app
# mid-drain — set it in one place here and both stay in sync. A single upload or
# video stream that outlasts this window is still cut; raise it if you routinely
# move very large files. Accepts Go/Compose durations (s, m, h).
SHUTDOWN_TIMEOUT=15s
# Reverse-proxy hops (comma-separated CIDRs/IPs) whose X-Forwarded-For is trusted,
# so the auth rate limiter sees real client IPs instead of the proxy's. The default
# covers loopback and the Docker bridge ranges a host nginx reaches the container
# through; widen/narrow it to match your proxy. Leave at the default for the
# standard "host nginx → 127.0.0.1" setup.
TRUSTED_PROXIES=127.0.0.1/32,::1/128,172.16.0.0/12
# Initial administrator, created on first startup if it does not yet exist. # Initial administrator, created on first startup if it does not yet exist.
# Changing the password later (via the API) is preserved across restarts. # Changing the password later (via the API) is preserved across restarts.
ADMIN_USERNAME=admin ADMIN_USERNAME=admin
@@ -92,12 +119,34 @@ THUMB_WIDTH=160
THUMB_HEIGHT=160 THUMB_HEIGHT=160
PREVIEW_WIDTH=1920 PREVIEW_WIDTH=1920
PREVIEW_HEIGHT=1080 PREVIEW_HEIGHT=1080
# Pixel cap (width×height) for the pure-Go fallback decoder, used only when
# vipsthumbnail is NOT installed; larger images then get a placeholder. With vips
# present (the default image) thumbnails shrink on load, so this limit — and its
# RAM cost — don't apply. Also bounds a decompression bomb. Default ~300 Mpx.
THUMB_MAX_PIXELS=300000000
# How many thumbnails/previews may be generated at once. Each resize already uses
# every core, so a burst of large images otherwise pegs the CPU and RAM. 0 = auto
# (half the available CPUs).
THUMB_CONCURRENCY=0
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Import # Import
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
IMPORT_PATH=/data/import IMPORT_PATH=/data/import
# ---------------------------------------------------------------------------
# Duplicate detection
# ---------------------------------------------------------------------------
# Maximum perceptual-hash distance (Hamming, out of 64 bits) for two files to be
# treated as duplicate candidates. Lower = stricter (fewer, more confident
# matches); higher = looser (catches more re-encodes/resizes but risks false
# positives). On real libraries the distance histogram climbs steeply in the 810
# band — coincidental "vaguely similar" pairs, not duplicates — so 4 keeps the
# genuine-duplicate signal without that noise (and far fewer pairs to cluster).
# Used only by the dedup tool's pairs rebuild — see the dedup CLI /
# `docker compose run --rm dedup`. Code default is 10.
DUPLICATE_HASH_THRESHOLD=4
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Static SPA # Static SPA
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
+44 -1
View File
@@ -19,7 +19,9 @@ jobs:
# existing clone in /opt/tanabata. See docs/DEPLOY.md for runner setup. # existing clone in /opt/tanabata. See docs/DEPLOY.md for runner setup.
# #
# Only shell steps here (no `uses:` actions), so the host needs git + docker # Only shell steps here (no `uses:` actions), so the host needs git + docker
# and nothing else — no node, no rsync. # and nothing else — no node, no go, no rsync. The test step below runs the
# Go and Node toolchains inside throwaway containers, so nothing has to be
# installed on the host.
runs-on: host runs-on: host
env: env:
@@ -35,6 +37,47 @@ jobs:
git fetch --prune origin git fetch --prune origin
git reset --hard origin/master git reset --hard origin/master
- name: Run tests
working-directory: /opt/tanabata
# Everything runs INSIDE throwaway toolchain containers — the host only
# needs docker (which it already uses for `docker compose`). Nothing (Go,
# Node, Postgres, vips/ffmpeg/exiftool) has to be installed on the host.
# The orchestration below uses only bash builtins + docker. A failure here
# fails the job, so a red build never reaches production. Module/npm caches
# persist in named volumes for speed.
shell: bash
run: |
set -euo pipefail
docker rm -f tfm-ci-db >/dev/null 2>&1 || true
docker network rm tfm-ci-net >/dev/null 2>&1 || true
docker network create tfm-ci-net >/dev/null
trap 'docker rm -f tfm-ci-db >/dev/null 2>&1 || true; docker network rm tfm-ci-net >/dev/null 2>&1 || true' EXIT
docker run -d --name tfm-ci-db --network tfm-ci-net \
-e POSTGRES_PASSWORD=postgres postgres:14-alpine >/dev/null
# Wait for Postgres, using pg_isready inside the DB container (no host tools).
for ((i = 0; i < 30; i++)); do
docker exec tfm-ci-db pg_isready -U postgres >/dev/null 2>&1 && break
sleep 1
done
# Backend: full suite, including the integration tests, against the
# ephemeral Postgres. -buildvcs=false since the .git dir isn't mounted.
docker run --rm --network tfm-ci-net \
-v /opt/tanabata/backend:/src -w /src \
-v tfm-ci-gomod:/go/pkg/mod -v tfm-ci-gocache:/root/.cache/go-build \
-e CGO_ENABLED=0 \
-e TANABATA_TEST_ADMIN_DSN="postgres://postgres:postgres@tfm-ci-db:5432/postgres?sslmode=disable" \
golang:1.26-alpine go test -buildvcs=false -count=1 ./...
# Frontend: type-check + production build (also validates openapi via
# the generate:types prestep).
docker run --rm \
-v /opt/tanabata:/repo -w /repo/frontend \
-v tfm-ci-npm:/root/.npm \
node:22-alpine sh -c "npm ci && npm run check && npm run build"
- name: Build image and start the stack - name: Build image and start the stack
working-directory: /opt/tanabata working-directory: /opt/tanabata
# .env must already exist in DEPLOY_DIR on the host (secrets + DB mode). # .env must already exist in DEPLOY_DIR on the host (secrets + DB mode).
+10 -6
View File
@@ -13,16 +13,18 @@ Monorepo: `backend/` (Go) + `frontend/` (SvelteKit).
## Key documents (read before coding) ## Key documents (read before coding)
- `openapi.yaml` — full REST API specification (36 paths, 58 operations) - `openapi.yaml` — full REST API specification (44 paths, 67 operations)
- `docs/REQUIREMENTS.md` — product requirements (functional + non-functional)
- `docs/ARCHITECTURE.md` — system architecture overview (layers, data flow, decisions)
- `docs/GO_PROJECT_STRUCTURE.md` — backend architecture, layer rules, DI pattern - `docs/GO_PROJECT_STRUCTURE.md` — backend architecture, layer rules, DI pattern
- `docs/FRONTEND_STRUCTURE.md` — frontend architecture, CSS approach, API client - `docs/FRONTEND_STRUCTURE.md` — frontend architecture, CSS approach, API client
- `docs/Описание.md` — product requirements in Russian - `backend/migrations/` — database schema as goose migrations (4 schemas, 19 tables)
- `backend/migrations/001_init.sql` — database schema (4 schemas, 16 tables)
## Design reference ## Design reference
The `docs/reference/` directory contains the previous Python/Flask version. Visual design tokens for the frontend (carried over from the previous
Use its visual design as the basis for the new frontend: Python/Flask version):
- Color palette: #312F45 (bg), #9592B5 (accent), #444455 (tag default), #111118 (elevated) - Color palette: #312F45 (bg), #9592B5 (accent), #444455 (tag default), #111118 (elevated)
- Font: Epilogue (variable weight) - Font: Epilogue (variable weight)
- Dark theme is primary - Dark theme is primary
@@ -32,6 +34,7 @@ Use its visual design as the basis for the new frontend:
- Floating selection bar for multi-select - Floating selection bar for multi-select
## Backend commands ## Backend commands
```bash ```bash
cd backend cd backend
go run ./cmd/server # run dev server go run ./cmd/server # run dev server
@@ -39,6 +42,7 @@ go test ./... # run all tests
``` ```
## Frontend commands ## Frontend commands
```bash ```bash
cd frontend cd frontend
npm run dev # vite dev server npm run dev # vite dev server
@@ -55,4 +59,4 @@ npm run generate:types # regenerate API types from openapi.yaml
- Git: conventional commits with scope — `type(scope): message` - Git: conventional commits with scope — `type(scope): message`
- `(backend)` for Go backend code - `(backend)` for Go backend code
- `(frontend)` for SvelteKit/TypeScript code - `(frontend)` for SvelteKit/TypeScript code
- `(project)` for root-level files (.gitignore, docs/reference, structure) - `(project)` for root-level files (.gitignore, docs, structure)
+15 -10
View File
@@ -1,5 +1,3 @@
# syntax=docker/dockerfile:1
# ============================================================================= # =============================================================================
# Tanabata File Manager — single-image build # Tanabata File Manager — single-image build
# #
@@ -41,23 +39,28 @@ RUN go mod download
COPY backend/ ./ COPY backend/ ./
# CGO is disabled: image processing is pure Go (disintegration/imaging) and # CGO is disabled: the binary shells out to external tools at runtime
# video thumbnails shell out to the ffmpeg binary at runtime, so the resulting # (vipsthumbnail for image thumbnails, ffmpeg for video frames, exiftool for
# binary is fully static and portable across base images. # metadata) and falls back to pure-Go image processing (disintegration/imaging)
# when vips is absent, so it stays fully static and portable across base images.
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/server ./cmd/server RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/server ./cmd/server
# dedup: offline maintenance CLI for duplicate detection (hash backfill + pairs
# rescan). Shipped alongside the server so it can be run with `docker exec`.
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/dedup ./cmd/dedup
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
# Stage 3 — minimal runtime # Stage 3 — minimal runtime
# #
# Alpine (not distroless/scratch) because video thumbnailing invokes ffmpeg and # Alpine (not distroless/scratch) because thumbnailing and metadata extraction
# metadata extraction invokes exiftool as external processes; both must be # invoke external processes (vipsthumbnail, ffmpeg, exiftool) that must be present
# present on the runtime image. # on the runtime image.
# ----------------------------------------------------------------------------- # -----------------------------------------------------------------------------
FROM alpine:3.21 AS runtime FROM alpine:3.21 AS runtime
# ffmpeg: video frame extraction. exiftool: rich image/video/audio metadata. # vips-tools: fast, low-memory image thumbnails (shrink-on-load, so multi-hundred-
# Mpx photos cost little). ffmpeg: video frame extraction. exiftool: rich metadata.
# ca-certificates/tzdata: TLS + time zones. # ca-certificates/tzdata: TLS + time zones.
RUN apk add --no-cache ffmpeg exiftool ca-certificates tzdata RUN apk add --no-cache vips-tools ffmpeg exiftool ca-certificates tzdata
# Run as an unprivileged user. # Run as an unprivileged user.
RUN addgroup -S -g 42776 tanabata && adduser -S -G tanabata -u 42776 tanabata RUN addgroup -S -g 42776 tanabata && adduser -S -G tanabata -u 42776 tanabata
@@ -68,6 +71,8 @@ WORKDIR /app
COPY --from=frontend --chown=tanabata:tanabata /src/frontend/build /app/static COPY --from=frontend --chown=tanabata:tanabata /src/frontend/build /app/static
# The server binary. # The server binary.
COPY --from=backend --chown=tanabata:tanabata /out/server /app/server COPY --from=backend --chown=tanabata:tanabata /out/server /app/server
# The dedup maintenance CLI (run via `docker exec`, not the entrypoint).
COPY --from=backend --chown=tanabata:tanabata /out/dedup /app/dedup
# Data directories (overridable via FILES_PATH/THUMBS_CACHE_PATH/IMPORT_PATH). # Data directories (overridable via FILES_PATH/THUMBS_CACHE_PATH/IMPORT_PATH).
# Created and owned by the tanabata user so a fresh named volume inherits write access. # Created and owned by the tanabata user so a fresh named volume inherits write access.
+661
View File
@@ -0,0 +1,661 @@
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU Affero General Public License is a free, copyleft license for
software and other kinds of works, specifically designed to ensure
cooperation with the community in the case of network server software.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
our General Public Licenses are intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
Developers that use our General Public Licenses protect your rights
with two steps: (1) assert copyright on the software, and (2) offer
you this License which gives you legal permission to copy, distribute
and/or modify the software.
A secondary benefit of defending all users' freedom is that
improvements made in alternate versions of the program, if they
receive widespread use, become available for other developers to
incorporate. Many developers of free software are heartened and
encouraged by the resulting cooperation. However, in the case of
software used on network servers, this result may fail to come about.
The GNU General Public License permits making a modified version and
letting the public access it on a server without ever releasing its
source code to the public.
The GNU Affero General Public License is designed specifically to
ensure that, in such cases, the modified source code becomes available
to the community. It requires the operator of a network server to
provide the source code of the modified version running there to the
users of that server. Therefore, public use of a modified version, on
a publicly accessible server, gives the public access to the source
code of the modified version.
An older license, called the Affero General Public License and
published by Affero, was designed to accomplish similar goals. This is
a different license, not a version of the Affero GPL, but Affero has
released a new version of the Affero GPL which permits relicensing under
this license.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU Affero General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Remote Network Interaction; Use with the GNU General Public License.
Notwithstanding any other provision of this License, if you modify the
Program, your modified version must prominently offer all users
interacting with it remotely through a computer network (if your version
supports such interaction) an opportunity to receive the Corresponding
Source of your version by providing access to the Corresponding Source
from a network server at no charge, through some standard or customary
means of facilitating copying of software. This Corresponding Source
shall include the Corresponding Source for any work covered by version 3
of the GNU General Public License that is incorporated pursuant to the
following paragraph.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the work with which it is combined will remain governed by version
3 of the GNU General Public License.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU Affero General Public License from time to time. Such new versions
will be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU Affero General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU Affero General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU Affero General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If your software can interact with users remotely through a computer
network, you should also make sure that it provides a way for users to
get its source. For example, if your program is a web application, its
interface could display a "Source" link that leads users to an archive
of the code. There are many ways you could offer source, and different
solutions will be better for different programs; see section 13 for the
specific requirements.
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU AGPL, see
<https://www.gnu.org/licenses/>.
+105
View File
@@ -0,0 +1,105 @@
# Tanabata File Manager
A multi-user, tag-based web file manager for images and video. Go + Gin backend
(Clean Architecture, pgx, goose migrations), SvelteKit SPA frontend, PostgreSQL,
JWT auth — shipped as a single Docker image that serves both the API and the
built SPA on one port.
## Documentation
- [`openapi.yaml`](openapi.yaml) — full REST API specification
- [`docs/REQUIREMENTS.md`](docs/REQUIREMENTS.md) — product requirements
- [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) — system architecture overview
- [`docs/DEPLOY.md`](docs/DEPLOY.md) — production deploy (Gitea Actions → host)
- [`docs/GO_PROJECT_STRUCTURE.md`](docs/GO_PROJECT_STRUCTURE.md) — backend architecture
- [`docs/FRONTEND_STRUCTURE.md`](docs/FRONTEND_STRUCTURE.md) — frontend architecture
- [`.env.example`](.env.example) — every configuration variable, documented
## Quick start
```bash
cp .env.example .env # then edit the secrets (JWT_SECRET, ADMIN_PASSWORD, …)
docker compose up -d --build
```
By default this runs the app plus a bundled PostgreSQL container
(`COMPOSE_PROFILES=with-db`). To point at a Postgres already on the host, set
`COMPOSE_PROFILES=` empty and aim `DATABASE_URL` at `host.docker.internal`. See
[`.env.example`](.env.example) for the full matrix.
The app is published on **127.0.0.1** only and expects a reverse proxy in front
(see below). The default port is **42776** — the sum of the Unicode code points
of 七夕.
## Reverse proxy (nginx)
The container publishes its port on loopback (`127.0.0.1:${APP_PORT}:42776` in
[`docker-compose.yml`](docker-compose.yml)), so a reverse proxy on the host
terminates TLS and forwards to it. Three settings matter for this app:
1. **`client_max_body_size`** — uploads go up to `MAX_UPLOAD_BYTES` (500 MiB by
default). nginx caps request bodies at **1 MiB** out of the box, so without
this every large upload fails with `413`.
2. **Forwarded headers** — the app trusts `X-Forwarded-For` only from the hops in
`TRUSTED_PROXIES` (default: loopback + Docker bridge ranges) and keys its
login/refresh rate limiter on the resulting client IP. If the proxy doesn't
send the header, every request looks like it comes from the proxy and shares
one rate-limit bucket.
3. **Streaming for big media** — turning request/response buffering off lets
large uploads stream straight to the app and lets video range-seeks work
without nginx spooling whole files to disk first.
```nginx
server {
listen 443 ssl;
server_name tanabata.example.com;
# ssl_certificate / ssl_certificate_key ... (e.g. from certbot)
# Match MAX_UPLOAD_BYTES (500 MiB default); nginx defaults to 1m → 413.
client_max_body_size 512m;
location / {
proxy_pass http://127.0.0.1:42776; # APP_PORT
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Stream large uploads/downloads instead of buffering to disk; keeps
# video range-seek responsive. Scope these to file/preview locations
# instead if you'd rather keep buffering for small JSON responses.
proxy_request_buffering off;
proxy_buffering off;
proxy_read_timeout 300s;
proxy_send_timeout 300s;
}
}
```
If you run the app **without** a proxy and want it reachable on the LAN, drop the
`127.0.0.1:` prefix from the `ports` line in
[`docker-compose.yml`](docker-compose.yml) and adjust `TRUSTED_PROXIES`
accordingly.
## Development
```bash
# Backend
cd backend
go run ./cmd/server # dev server
go test ./... # all tests
# Frontend
cd frontend
npm run dev # Vite dev server
npm run build # production build
npm run generate:types # regenerate API types from openapi.yaml
```
## License
Licensed under the GNU Affero General Public License v3.0 or later
(AGPL-3.0-or-later). See [`LICENSE`](LICENSE) for the full text.
+217
View File
@@ -0,0 +1,217 @@
// Command dedup is the offline maintenance tool for duplicate detection. It runs
// in two phases:
//
// hashes — compute the perceptual hash of every live image/video that has none
// yet (images from their bytes, videos from a middle frame via ffmpeg).
// pairs — rebuild data.duplicate_pairs from all current hashes.
//
// Both phases run by default; pass -hashes or -pairs to run only one. It reuses
// the server's configuration (DATABASE_URL, FILES_PATH, THUMBS_CACHE_PATH, …) and
// is safe to re-run: hashing only touches files whose phash is NULL, and the
// pairs rebuild is a full replace.
//
// go run ./cmd/dedup # hashes, then pairs
// go run ./cmd/dedup -pairs # only rebuild pairs
package main
import (
"context"
"flag"
"fmt"
"io"
"os"
"strings"
"github.com/google/uuid"
"tanabata/backend/internal/config"
"tanabata/backend/internal/db/postgres"
"tanabata/backend/internal/imagehash"
"tanabata/backend/internal/service"
"tanabata/backend/internal/storage"
)
func main() {
hashesOnly := flag.Bool("hashes", false, "only (re)compute missing perceptual hashes")
pairsOnly := flag.Bool("pairs", false, "only rebuild the duplicate pairs table")
flag.Parse()
// No flag, or both, means run everything.
doHashes := *hashesOnly || !*pairsOnly
doPairs := *pairsOnly || !*hashesOnly
cfg, err := config.Load()
if err != nil {
fatal("load config", err)
}
ctx := context.Background()
pool, err := postgres.NewPool(ctx, cfg.DatabaseURL)
if err != nil {
fatal("connect to database", err)
}
defer pool.Close()
diskStorage, err := storage.NewDiskStorage(
cfg.FilesPath, cfg.ThumbsCachePath,
cfg.ThumbWidth, cfg.ThumbHeight,
cfg.PreviewWidth, cfg.PreviewHeight,
cfg.ThumbMaxPixels, cfg.ThumbConcurrency,
)
if err != nil {
fatal("init storage", err)
}
fileRepo := postgres.NewFileRepo(pool)
pairRepo := postgres.NewDuplicatePairRepo(pool)
dismissalRepo := postgres.NewDismissalRepo(pool)
aclRepo := postgres.NewACLRepo(pool)
auditRepo := postgres.NewAuditRepo(pool)
tagRepo := postgres.NewTagRepo(pool)
categoryRepo := postgres.NewCategoryRepo(pool)
poolRepo := postgres.NewPoolRepo(pool)
transactor := postgres.NewTransactor(pool)
aclSvc := service.NewACLService(aclRepo, fileRepo, tagRepo, categoryRepo, poolRepo, transactor)
auditSvc := service.NewAuditService(auditRepo)
dupSvc := service.NewDuplicateService(
fileRepo, pairRepo, dismissalRepo, aclSvc, auditSvc, transactor, cfg.DuplicateHashThreshold,
)
if doHashes {
if err := backfillHashes(ctx, fileRepo, diskStorage); err != nil {
fatal("backfill hashes", err)
}
}
if doPairs {
fmt.Printf("rebuilding duplicate pairs (threshold %d)...\n", cfg.DuplicateHashThreshold)
// total is only known once Rescan has loaded the hashes, so create the bar
// lazily on the first progress callback.
var prog *progress
if err := dupSvc.Rescan(ctx, func(done, total int) {
if prog == nil {
prog = newProgress("matching", total)
}
prog.set(done)
}); err != nil {
fatal("rescan pairs", err)
}
if prog != nil {
prog.finish()
}
fmt.Println(" done")
}
}
// backfillHashes computes and stores a perceptual hash for every live image/video
// that lacks one. Failures on individual files are counted and reported, not
// fatal, so one unreadable file doesn't abort the whole run.
func backfillHashes(ctx context.Context, files *postgres.FileRepo, store *storage.DiskStorage) error {
pending, err := files.ListMissingPHash(ctx)
if err != nil {
return err
}
total := len(pending)
fmt.Printf("hashing %d files without a perceptual hash...\n", total)
var hashed, skipped, failed int
prog := newProgress("hashing", total)
for i, f := range pending {
ph, err := hashOne(ctx, store, f.ID, f.MIMEType)
switch {
case err != nil:
failed++
fmt.Fprintf(os.Stderr, "\n %s (%s): %v\n", f.ID, f.MIMEType, err)
case ph == nil:
skipped++ // not decodable; leave phash NULL
default:
if err := files.SetPHash(ctx, f.ID, ph); err != nil {
return fmt.Errorf("set phash for %s: %w", f.ID, err)
}
hashed++
}
prog.set(i + 1)
}
prog.finish()
fmt.Printf(" hashed %d, skipped %d, failed %d\n", hashed, skipped, failed)
return nil
}
// hashOne returns the perceptual hash for one file, or nil when it isn't hashable
// (e.g. an image that won't decode). Images are hashed from their bytes; videos
// from a middle frame.
func hashOne(ctx context.Context, store *storage.DiskStorage, id uuid.UUID, mime string) (*int64, error) {
switch {
case strings.HasPrefix(mime, "image/"):
rc, err := store.Read(ctx, id)
if err != nil {
return nil, err
}
defer rc.Close()
data, err := io.ReadAll(rc)
if err != nil {
return nil, err
}
if h, ok := imagehash.FromBytes(data); ok {
return &h, nil
}
return nil, nil
case strings.HasPrefix(mime, "video/"):
img, err := store.VideoFrameMiddle(ctx, id)
if err != nil {
return nil, err
}
h := imagehash.FromImage(img)
return &h, nil
default:
return nil, nil
}
}
// progress renders a dependency-free progress indicator. On a TTY it draws an
// in-place bar; otherwise (pipe, cron, CI) it prints a line every 10% so logs
// stay readable instead of filling with carriage returns.
type progress struct {
label string
tty bool
total int
lastDec int // last 10%-decile printed in non-TTY mode
}
func newProgress(label string, total int) *progress {
fi, _ := os.Stdout.Stat()
tty := fi != nil && fi.Mode()&os.ModeCharDevice != 0
return &progress{label: label, tty: tty, total: total, lastDec: -1}
}
func (p *progress) set(done int) {
if p.total <= 0 {
return
}
pct := done * 100 / p.total
if p.tty {
const w = 30
filled := done * w / p.total
fmt.Printf("\r %s [%s%s] %3d%% (%d/%d)",
p.label,
strings.Repeat("#", filled), strings.Repeat("-", w-filled),
pct, done, p.total)
return
}
if dec := pct / 10; dec != p.lastDec {
p.lastDec = dec
fmt.Printf(" %s %d%% (%d/%d)\n", p.label, pct, done, p.total)
}
}
// finish ends the in-place bar with a newline (TTY only).
func (p *progress) finish() {
if p.tty && p.total > 0 {
fmt.Println()
}
}
func fatal(what string, err error) {
fmt.Fprintf(os.Stderr, "dedup: %s: %v\n", what, err)
os.Exit(1)
}
+46 -4
View File
@@ -2,9 +2,12 @@ package main
import ( import (
"context" "context"
"errors"
"log/slog" "log/slog"
"net/http" "net/http"
"os" "os"
"os/signal"
"syscall"
"time" "time"
"github.com/jackc/pgx/v5/stdlib" "github.com/jackc/pgx/v5/stdlib"
@@ -52,6 +55,7 @@ func main() {
cfg.ThumbsCachePath, cfg.ThumbsCachePath,
cfg.ThumbWidth, cfg.ThumbHeight, cfg.ThumbWidth, cfg.ThumbHeight,
cfg.PreviewWidth, cfg.PreviewHeight, cfg.PreviewWidth, cfg.PreviewHeight,
cfg.ThumbMaxPixels, cfg.ThumbConcurrency,
) )
if err != nil { if err != nil {
slog.Error("failed to initialise storage", "err", err) slog.Error("failed to initialise storage", "err", err)
@@ -69,6 +73,8 @@ func main() {
tagRuleRepo := postgres.NewTagRuleRepo(pool) tagRuleRepo := postgres.NewTagRuleRepo(pool)
categoryRepo := postgres.NewCategoryRepo(pool) categoryRepo := postgres.NewCategoryRepo(pool)
poolRepo := postgres.NewPoolRepo(pool) poolRepo := postgres.NewPoolRepo(pool)
duplicatePairRepo := postgres.NewDuplicatePairRepo(pool)
dismissalRepo := postgres.NewDismissalRepo(pool)
transactor := postgres.NewTransactor(pool) transactor := postgres.NewTransactor(pool)
// Services // Services
@@ -78,12 +84,16 @@ func main() {
cfg.JWTSecret, cfg.JWTSecret,
cfg.JWTAccessTTL, cfg.JWTAccessTTL,
cfg.JWTRefreshTTL, cfg.JWTRefreshTTL,
cfg.ContentTokenTTL,
) )
aclSvc := service.NewACLService(aclRepo, fileRepo, tagRepo, categoryRepo, poolRepo, transactor) aclSvc := service.NewACLService(aclRepo, fileRepo, tagRepo, categoryRepo, poolRepo, transactor)
auditSvc := service.NewAuditService(auditRepo) auditSvc := service.NewAuditService(auditRepo)
tagSvc := service.NewTagService(tagRepo, tagRuleRepo, aclSvc, auditSvc, transactor) tagSvc := service.NewTagService(tagRepo, tagRuleRepo, aclSvc, auditSvc, transactor)
categorySvc := service.NewCategoryService(categoryRepo, tagRepo, aclSvc, auditSvc) categorySvc := service.NewCategoryService(categoryRepo, tagRepo, aclSvc, auditSvc)
poolSvc := service.NewPoolService(poolRepo, aclSvc, auditSvc) poolSvc := service.NewPoolService(poolRepo, aclSvc, auditSvc)
duplicateSvc := service.NewDuplicateService(
fileRepo, duplicatePairRepo, dismissalRepo, aclSvc, auditSvc, transactor, cfg.DuplicateHashThreshold,
)
fileSvc := service.NewFileService( fileSvc := service.NewFileService(
fileRepo, fileRepo,
mimeRepo, mimeRepo,
@@ -105,7 +115,8 @@ func main() {
// Handlers // Handlers
authMiddleware := handler.NewAuthMiddleware(authSvc) authMiddleware := handler.NewAuthMiddleware(authSvc)
authHandler := handler.NewAuthHandler(authSvc) authHandler := handler.NewAuthHandler(authSvc)
fileHandler := handler.NewFileHandler(fileSvc, tagSvc, cfg.MaxUploadBytes) fileHandler := handler.NewFileHandler(fileSvc, tagSvc, authSvc, cfg.MaxUploadBytes)
duplicateHandler := handler.NewDuplicateHandler(duplicateSvc)
tagHandler := handler.NewTagHandler(tagSvc, fileSvc) tagHandler := handler.NewTagHandler(tagSvc, fileSvc)
categoryHandler := handler.NewCategoryHandler(categorySvc) categoryHandler := handler.NewCategoryHandler(categorySvc)
poolHandler := handler.NewPoolHandler(poolSvc) poolHandler := handler.NewPoolHandler(poolSvc)
@@ -113,12 +124,17 @@ func main() {
aclHandler := handler.NewACLHandler(aclSvc) aclHandler := handler.NewACLHandler(aclSvc)
auditHandler := handler.NewAuditHandler(auditSvc) auditHandler := handler.NewAuditHandler(auditSvc)
r := handler.NewRouter( r, err := handler.NewRouter(
authMiddleware, authHandler, authMiddleware, authHandler,
fileHandler, tagHandler, categoryHandler, poolHandler, fileHandler, duplicateHandler, tagHandler, categoryHandler, poolHandler,
userHandler, aclHandler, auditHandler, userHandler, aclHandler, auditHandler,
cfg.StaticDir, cfg.StaticDir,
cfg.TrustedProxies,
) )
if err != nil {
slog.Error("building router", "err", err)
os.Exit(1)
}
// ReadHeaderTimeout bounds slow-header (Slowloris) attacks; body read/write // ReadHeaderTimeout bounds slow-header (Slowloris) attacks; body read/write
// are left unbounded so large file uploads and downloads can stream. // are left unbounded so large file uploads and downloads can stream.
@@ -129,9 +145,35 @@ func main() {
IdleTimeout: 120 * time.Second, IdleTimeout: 120 * time.Second,
} }
// Trigger a graceful shutdown on SIGINT/SIGTERM (the latter is what Docker
// sends when the container is stopped or recreated on deploy).
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
go func() {
slog.Info("starting server", "addr", cfg.ListenAddr) slog.Info("starting server", "addr", cfg.ListenAddr)
if err := srv.ListenAndServe(); err != nil { // ListenAndServe returns ErrServerClosed after a graceful Shutdown; that
// is the expected exit, not a failure.
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
slog.Error("server error", "err", err) slog.Error("server error", "err", err)
os.Exit(1) os.Exit(1)
} }
}()
<-ctx.Done()
// Restore default signal handling so a second Ctrl+C / SIGTERM force-quits
// instead of waiting on the drain.
stop()
slog.Info("shutting down", "timeout", cfg.ShutdownTimeout)
// Stop accepting new connections and let in-flight requests finish, up to the
// timeout. Docker's stop grace period reads the same SHUTDOWN_TIMEOUT, so it
// won't SIGKILL before this returns.
shutdownCtx, cancel := context.WithTimeout(context.Background(), cfg.ShutdownTimeout)
defer cancel()
if err := srv.Shutdown(shutdownCtx); err != nil {
slog.Error("graceful shutdown failed", "err", err)
os.Exit(1)
}
slog.Info("shutdown complete")
} }
+66
View File
@@ -5,6 +5,7 @@ import (
"fmt" "fmt"
"os" "os"
"strconv" "strconv"
"strings"
"time" "time"
"github.com/joho/godotenv" "github.com/joho/godotenv"
@@ -17,6 +18,26 @@ type Config struct {
JWTSecret string JWTSecret string
JWTAccessTTL time.Duration JWTAccessTTL time.Duration
JWTRefreshTTL time.Duration JWTRefreshTTL time.Duration
// ContentTokenTTL is how long a content token stays valid. The token is a
// single-file capability used to open or stream an original by URL (e.g. a
// long video in a new tab); it is deliberately longer-lived than the access
// token and independent of the session, so playback survives access-token
// expiry and refresh rotation. Keep it only as long as a viewing session
// plausibly lasts — it is a bearer credential for that one file until expiry.
ContentTokenTTL time.Duration
// ShutdownTimeout bounds how long a graceful shutdown waits for in-flight
// requests to finish after a SIGINT/SIGTERM before the process exits. Keep it
// in step with the container's stop grace period — docker-compose.yml reads
// the same SHUTDOWN_TIMEOUT for `stop_grace_period`, so Docker doesn't SIGKILL
// mid-drain. A long upload/stream can still be cut if it outlasts this window.
ShutdownTimeout time.Duration
// TrustedProxies lists the reverse-proxy hops (CIDRs or IPs) whose
// X-Forwarded-For header is trusted. The auth rate limiter keys on the
// client IP, so this must match the proxy in front of the app — otherwise
// every request appears to come from the proxy (one shared bucket) or a
// direct caller could forge the header. Default covers loopback and the
// Docker bridge ranges a host reverse proxy reaches the container through.
TrustedProxies []string
// Initial admin bootstrap (applied on startup if the user does not exist) // Initial admin bootstrap (applied on startup if the user does not exist)
AdminUsername string AdminUsername string
@@ -35,10 +56,25 @@ type Config struct {
ThumbHeight int ThumbHeight int
PreviewWidth int PreviewWidth int
PreviewHeight int PreviewHeight int
// ThumbMaxPixels caps the pixel count of a source image decoded in-process by
// the pure-Go fallback (a decompression-bomb guard and memory bound); larger
// images then get a placeholder. It does not apply when vipsthumbnail is
// installed, which shrinks on load regardless of source size.
ThumbMaxPixels int
// ThumbConcurrency bounds how many thumbnails/previews are generated at once,
// so a burst of large images can't saturate every core or exhaust RAM. 0 =
// auto (half the available CPUs).
ThumbConcurrency int
// Import // Import
ImportPath string ImportPath string
// DuplicateHashThreshold is the maximum Hamming distance (out of 64) between
// two perceptual hashes for the files to be treated as duplicate candidates.
// Lower = stricter (fewer, more confident matches); higher = looser. Used only
// by the dedup rescan that (re)builds data.duplicate_pairs.
DuplicateHashThreshold int
// Static SPA. When set, the server serves the built frontend (and falls // Static SPA. When set, the server serves the built frontend (and falls
// back to index.html for client routes) on the same port as the API. Empty // back to index.html for client routes) on the same port as the API. Empty
// in local development, where the Vite dev server serves the UI separately. // in local development, where the Vite dev server serves the UI separately.
@@ -68,6 +104,10 @@ func Load() (*Config, error) {
return def return def
} }
// parseDuration parses a duration env var. Every duration in this config is a
// token TTL, which must be strictly positive — a zero/negative TTL would mint
// already-expired tokens (no login, no media playback) — so reject those here
// rather than fail mysteriously at runtime.
parseDuration := func(key, def string) time.Duration { parseDuration := func(key, def string) time.Duration {
raw := defaultStr(key, def) raw := defaultStr(key, def)
d, err := time.ParseDuration(raw) d, err := time.ParseDuration(raw)
@@ -75,6 +115,10 @@ func Load() (*Config, error) {
errs = append(errs, fmt.Errorf("%s: invalid duration %q: %w", key, raw, err)) errs = append(errs, fmt.Errorf("%s: invalid duration %q: %w", key, raw, err))
return 0 return 0
} }
if d <= 0 {
errs = append(errs, fmt.Errorf("%s must be positive, got %q", key, raw))
return 0
}
return d return d
} }
@@ -91,6 +135,18 @@ func Load() (*Config, error) {
return n return n
} }
parseCSV := func(key, def string) []string {
raw := defaultStr(key, def)
parts := strings.Split(raw, ",")
out := make([]string, 0, len(parts))
for _, p := range parts {
if p = strings.TrimSpace(p); p != "" {
out = append(out, p)
}
}
return out
}
parseInt64 := func(key string, def int64) int64 { parseInt64 := func(key string, def int64) int64 {
raw := os.Getenv(key) raw := os.Getenv(key)
if raw == "" { if raw == "" {
@@ -110,6 +166,12 @@ func Load() (*Config, error) {
JWTAccessTTL: parseDuration("JWT_ACCESS_TTL", "15m"), JWTAccessTTL: parseDuration("JWT_ACCESS_TTL", "15m"),
JWTRefreshTTL: parseDuration("JWT_REFRESH_TTL", "720h"), JWTRefreshTTL: parseDuration("JWT_REFRESH_TTL", "720h"),
ContentTokenTTL: parseDuration("CONTENT_TOKEN_TTL", "6h"),
ShutdownTimeout: parseDuration("SHUTDOWN_TIMEOUT", "15s"),
TrustedProxies: parseCSV("TRUSTED_PROXIES", "127.0.0.1/32,::1/128,172.16.0.0/12"),
AdminUsername: defaultStr("ADMIN_USERNAME", "admin"), AdminUsername: defaultStr("ADMIN_USERNAME", "admin"),
AdminPassword: requireStr("ADMIN_PASSWORD"), AdminPassword: requireStr("ADMIN_PASSWORD"),
@@ -123,9 +185,13 @@ func Load() (*Config, error) {
ThumbHeight: parseInt("THUMB_HEIGHT", 160), ThumbHeight: parseInt("THUMB_HEIGHT", 160),
PreviewWidth: parseInt("PREVIEW_WIDTH", 1920), PreviewWidth: parseInt("PREVIEW_WIDTH", 1920),
PreviewHeight: parseInt("PREVIEW_HEIGHT", 1080), PreviewHeight: parseInt("PREVIEW_HEIGHT", 1080),
ThumbMaxPixels: parseInt("THUMB_MAX_PIXELS", 300_000_000), // ~300 Mpx (e.g. 13000×17000)
ThumbConcurrency: parseInt("THUMB_CONCURRENCY", 0), // 0 = auto
ImportPath: requireStr("IMPORT_PATH"), ImportPath: requireStr("IMPORT_PATH"),
DuplicateHashThreshold: parseInt("DUPLICATE_HASH_THRESHOLD", 10),
StaticDir: defaultStr("STATIC_DIR", ""), StaticDir: defaultStr("STATIC_DIR", ""),
} }
+57
View File
@@ -0,0 +1,57 @@
package config
import (
"strings"
"testing"
)
// setValidEnv sets every required variable to a valid dummy value, so a test can
// then override one var to exercise a single validation path.
func setValidEnv(t *testing.T) {
t.Helper()
t.Setenv("JWT_SECRET", "test-secret")
t.Setenv("ADMIN_PASSWORD", "test-password")
t.Setenv("DATABASE_URL", "postgres://u:p@localhost:5432/db?sslmode=disable")
t.Setenv("FILES_PATH", "/tmp/files")
t.Setenv("THUMBS_CACHE_PATH", "/tmp/thumbs")
t.Setenv("IMPORT_PATH", "/tmp/import")
// Pin the TTLs to valid values so an ambient env var can't perturb the case
// under test; individual tests override the one they exercise.
t.Setenv("JWT_ACCESS_TTL", "15m")
t.Setenv("JWT_REFRESH_TTL", "720h")
t.Setenv("CONTENT_TOKEN_TTL", "6h")
}
func TestLoadValid(t *testing.T) {
setValidEnv(t)
cfg, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
if cfg.JWTAccessTTL <= 0 || cfg.JWTRefreshTTL <= 0 || cfg.ContentTokenTTL <= 0 {
t.Fatalf("TTLs should be positive: access=%v refresh=%v content=%v",
cfg.JWTAccessTTL, cfg.JWTRefreshTTL, cfg.ContentTokenTTL)
}
}
func TestLoadRejectsNonPositiveTTL(t *testing.T) {
cases := []struct{ key, val string }{
{"JWT_ACCESS_TTL", "0"},
{"JWT_REFRESH_TTL", "-1h"},
{"CONTENT_TOKEN_TTL", "0s"},
}
for _, tc := range cases {
t.Run(tc.key, func(t *testing.T) {
setValidEnv(t)
t.Setenv(tc.key, tc.val)
_, err := Load()
if err == nil {
t.Fatalf("expected error for %s=%q", tc.key, tc.val)
}
if !strings.Contains(err.Error(), tc.key) || !strings.Contains(err.Error(), "must be positive") {
t.Fatalf("error should name %s and mention positivity, got: %v", tc.key, err)
}
})
}
}
@@ -0,0 +1,145 @@
package postgres
import (
"bytes"
"context"
"fmt"
"github.com/google/uuid"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
"tanabata/backend/internal/domain"
"tanabata/backend/internal/port"
)
// ---------------------------------------------------------------------------
// DuplicatePairRepo
// ---------------------------------------------------------------------------
// DuplicatePairRepo implements port.DuplicatePairRepo using PostgreSQL.
type DuplicatePairRepo struct {
pool *pgxpool.Pool
}
// NewDuplicatePairRepo creates a DuplicatePairRepo backed by pool.
func NewDuplicatePairRepo(pool *pgxpool.Pool) *DuplicatePairRepo {
return &DuplicatePairRepo{pool: pool}
}
var _ port.DuplicatePairRepo = (*DuplicatePairRepo)(nil)
// ReplaceAll atomically replaces the entire pairs table with the given set.
// The rescan recomputes pairs from scratch, so a full DELETE + COPY is both
// correct and the simplest way to drop pairs that no longer qualify.
func (r *DuplicatePairRepo) ReplaceAll(ctx context.Context, pairs []domain.DuplicatePair) error {
tx, err := r.pool.Begin(ctx)
if err != nil {
return fmt.Errorf("DuplicatePairRepo.ReplaceAll begin: %w", err)
}
defer tx.Rollback(ctx) //nolint:errcheck // no-op after a successful commit
if _, err := tx.Exec(ctx, `DELETE FROM data.duplicate_pairs`); err != nil {
return fmt.Errorf("DuplicatePairRepo.ReplaceAll delete: %w", err)
}
if len(pairs) > 0 {
rows := make([][]any, len(pairs))
for i, p := range pairs {
rows[i] = []any{p.FileA, p.FileB, int16(p.Distance)}
}
if _, err := tx.CopyFrom(ctx,
pgx.Identifier{"data", "duplicate_pairs"},
[]string{"file_a", "file_b", "distance"},
pgx.CopyFromRows(rows),
); err != nil {
return fmt.Errorf("DuplicatePairRepo.ReplaceAll copy: %w", err)
}
}
if err := tx.Commit(ctx); err != nil {
return fmt.Errorf("DuplicatePairRepo.ReplaceAll commit: %w", err)
}
return nil
}
type pairRow struct {
FileA uuid.UUID `db:"file_a"`
FileB uuid.UUID `db:"file_b"`
Distance int16 `db:"distance"`
}
// ListVisible returns every stored pair where both files are live (not trashed),
// the pair is not dismissed, and — for non-admins — both files are visible to the
// viewer under the private-by-default model. This is the input to clustering.
func (r *DuplicatePairRepo) ListVisible(ctx context.Context, viewerID int16, isAdmin bool) ([]domain.DuplicatePair, error) {
args := make([]any, 0, 4)
n := 1
aclWhere := ""
if !isAdmin {
var ca, cb string
ca, n, args = aclVisibilityCond("fa", objTypeFile, viewerID, n, args)
cb, n, args = aclVisibilityCond("fb", objTypeFile, viewerID, n, args)
aclWhere = "AND " + ca + " AND " + cb
}
sqlStr := fmt.Sprintf(`
SELECT p.file_a, p.file_b, p.distance
FROM data.duplicate_pairs p
JOIN data.files fa ON fa.id = p.file_a AND fa.is_deleted = false
JOIN data.files fb ON fb.id = p.file_b AND fb.is_deleted = false
WHERE NOT EXISTS (
SELECT 1 FROM data.duplicate_dismissals d
WHERE d.file_a = p.file_a AND d.file_b = p.file_b
)
%s
ORDER BY p.file_a, p.file_b`, aclWhere)
rows, err := r.pool.Query(ctx, sqlStr, args...)
if err != nil {
return nil, fmt.Errorf("DuplicatePairRepo.ListVisible: %w", err)
}
collected, err := pgx.CollectRows(rows, pgx.RowToStructByName[pairRow])
if err != nil {
return nil, fmt.Errorf("DuplicatePairRepo.ListVisible scan: %w", err)
}
out := make([]domain.DuplicatePair, len(collected))
for i, row := range collected {
out[i] = domain.DuplicatePair{FileA: row.FileA, FileB: row.FileB, Distance: int(row.Distance)}
}
return out, nil
}
// ---------------------------------------------------------------------------
// DismissalRepo
// ---------------------------------------------------------------------------
// DismissalRepo implements port.DismissalRepo using PostgreSQL.
type DismissalRepo struct {
pool *pgxpool.Pool
}
// NewDismissalRepo creates a DismissalRepo backed by pool.
func NewDismissalRepo(pool *pgxpool.Pool) *DismissalRepo {
return &DismissalRepo{pool: pool}
}
var _ port.DismissalRepo = (*DismissalRepo)(nil)
// Add records a pair as "not a duplicate". The two ids are stored in canonical
// (file_a < file_b) order to match the table's CHECK and avoid (a,b)/(b,a)
// duplicates; a repeated dismissal is a no-op.
func (r *DismissalRepo) Add(ctx context.Context, a, b uuid.UUID, userID int16) error {
if bytes.Compare(a[:], b[:]) > 0 {
a, b = b, a
}
const sqlStr = `
INSERT INTO data.duplicate_dismissals (file_a, file_b, dismissed_by)
VALUES ($1, $2, $3)
ON CONFLICT (file_a, file_b) DO NOTHING`
q := connOrTx(ctx, r.pool)
if _, err := q.Exec(ctx, sqlStr, a, b, userID); err != nil {
return fmt.Errorf("DismissalRepo.Add: %w", err)
}
return nil
}
+120 -6
View File
@@ -36,6 +36,7 @@ type fileRow struct {
CreatorName string `db:"creator_name"` CreatorName string `db:"creator_name"`
IsPublic bool `db:"is_public"` IsPublic bool `db:"is_public"`
IsDeleted bool `db:"is_deleted"` IsDeleted bool `db:"is_deleted"`
NeedsReview bool `db:"needs_review"`
} }
// fileTagRow is used for both single-file and batch tag loading. // fileTagRow is used for both single-file and batch tag loading.
@@ -81,6 +82,7 @@ func toFile(r fileRow) domain.File {
CreatorName: r.CreatorName, CreatorName: r.CreatorName,
IsPublic: r.IsPublic, IsPublic: r.IsPublic,
IsDeleted: r.IsDeleted, IsDeleted: r.IsDeleted,
NeedsReview: r.NeedsReview,
CreatedAt: domain.UUIDCreatedAt(r.ID), CreatedAt: domain.UUIDCreatedAt(r.ID),
} }
} }
@@ -293,7 +295,7 @@ const fileSelectCTE = `
mt.name AS mime_type, mt.extension AS mime_extension, mt.name AS mime_type, mt.extension AS mime_extension,
r.content_datetime, r.notes, r.metadata, r.exif, r.phash, r.content_datetime, r.notes, r.metadata, r.exif, r.phash,
r.creator_id, u.name AS creator_name, r.creator_id, u.name AS creator_name,
r.is_public, r.is_deleted r.is_public, r.is_deleted, r.needs_review
FROM r FROM r
JOIN core.mime_types mt ON mt.id = r.mime_id JOIN core.mime_types mt ON mt.id = r.mime_id
JOIN core.users u ON u.id = r.creator_id` JOIN core.users u ON u.id = r.creator_id`
@@ -316,7 +318,8 @@ func (r *FileRepo) Create(ctx context.Context, f *domain.File) (*domain.File, er
$4, $5, $6, $7, $8, $9, $10 $4, $5, $6, $7, $8, $9, $10
) )
RETURNING id, original_name, mime_id, content_datetime, notes, RETURNING id, original_name, mime_id, content_datetime, notes,
metadata, exif, phash, creator_id, is_public, is_deleted metadata, exif, phash, creator_id, is_public, is_deleted,
needs_review
)` + fileSelectCTE )` + fileSelectCTE
q := connOrTx(ctx, r.pool) q := connOrTx(ctx, r.pool)
@@ -346,7 +349,7 @@ func (r *FileRepo) GetByID(ctx context.Context, id uuid.UUID) (*domain.File, err
mt.name AS mime_type, mt.extension AS mime_extension, mt.name AS mime_type, mt.extension AS mime_extension,
f.content_datetime, f.notes, f.metadata, f.exif, f.phash, f.content_datetime, f.notes, f.metadata, f.exif, f.phash,
f.creator_id, u.name AS creator_name, f.creator_id, u.name AS creator_name,
f.is_public, f.is_deleted f.is_public, f.is_deleted, f.needs_review
FROM data.files f FROM data.files f
JOIN core.mime_types mt ON mt.id = f.mime_id JOIN core.mime_types mt ON mt.id = f.mime_id
JOIN core.users u ON u.id = f.creator_id JOIN core.users u ON u.id = f.creator_id
@@ -389,7 +392,8 @@ func (r *FileRepo) Update(ctx context.Context, id uuid.UUID, f *domain.File) (*d
is_public = $6 is_public = $6
WHERE id = $1 WHERE id = $1
RETURNING id, original_name, mime_id, content_datetime, notes, RETURNING id, original_name, mime_id, content_datetime, notes,
metadata, exif, phash, creator_id, is_public, is_deleted metadata, exif, phash, creator_id, is_public, is_deleted,
needs_review
)` + fileSelectCTE )` + fileSelectCTE
q := connOrTx(ctx, r.pool) q := connOrTx(ctx, r.pool)
@@ -416,6 +420,115 @@ func (r *FileRepo) Update(ctx context.Context, id uuid.UUID, f *domain.File) (*d
return &updated, nil return &updated, nil
} }
// SetNeedsReview sets the review status on the given files in one statement.
// Trashed files are left untouched. No-op for an empty id list.
func (r *FileRepo) SetNeedsReview(ctx context.Context, ids []uuid.UUID, value bool) error {
if len(ids) == 0 {
return nil
}
const sqlStr = `UPDATE data.files SET needs_review = $2 WHERE id = ANY($1) AND is_deleted = false`
q := connOrTx(ctx, r.pool)
if _, err := q.Exec(ctx, sqlStr, ids, value); err != nil {
return fmt.Errorf("FileRepo.SetNeedsReview: %w", err)
}
return nil
}
// SetPHash sets (or clears, when phash is nil) the perceptual hash of a file.
// Used by the dedup backfill and on content replacement; phash is non-critical,
// recomputable metadata, so callers may treat failures as best-effort.
func (r *FileRepo) SetPHash(ctx context.Context, id uuid.UUID, phash *int64) error {
const sqlStr = `UPDATE data.files SET phash = $2 WHERE id = $1`
q := connOrTx(ctx, r.pool)
if _, err := q.Exec(ctx, sqlStr, id, phash); err != nil {
return fmt.Errorf("FileRepo.SetPHash: %w", err)
}
return nil
}
// ---------------------------------------------------------------------------
// Perceptual-hash / duplicate support
// ---------------------------------------------------------------------------
// ListMissingPHash returns live image/video files that have no perceptual hash
// yet — the work list for the dedup backfill. Tags are not loaded (the backfill
// only needs the id and MIME type to choose image vs video hashing).
func (r *FileRepo) ListMissingPHash(ctx context.Context) ([]domain.File, error) {
const sqlStr = `
SELECT f.id, f.original_name,
mt.name AS mime_type, mt.extension AS mime_extension,
f.content_datetime, f.notes, f.metadata, f.exif, f.phash,
f.creator_id, u.name AS creator_name,
f.is_public, f.is_deleted, f.needs_review
FROM data.files f
JOIN core.mime_types mt ON mt.id = f.mime_id
JOIN core.users u ON u.id = f.creator_id
WHERE f.phash IS NULL AND f.is_deleted = false
AND (mt.name LIKE 'image/%' OR mt.name LIKE 'video/%')
ORDER BY f.id`
q := connOrTx(ctx, r.pool)
rows, err := q.Query(ctx, sqlStr)
if err != nil {
return nil, fmt.Errorf("FileRepo.ListMissingPHash: %w", err)
}
collected, err := pgx.CollectRows(rows, pgx.RowToStructByName[fileRow])
if err != nil {
return nil, fmt.Errorf("FileRepo.ListMissingPHash scan: %w", err)
}
files := make([]domain.File, len(collected))
for i, row := range collected {
files[i] = toFile(row)
}
return files, nil
}
// phashRow is the minimal projection used to build duplicate clusters.
type phashRow struct {
ID uuid.UUID `db:"id"`
PHash int64 `db:"phash"`
}
// ListAllPHashes returns the id and perceptual hash of every live, hashed file.
// It is the global input to the dedup rescan, so it deliberately ignores ACL —
// the rescan builds the shared pairs table; visibility is enforced on read.
func (r *FileRepo) ListAllPHashes(ctx context.Context) ([]domain.PHashEntry, error) {
const sqlStr = `SELECT id, phash FROM data.files WHERE is_deleted = false AND phash IS NOT NULL`
q := connOrTx(ctx, r.pool)
rows, err := q.Query(ctx, sqlStr)
if err != nil {
return nil, fmt.Errorf("FileRepo.ListAllPHashes: %w", err)
}
collected, err := pgx.CollectRows(rows, pgx.RowToStructByName[phashRow])
if err != nil {
return nil, fmt.Errorf("FileRepo.ListAllPHashes scan: %w", err)
}
out := make([]domain.PHashEntry, len(collected))
for i, row := range collected {
out[i] = domain.PHashEntry{ID: row.ID, PHash: row.PHash}
}
return out, nil
}
// CopyPoolMemberships adds targetID to every pool sourceID belongs to (copying
// the source's position), skipping pools the target is already in. Used by the
// duplicate merge to preserve the discarded file's pool memberships on the
// survivor. The merge is authorised at the file level, so pool ACL is not
// re-checked here.
func (r *FileRepo) CopyPoolMemberships(ctx context.Context, targetID, sourceID uuid.UUID) error {
const sqlStr = `
INSERT INTO data.file_pool (file_id, pool_id, position)
SELECT $1, fp.pool_id, fp.position
FROM data.file_pool fp
WHERE fp.file_id = $2
ON CONFLICT (file_id, pool_id) DO NOTHING`
q := connOrTx(ctx, r.pool)
if _, err := q.Exec(ctx, sqlStr, targetID, sourceID); err != nil {
return fmt.Errorf("FileRepo.CopyPoolMemberships: %w", err)
}
return nil
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// SoftDelete / Restore / DeletePermanent // SoftDelete / Restore / DeletePermanent
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -444,7 +557,8 @@ func (r *FileRepo) Restore(ctx context.Context, id uuid.UUID) (*domain.File, err
SET is_deleted = false SET is_deleted = false
WHERE id = $1 AND is_deleted = true WHERE id = $1 AND is_deleted = true
RETURNING id, original_name, mime_id, content_datetime, notes, RETURNING id, original_name, mime_id, content_datetime, notes,
metadata, exif, phash, creator_id, is_public, is_deleted metadata, exif, phash, creator_id, is_public, is_deleted,
needs_review
)` + fileSelectCTE )` + fileSelectCTE
q := connOrTx(ctx, r.pool) q := connOrTx(ctx, r.pool)
@@ -638,7 +752,7 @@ func (r *FileRepo) List(ctx context.Context, params domain.FileListParams) (*dom
mt.name AS mime_type, mt.extension AS mime_extension, mt.name AS mime_type, mt.extension AS mime_extension,
f.content_datetime, f.notes, f.metadata, f.exif, f.phash, f.content_datetime, f.notes, f.metadata, f.exif, f.phash,
f.creator_id, u.name AS creator_name, f.creator_id, u.name AS creator_name,
f.is_public, f.is_deleted f.is_public, f.is_deleted, f.needs_review
FROM data.files f FROM data.files f
JOIN core.mime_types mt ON mt.id = f.mime_id JOIN core.mime_types mt ON mt.id = f.mime_id
JOIN core.users u ON u.id = f.creator_id JOIN core.users u ON u.id = f.creator_id
+14 -1
View File
@@ -23,6 +23,7 @@ const (
ftkTag // t=<uuid> ftkTag // t=<uuid>
ftkMimeExact // m=<int> ftkMimeExact // m=<int>
ftkMimeLike // m~<pattern> ftkMimeLike // m~<pattern>
ftkReview // r=<0|1>
) )
type filterToken struct { type filterToken struct {
@@ -31,6 +32,7 @@ type filterToken struct {
untagged bool // ftkTag with zero UUID → "file has no tags" untagged bool // ftkTag with zero UUID → "file has no tags"
mimeID int16 // ftkMimeExact mimeID int16 // ftkMimeExact
pattern string // ftkMimeLike pattern string // ftkMimeLike
review bool // ftkReview → needs_review value
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -80,6 +82,8 @@ func (l *leafNode) toSQL(n int, args []any) (string, int, []any) {
case ftkMimeLike: case ftkMimeLike:
// mt alias comes from the JOIN in the main file query (always present). // mt alias comes from the JOIN in the main file query (always present).
return fmt.Sprintf("mt.name LIKE $%d", n), n + 1, append(args, l.tok.pattern) return fmt.Sprintf("mt.name LIKE $%d", n), n + 1, append(args, l.tok.pattern)
case ftkReview:
return fmt.Sprintf("f.needs_review = $%d", n), n + 1, append(args, l.tok.review)
} }
panic("filterNode.toSQL: unknown leaf kind") panic("filterNode.toSQL: unknown leaf kind")
} }
@@ -130,6 +134,15 @@ func lexFilter(dsl string) ([]filterToken, error) {
case strings.HasPrefix(p, "m~"): case strings.HasPrefix(p, "m~"):
// The pattern value is passed as a query parameter, so no SQL injection risk. // The pattern value is passed as a query parameter, so no SQL injection risk.
tokens = append(tokens, filterToken{kind: ftkMimeLike, pattern: p[2:]}) tokens = append(tokens, filterToken{kind: ftkMimeLike, pattern: p[2:]})
case strings.HasPrefix(p, "r="):
switch p[2:] {
case "1":
tokens = append(tokens, filterToken{kind: ftkReview, review: true})
case "0":
tokens = append(tokens, filterToken{kind: ftkReview, review: false})
default:
return nil, fmt.Errorf("filter: invalid review flag %q (want r=0 or r=1)", p[2:])
}
default: default:
return nil, fmt.Errorf("filter: unknown token %q", p) return nil, fmt.Errorf("filter: unknown token %q", p)
} }
@@ -241,7 +254,7 @@ func (p *filterParser) parseAtom() (filterNode, error) {
return expr, nil return expr, nil
} }
switch t.kind { switch t.kind {
case ftkTag, ftkMimeExact, ftkMimeLike: case ftkTag, ftkMimeExact, ftkMimeLike, ftkReview:
p.next() p.next()
return &leafNode{t}, nil return &leafNode{t}, nil
default: default:
@@ -6,6 +6,50 @@ import (
"github.com/google/uuid" "github.com/google/uuid"
) )
func TestParseFilterReview(t *testing.T) {
t.Run("r=1 needs review", func(t *testing.T) {
sql, n, args, err := ParseFilter("{r=1}", 1)
if err != nil {
t.Fatalf("ParseFilter: %v", err)
}
if sql != "f.needs_review = $1" {
t.Fatalf("sql = %q", sql)
}
if n != 2 || len(args) != 1 || args[0] != true {
t.Fatalf("n=%d args=%v", n, args)
}
})
t.Run("r=0 reviewed", func(t *testing.T) {
sql, _, args, err := ParseFilter("{r=0}", 1)
if err != nil {
t.Fatalf("ParseFilter: %v", err)
}
if sql != "f.needs_review = $1" || len(args) != 1 || args[0] != false {
t.Fatalf("sql=%q args=%v", sql, args)
}
})
t.Run("combined with mime", func(t *testing.T) {
sql, n, args, err := ParseFilter("{r=1,&,m~image/%}", 1)
if err != nil {
t.Fatalf("ParseFilter: %v", err)
}
if sql != "(f.needs_review = $1 AND mt.name LIKE $2)" {
t.Fatalf("sql = %q", sql)
}
if n != 3 || len(args) != 2 || args[0] != true || args[1] != "image/%" {
t.Fatalf("n=%d args=%v", n, args)
}
})
t.Run("invalid flag rejected", func(t *testing.T) {
if _, _, _, err := ParseFilter("{r=2}", 1); err == nil {
t.Fatal("expected error for r=2")
}
})
}
func TestFilterTagUses(t *testing.T) { func TestFilterTagUses(t *testing.T) {
a := uuid.MustParse("11111111-1111-1111-1111-111111111111") a := uuid.MustParse("11111111-1111-1111-1111-111111111111")
b := uuid.MustParse("22222222-2222-2222-2222-222222222222") b := uuid.MustParse("22222222-2222-2222-2222-222222222222")
+104 -14
View File
@@ -30,6 +30,8 @@ type poolRow struct {
CreatorID int16 `db:"creator_id"` CreatorID int16 `db:"creator_id"`
CreatorName string `db:"creator_name"` CreatorName string `db:"creator_name"`
IsPublic bool `db:"is_public"` IsPublic bool `db:"is_public"`
SortKey string `db:"sort_key"`
SortOrder string `db:"sort_order"`
FileCount int `db:"file_count"` FileCount int `db:"file_count"`
} }
@@ -68,6 +70,8 @@ func toPool(r poolRow) domain.Pool {
CreatorID: r.CreatorID, CreatorID: r.CreatorID,
CreatorName: r.CreatorName, CreatorName: r.CreatorName,
IsPublic: r.IsPublic, IsPublic: r.IsPublic,
SortKey: r.SortKey,
SortOrder: r.SortOrder,
FileCount: r.FileCount, FileCount: r.FileCount,
CreatedAt: domain.UUIDCreatedAt(r.ID), CreatedAt: domain.UUIDCreatedAt(r.ID),
} }
@@ -103,8 +107,14 @@ func toPoolFile(r poolFileRow) domain.PoolFile {
// Cursor // Cursor
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// poolFileCursor is the keyset paging cursor for pool files. Which fields are
// populated depends on the pool's sort key: Pos for manual (position), Val for
// content_datetime (RFC3339Nano) / original_name (the coalesced name); the
// "created" sort orders by file id alone, so only FileID matters. FileID is
// always the final tiebreak.
type poolFileCursor struct { type poolFileCursor struct {
Position int `json:"p"` Pos int `json:"p,omitempty"`
Val string `json:"v,omitempty"`
FileID string `json:"id"` FileID string `json:"id"`
} }
@@ -135,6 +145,7 @@ const poolCountSubquery = `(SELECT pool_id, COUNT(*) AS cnt FROM data.file_pool
const poolSelectFrom = ` const poolSelectFrom = `
SELECT p.id, p.name, p.notes, p.metadata, SELECT p.id, p.name, p.notes, p.metadata,
p.creator_id, u.name AS creator_name, p.is_public, p.creator_id, u.name AS creator_name, p.is_public,
p.sort_key, p.sort_order,
COALESCE(fc.cnt, 0) AS file_count COALESCE(fc.cnt, 0) AS file_count
FROM data.pools p FROM data.pools p
JOIN core.users u ON u.id = p.creator_id JOIN core.users u ON u.id = p.creator_id
@@ -147,6 +158,29 @@ func poolSortColumn(s string) string {
return "p.id" // "created" return "p.id" // "created"
} }
// poolFileSort maps a pool's stored sort settings to the SQL column expression,
// the ORDER BY direction, and the keyset comparison operator used for paging.
// The column is chosen so it is never NULL (original_name is coalesced), which
// keeps the keyset comparison total.
func poolFileSort(sortKey, sortOrder string) (col, dir, cmp string) {
dir, cmp = "ASC", ">"
if strings.EqualFold(sortOrder, domain.SortOrderDesc) {
dir, cmp = "DESC", "<"
}
switch sortKey {
case domain.PoolSortContentDatetime:
col = "f.content_datetime"
case domain.PoolSortOriginalName:
col = "COALESCE(f.original_name, '')"
case domain.PoolSortCreated:
col = "f.id"
default: // manual — the user-arranged sequence; direction does not apply
col = "fp.position"
dir, cmp = "ASC", ">"
}
return col, dir, cmp
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// PoolRepo // PoolRepo
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -207,6 +241,7 @@ func (r *PoolRepo) List(ctx context.Context, params port.OffsetParams) (*domain.
query := fmt.Sprintf(` query := fmt.Sprintf(`
SELECT p.id, p.name, p.notes, p.metadata, SELECT p.id, p.name, p.notes, p.metadata,
p.creator_id, u.name AS creator_name, p.is_public, p.creator_id, u.name AS creator_name, p.is_public,
p.sort_key, p.sort_order,
COALESCE(fc.cnt, 0) AS file_count, COALESCE(fc.cnt, 0) AS file_count,
COUNT(*) OVER() AS total COUNT(*) OVER() AS total
FROM data.pools p FROM data.pools p
@@ -266,6 +301,16 @@ WHERE p.id = $1`
return &p, nil return &p, nil
} }
// RecordView appends a row to activity.pool_views. viewed_at defaults to
// statement_timestamp(), so each call records a distinct view in the history.
func (r *PoolRepo) RecordView(ctx context.Context, poolID uuid.UUID, userID int16) error {
const query = `INSERT INTO activity.pool_views (pool_id, user_id) VALUES ($1, $2)`
if _, err := connOrTx(ctx, r.pool).Exec(ctx, query, poolID, userID); err != nil {
return fmt.Errorf("PoolRepo.RecordView: %w", err)
}
return nil
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Create // Create
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -279,6 +324,7 @@ WITH ins AS (
) )
SELECT ins.id, ins.name, ins.notes, ins.metadata, SELECT ins.id, ins.name, ins.notes, ins.metadata,
ins.creator_id, u.name AS creator_name, ins.is_public, ins.creator_id, u.name AS creator_name, ins.is_public,
ins.sort_key, ins.sort_order,
0 AS file_count 0 AS file_count
FROM ins FROM ins
JOIN core.users u ON u.id = ins.creator_id` JOIN core.users u ON u.id = ins.creator_id`
@@ -315,12 +361,15 @@ WITH upd AS (
name = $2, name = $2,
notes = $3, notes = $3,
metadata = COALESCE($4, metadata), metadata = COALESCE($4, metadata),
is_public = $5 is_public = $5,
sort_key = $6,
sort_order = $7
WHERE id = $1 WHERE id = $1
RETURNING * RETURNING *
) )
SELECT upd.id, upd.name, upd.notes, upd.metadata, SELECT upd.id, upd.name, upd.notes, upd.metadata,
upd.creator_id, u.name AS creator_name, upd.is_public, upd.creator_id, u.name AS creator_name, upd.is_public,
upd.sort_key, upd.sort_order,
COALESCE(fc.cnt, 0) AS file_count COALESCE(fc.cnt, 0) AS file_count
FROM upd FROM upd
JOIN core.users u ON u.id = upd.creator_id JOIN core.users u ON u.id = upd.creator_id
@@ -333,7 +382,7 @@ LEFT JOIN (SELECT pool_id, COUNT(*) AS cnt FROM data.file_pool WHERE pool_id = $
} }
q := connOrTx(ctx, r.pool) q := connOrTx(ctx, r.pool)
rows, err := q.Query(ctx, query, id, p.Name, p.Notes, meta, p.IsPublic) rows, err := q.Query(ctx, query, id, p.Name, p.Notes, meta, p.IsPublic, p.SortKey, p.SortOrder)
if err != nil { if err != nil {
return nil, fmt.Errorf("PoolRepo.Update: %w", err) return nil, fmt.Errorf("PoolRepo.Update: %w", err)
} }
@@ -402,8 +451,16 @@ func (r *PoolRepo) ListFiles(ctx context.Context, poolID uuid.UUID, params port.
} }
} }
// Cursor condition. // Resolve the pool's sort setting (defaulting to manual position order) into
var orderBy string // the ORDER BY column, direction, and keyset comparison operator.
sortKey := params.SortKey
if !domain.ValidPoolSortKey(sortKey) {
sortKey = domain.PoolSortManual
}
col, dir, cmp := poolFileSort(sortKey, params.SortOrder)
// Keyset cursor condition. For "created" the file id is both the sort key and
// the tiebreak, so a single comparison suffices; the others compare (col, id).
if params.Cursor != "" { if params.Cursor != "" {
cur, err := decodePoolCursor(params.Cursor) cur, err := decodePoolCursor(params.Cursor)
if err != nil { if err != nil {
@@ -413,13 +470,36 @@ func (r *PoolRepo) ListFiles(ctx context.Context, poolID uuid.UUID, params port.
if err != nil { if err != nil {
return nil, domain.ErrValidation return nil, domain.ErrValidation
} }
if sortKey == domain.PoolSortCreated {
conds = append(conds, fmt.Sprintf("f.id %s $%d", cmp, n))
args = append(args, fileID)
n++
} else {
conds = append(conds, fmt.Sprintf( conds = append(conds, fmt.Sprintf(
"(fp.position > $%d OR (fp.position = $%d AND fp.file_id > $%d))", "(%s %s $%d OR (%s = $%d AND f.id %s $%d))", col, cmp, n, col, n, cmp, n+1))
n, n, n+1)) switch sortKey {
args = append(args, cur.Position, fileID) case domain.PoolSortContentDatetime:
t, err := time.Parse(time.RFC3339Nano, cur.Val)
if err != nil {
return nil, domain.ErrValidation
}
args = append(args, t)
case domain.PoolSortOriginalName:
args = append(args, cur.Val)
default: // manual
args = append(args, cur.Pos)
}
args = append(args, fileID)
n += 2 n += 2
} }
orderBy = "fp.position ASC, fp.file_id ASC" }
var orderBy string
if sortKey == domain.PoolSortCreated {
orderBy = fmt.Sprintf("f.id %s", dir)
} else {
orderBy = fmt.Sprintf("%s %s, f.id %s", col, dir, dir)
}
where := "WHERE " + strings.Join(conds, " AND ") where := "WHERE " + strings.Join(conds, " AND ")
args = append(args, limit+1) args = append(args, limit+1)
@@ -458,11 +538,21 @@ LIMIT $%d`, fileSelectForPool, where, orderBy, n)
if hasMore && len(collected) > 0 { if hasMore && len(collected) > 0 {
last := collected[len(collected)-1] last := collected[len(collected)-1]
cur := encodePoolCursor(poolFileCursor{ cursor := poolFileCursor{FileID: last.ID.String()}
Position: last.Position, switch sortKey {
FileID: last.ID.String(), case domain.PoolSortContentDatetime:
}) cursor.Val = last.ContentDatetime.UTC().Format(time.RFC3339Nano)
page.NextCursor = &cur case domain.PoolSortOriginalName:
if last.OriginalName != nil {
cursor.Val = *last.OriginalName
}
case domain.PoolSortCreated:
// file id alone orders; nothing else to carry
default: // manual
cursor.Pos = last.Position
}
enc := encodePoolCursor(cursor)
page.NextCursor = &enc
} }
// Batch-load tags. // Batch-load tags.
+20 -2
View File
@@ -11,12 +11,30 @@ import (
"tanabata/backend/internal/db" "tanabata/backend/internal/db"
) )
// appName tags every connection as application_name, so the backend's sessions
// are identifiable in pg_stat_activity and server logs (and distinguishable from
// e.g. goose migrations or a psql shell).
const appName = "tanabata-backend"
// NewPool creates and validates a *pgxpool.Pool from the given connection URL. // NewPool creates and validates a *pgxpool.Pool from the given connection URL.
// The pool is ready to use; the caller is responsible for closing it. // The pool is ready to use; the caller is responsible for closing it.
func NewPool(ctx context.Context, url string) (*pgxpool.Pool, error) { func NewPool(ctx context.Context, url string) (*pgxpool.Pool, error) {
pool, err := pgxpool.New(ctx, url) cfg, err := pgxpool.ParseConfig(url)
if err != nil { if err != nil {
return nil, fmt.Errorf("pgxpool.New: %w", err) return nil, fmt.Errorf("pgxpool.ParseConfig: %w", err)
}
// Set application_name unless the operator already specified one in the DSN
// (or via PGAPPNAME), so an explicit override still wins.
if cfg.ConnConfig.RuntimeParams == nil {
cfg.ConnConfig.RuntimeParams = map[string]string{}
}
if cfg.ConnConfig.RuntimeParams["application_name"] == "" {
cfg.ConnConfig.RuntimeParams["application_name"] = appName
}
pool, err := pgxpool.NewWithConfig(ctx, cfg)
if err != nil {
return nil, fmt.Errorf("pgxpool.NewWithConfig: %w", err)
} }
if err := pool.Ping(ctx); err != nil { if err := pool.Ping(ctx); err != nil {
pool.Close() pool.Close()
+11 -6
View File
@@ -272,7 +272,7 @@ func (r *TagRepo) Create(ctx context.Context, t *domain.Tag) (*domain.Tag, error
const query = ` const query = `
WITH ins AS ( WITH ins AS (
INSERT INTO data.tags (name, notes, color, category_id, metadata, creator_id, is_public) INSERT INTO data.tags (name, notes, color, category_id, metadata, creator_id, is_public)
VALUES ($1, $2, $3, $4, $5, $6, $7) VALUES ($1, $2, NULLIF($3, ''), $4, $5, $6, $7)
RETURNING * RETURNING *
) )
SELECT SELECT
@@ -321,7 +321,7 @@ WITH upd AS (
UPDATE data.tags SET UPDATE data.tags SET
name = $2, name = $2,
notes = $3, notes = $3,
color = $4, color = NULLIF($4, ''),
category_id = $5, category_id = $5,
metadata = COALESCE($6, metadata), metadata = COALESCE($6, metadata),
is_public = $7 is_public = $7
@@ -604,10 +604,14 @@ WHERE when_tag_id = $1 AND then_tag_id = $2`
if !active || !applyToExisting { if !active || !applyToExisting {
return nil return nil
} }
return r.ApplyToExisting(ctx, whenTagID, thenTagID)
}
// Retroactively apply the full transitive expansion of thenTagID to all // ApplyToExisting retroactively applies the full transitive expansion of
// files that already carry whenTagID. The recursive CTE walks active rules // thenTagID to all files that already carry whenTagID. The recursive CTE walks
// starting from thenTagID (mirrors the Go expandTagSet BFS). // active rules starting from thenTagID (mirrors the Go expandTagSet BFS), so
// inactive downstream rules are not followed. Idempotent via ON CONFLICT.
func (r *TagRuleRepo) ApplyToExisting(ctx context.Context, whenTagID, thenTagID uuid.UUID) error {
const retroQuery = ` const retroQuery = `
WITH RECURSIVE expansion(tag_id) AS ( WITH RECURSIVE expansion(tag_id) AS (
SELECT $2::uuid SELECT $2::uuid
@@ -624,8 +628,9 @@ CROSS JOIN expansion e
WHERE ft.tag_id = $1 WHERE ft.tag_id = $1
ON CONFLICT DO NOTHING` ON CONFLICT DO NOTHING`
q := connOrTx(ctx, r.pool)
if _, err := q.Exec(ctx, retroQuery, whenTagID, thenTagID); err != nil { if _, err := q.Exec(ctx, retroQuery, whenTagID, thenTagID); err != nil {
return fmt.Errorf("TagRuleRepo.SetActive retroactive apply: %w", err) return fmt.Errorf("TagRuleRepo.ApplyToExisting: %w", err)
} }
return nil return nil
} }
+18
View File
@@ -0,0 +1,18 @@
package domain
import "github.com/google/uuid"
// PHashEntry is a file's perceptual hash, the input to duplicate clustering.
type PHashEntry struct {
ID uuid.UUID
PHash int64
}
// DuplicatePair is an unordered pair of files whose perceptual hashes are within
// the configured Hamming threshold. FileA < FileB by UUID byte order (canonical),
// so a pair is represented exactly once.
type DuplicatePair struct {
FileA uuid.UUID
FileB uuid.UUID
Distance int
}
+1
View File
@@ -29,6 +29,7 @@ type File struct {
CreatorName string // denormalized from core.users CreatorName string // denormalized from core.users
IsPublic bool IsPublic bool
IsDeleted bool IsDeleted bool
NeedsReview bool // tagging not yet marked done; cleared by an explicit review action
CreatedAt time.Time // extracted from UUID v7 via UUIDCreatedAt CreatedAt time.Time // extracted from UUID v7 via UUIDCreatedAt
Tags []Tag // loaded with the file Tags []Tag // loaded with the file
} }
+31
View File
@@ -7,6 +7,32 @@ import (
"github.com/google/uuid" "github.com/google/uuid"
) )
// Pool file sort keys. PoolSortManual keeps the user-defined order in
// file_pool.position; the others sort the pool's files by that file field.
const (
PoolSortManual = "manual"
PoolSortContentDatetime = "content_datetime"
PoolSortCreated = "created"
PoolSortOriginalName = "original_name"
SortOrderAsc = "asc"
SortOrderDesc = "desc"
)
// ValidPoolSortKey reports whether s is an accepted pool sort key.
func ValidPoolSortKey(s string) bool {
switch s {
case PoolSortManual, PoolSortContentDatetime, PoolSortCreated, PoolSortOriginalName:
return true
}
return false
}
// ValidSortOrder reports whether s is an accepted sort direction.
func ValidSortOrder(s string) bool {
return s == SortOrderAsc || s == SortOrderDesc
}
// Pool is an ordered collection of files. // Pool is an ordered collection of files.
type Pool struct { type Pool struct {
ID uuid.UUID ID uuid.UUID
@@ -16,6 +42,11 @@ type Pool struct {
CreatorID int16 CreatorID int16
CreatorName string // denormalized CreatorName string // denormalized
IsPublic bool IsPublic bool
// SortKey / SortOrder control how the pool's files are ordered. When SortKey
// is PoolSortManual, files follow the manual position order and can be
// reordered; otherwise they are sorted automatically and reordering is a no-op.
SortKey string
SortOrder string
FileCount int FileCount int
CreatedAt time.Time // extracted from UUID v7 via UUIDCreatedAt CreatedAt time.Time // extracted from UUID v7 via UUIDCreatedAt
} }
@@ -0,0 +1,126 @@
package handler
import (
"net/http"
"strconv"
"github.com/gin-gonic/gin"
"tanabata/backend/internal/domain"
"tanabata/backend/internal/service"
)
// DuplicateHandler handles the /files/duplicates endpoints.
type DuplicateHandler struct {
dupSvc *service.DuplicateService
}
// NewDuplicateHandler creates a DuplicateHandler.
func NewDuplicateHandler(dupSvc *service.DuplicateService) *DuplicateHandler {
return &DuplicateHandler{dupSvc: dupSvc}
}
// List handles GET /files/duplicates — an offset-paginated list of duplicate
// clusters, each a group of files within the perceptual-hash threshold.
func (h *DuplicateHandler) List(c *gin.Context) {
limit, offset := 20, 0
if n, err := strconv.Atoi(c.Query("limit")); err == nil {
limit = n
}
if n, err := strconv.Atoi(c.Query("offset")); err == nil {
offset = n
}
if limit < 1 {
limit = 1
}
if limit > 50 {
limit = 50
}
if offset < 0 {
offset = 0
}
clusters, total, err := h.dupSvc.Clusters(c.Request.Context(), limit, offset)
if err != nil {
respondError(c, err)
return
}
items := make([]gin.H, len(clusters))
for i, cl := range clusters {
fs := make([]fileJSON, len(cl.Files))
for j, f := range cl.Files {
fs[j] = toFileJSON(f)
}
dists := make([]gin.H, len(cl.Distances))
for j, d := range cl.Distances {
dists[j] = gin.H{"a": d.A, "b": d.B, "distance": d.Distance}
}
items[i] = gin.H{"files": fs, "distances": dists}
}
respondJSON(c, http.StatusOK, gin.H{
"items": items,
"total": total,
"limit": limit,
"offset": offset,
})
}
// Dismiss handles POST /files/duplicates/dismiss — mark a pair "not a duplicate".
func (h *DuplicateHandler) Dismiss(c *gin.Context) {
var body struct {
FileIDA string `json:"file_id_a" binding:"required"`
FileIDB string `json:"file_id_b" binding:"required"`
}
if err := c.ShouldBindJSON(&body); err != nil {
respondError(c, domain.ErrValidation)
return
}
ids, err := parseUUIDs([]string{body.FileIDA, body.FileIDB})
if err != nil {
respondError(c, domain.ErrValidation)
return
}
if err := h.dupSvc.Dismiss(c.Request.Context(), ids[0], ids[1]); err != nil {
respondError(c, err)
return
}
c.Status(http.StatusNoContent)
}
// Resolve handles POST /files/duplicates/resolve — merge a duplicate pair,
// keeping one file and folding the chosen fields in from the other. Returns the
// updated survivor. delete_discarded defaults to true.
func (h *DuplicateHandler) Resolve(c *gin.Context) {
var body struct {
Keep string `json:"keep" binding:"required"`
Discard string `json:"discard" binding:"required"`
Fields service.MergeFields `json:"fields"`
DeleteDiscarded *bool `json:"delete_discarded"`
}
if err := c.ShouldBindJSON(&body); err != nil {
respondError(c, domain.ErrValidation)
return
}
ids, err := parseUUIDs([]string{body.Keep, body.Discard})
if err != nil {
respondError(c, domain.ErrValidation)
return
}
del := true
if body.DeleteDiscarded != nil {
del = *body.DeleteDiscarded
}
f, err := h.dupSvc.Resolve(c.Request.Context(), service.MergeSpec{
Keep: ids[0],
Discard: ids[1],
Fields: body.Fields,
DeleteDiscarded: del,
})
if err != nil {
respondError(c, err)
return
}
respondJSON(c, http.StatusOK, toFileJSON(*f))
}
+65 -3
View File
@@ -22,13 +22,14 @@ import (
type FileHandler struct { type FileHandler struct {
fileSvc *service.FileService fileSvc *service.FileService
tagSvc *service.TagService tagSvc *service.TagService
authSvc *service.AuthService
maxUploadBytes int64 maxUploadBytes int64
} }
// NewFileHandler creates a FileHandler. maxUploadBytes caps the size of an // NewFileHandler creates a FileHandler. maxUploadBytes caps the size of an
// uploaded or replacement file. // uploaded or replacement file. authSvc mints content tokens for media URLs.
func NewFileHandler(fileSvc *service.FileService, tagSvc *service.TagService, maxUploadBytes int64) *FileHandler { func NewFileHandler(fileSvc *service.FileService, tagSvc *service.TagService, authSvc *service.AuthService, maxUploadBytes int64) *FileHandler {
return &FileHandler{fileSvc: fileSvc, tagSvc: tagSvc, maxUploadBytes: maxUploadBytes} return &FileHandler{fileSvc: fileSvc, tagSvc: tagSvc, authSvc: authSvc, maxUploadBytes: maxUploadBytes}
} }
// formFileLimited reads the "file" multipart field while bounding how many bytes // formFileLimited reads the "file" multipart field while bounding how many bytes
@@ -83,6 +84,7 @@ type fileJSON struct {
CreatorName string `json:"creator_name"` CreatorName string `json:"creator_name"`
IsPublic bool `json:"is_public"` IsPublic bool `json:"is_public"`
IsDeleted bool `json:"is_deleted"` IsDeleted bool `json:"is_deleted"`
NeedsReview bool `json:"needs_review"`
CreatedAt string `json:"created_at"` CreatedAt string `json:"created_at"`
Tags []tagJSON `json:"tags"` Tags []tagJSON `json:"tags"`
} }
@@ -130,6 +132,7 @@ func toFileJSON(f domain.File) fileJSON {
CreatorName: f.CreatorName, CreatorName: f.CreatorName,
IsPublic: f.IsPublic, IsPublic: f.IsPublic,
IsDeleted: f.IsDeleted, IsDeleted: f.IsDeleted,
NeedsReview: f.NeedsReview,
CreatedAt: f.CreatedAt.Format(time.RFC3339), CreatedAt: f.CreatedAt.Format(time.RFC3339),
Tags: tags, Tags: tags,
} }
@@ -383,6 +386,38 @@ func (h *FileHandler) SoftDelete(c *gin.Context) {
c.Status(http.StatusNoContent) c.Status(http.StatusNoContent)
} }
// ---------------------------------------------------------------------------
// POST /files/:id/content-token
// ---------------------------------------------------------------------------
// CreateContentToken mints a short-lived, single-file capability token the
// client can put in a content URL's access_token query parameter to open or
// stream the original by link (e.g. a long video in a new tab) without the URL
// dying when the 15-minute access token expires. It first enforces view
// permission via fileSvc.Get, so a token is only issued for a file the caller
// may actually read.
func (h *FileHandler) CreateContentToken(c *gin.Context) {
id, ok := parseFileID(c)
if !ok {
return
}
// Authorize (and confirm existence) the same way content serving does.
if _, err := h.fileSvc.Get(c.Request.Context(), id); err != nil {
respondError(c, err)
return
}
userID, isAdmin, _ := domain.UserFromContext(c.Request.Context())
token, expiresIn, err := h.authSvc.GenerateContentToken(id.String(), userID, isAdmin)
if err != nil {
respondError(c, err)
return
}
c.JSON(http.StatusOK, gin.H{"token": token, "expires_in": expiresIn})
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// GET /files/:id/content // GET /files/:id/content
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -628,6 +663,33 @@ func (h *FileHandler) BulkDelete(c *gin.Context) {
c.Status(http.StatusNoContent) c.Status(http.StatusNoContent)
} }
// BulkReview sets the review status on one or more files. A single-file toggle
// is just a one-element file_ids array. Files the caller cannot edit are
// silently skipped (handled in the service).
func (h *FileHandler) BulkReview(c *gin.Context) {
var body struct {
FileIDs []string `json:"file_ids" binding:"required"`
NeedsReview *bool `json:"needs_review" binding:"required"`
}
if err := c.ShouldBindJSON(&body); err != nil || body.NeedsReview == nil {
respondError(c, domain.ErrValidation)
return
}
fileIDs, err := parseUUIDs(body.FileIDs)
if err != nil {
respondError(c, domain.ErrValidation)
return
}
if err := h.fileSvc.SetNeedsReview(c.Request.Context(), fileIDs, *body.NeedsReview); err != nil {
respondError(c, err)
return
}
c.Status(http.StatusNoContent)
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// POST /files/bulk/common-tags // POST /files/bulk/common-tags
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
+50
View File
@@ -5,6 +5,7 @@ import (
"strings" "strings"
"github.com/gin-gonic/gin" "github.com/gin-gonic/gin"
"github.com/google/uuid"
"tanabata/backend/internal/domain" "tanabata/backend/internal/domain"
"tanabata/backend/internal/service" "tanabata/backend/internal/service"
@@ -50,6 +51,55 @@ func (m *AuthMiddleware) Handle() gin.HandlerFunc {
} }
} }
// HandleContent authenticates a file-content GET, accepting either a normal
// access token or a content token scoped (by its fid claim) to the :id in the
// path. The content token is what keeps a long media stream playing after the
// short access token would have expired. View permission is still enforced in
// the handler against the resolved user, so a content token only widens *when*
// a file may be read by URL, never *which* files.
func (m *AuthMiddleware) HandleContent() gin.HandlerFunc {
return func(c *gin.Context) {
token := bearerToken(c)
if token == "" {
contentUnauthorized(c)
return
}
// A regular access token grants access to everything as usual.
if claims, err := m.authSvc.ValidateAccessToken(c.Request.Context(), token); err == nil {
ctx := domain.WithUser(c.Request.Context(), claims.UserID, claims.IsAdmin, claims.SessionID)
c.Request = c.Request.WithContext(ctx)
c.Next()
return
}
// Otherwise accept a content token minted for exactly this file. Normalise
// the path id to canonical form so it matches the minted fid claim.
id, err := uuid.Parse(c.Param("id"))
if err != nil {
contentUnauthorized(c)
return
}
claims, err := m.authSvc.ValidateContentToken(token, id.String())
if err != nil {
contentUnauthorized(c)
return
}
// A content token carries no session (sid 0); it is session-independent.
ctx := domain.WithUser(c.Request.Context(), claims.UserID, claims.IsAdmin, claims.SessionID)
c.Request = c.Request.WithContext(ctx)
c.Next()
}
}
func contentUnauthorized(c *gin.Context) {
c.JSON(http.StatusUnauthorized, errorBody{
Code: domain.ErrUnauthorized.Code(),
Message: "invalid or expired token",
})
c.Abort()
}
// bearerToken extracts the access token from the Authorization header. As a // bearerToken extracts the access token from the Authorization header. As a
// fallback it accepts an ?access_token= query parameter, but only for GET // fallback it accepts an ?access_token= query parameter, but only for GET
// requests — this lets the browser open media (e.g. /files/{id}/content) via a // requests — this lets the browser open media (e.g. /files/{id}/content) via a
+33
View File
@@ -34,6 +34,8 @@ type poolJSON struct {
CreatorID int16 `json:"creator_id"` CreatorID int16 `json:"creator_id"`
CreatorName string `json:"creator_name"` CreatorName string `json:"creator_name"`
IsPublic bool `json:"is_public"` IsPublic bool `json:"is_public"`
SortKey string `json:"sort_key"`
SortOrder string `json:"sort_order"`
FileCount int `json:"file_count"` FileCount int `json:"file_count"`
CreatedAt string `json:"created_at"` CreatedAt string `json:"created_at"`
} }
@@ -51,6 +53,8 @@ func toPoolJSON(p domain.Pool) poolJSON {
CreatorID: p.CreatorID, CreatorID: p.CreatorID,
CreatorName: p.CreatorName, CreatorName: p.CreatorName,
IsPublic: p.IsPublic, IsPublic: p.IsPublic,
SortKey: p.SortKey,
SortOrder: p.SortOrder,
FileCount: p.FileCount, FileCount: p.FileCount,
CreatedAt: p.CreatedAt.UTC().Format(time.RFC3339), CreatedAt: p.CreatedAt.UTC().Format(time.RFC3339),
} }
@@ -160,6 +164,25 @@ func (h *PoolHandler) Get(c *gin.Context) {
respondJSON(c, http.StatusOK, toPoolJSON(*p)) respondJSON(c, http.StatusOK, toPoolJSON(*p))
} }
// ---------------------------------------------------------------------------
// POST /pools/:pool_id/views
// ---------------------------------------------------------------------------
// RecordView logs that the current user viewed the pool (activity.pool_views).
func (h *PoolHandler) RecordView(c *gin.Context) {
id, ok := parsePoolID(c)
if !ok {
return
}
if err := h.poolSvc.RecordView(c.Request.Context(), id); err != nil {
respondError(c, err)
return
}
c.Status(http.StatusNoContent)
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// PATCH /pools/:pool_id // PATCH /pools/:pool_id
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -195,6 +218,16 @@ func (h *PoolHandler) Update(c *gin.Context) {
params.IsPublic = &b params.IsPublic = &b
} }
} }
if v, ok := raw["sort_key"]; ok {
if s, ok := v.(string); ok {
params.SortKey = &s
}
}
if v, ok := raw["sort_order"]; ok {
if s, ok := v.(string); ok {
params.SortOrder = &s
}
}
updated, err := h.poolSvc.Update(c.Request.Context(), id, params) updated, err := h.poolSvc.Update(c.Request.Context(), id, params)
if err != nil { if err != nil {
+32 -4
View File
@@ -1,6 +1,7 @@
package handler package handler
import ( import (
"fmt"
"net/http" "net/http"
"time" "time"
@@ -25,6 +26,7 @@ func NewRouter(
auth *AuthMiddleware, auth *AuthMiddleware,
authHandler *AuthHandler, authHandler *AuthHandler,
fileHandler *FileHandler, fileHandler *FileHandler,
duplicateHandler *DuplicateHandler,
tagHandler *TagHandler, tagHandler *TagHandler,
categoryHandler *CategoryHandler, categoryHandler *CategoryHandler,
poolHandler *PoolHandler, poolHandler *PoolHandler,
@@ -32,10 +34,20 @@ func NewRouter(
aclHandler *ACLHandler, aclHandler *ACLHandler,
auditHandler *AuditHandler, auditHandler *AuditHandler,
staticDir string, staticDir string,
) *gin.Engine { trustedProxies []string,
) (*gin.Engine, error) {
r := gin.New() r := gin.New()
r.Use(gin.Logger(), gin.Recovery(), securityHeaders()) r.Use(gin.Logger(), gin.Recovery(), securityHeaders())
// Behind a reverse proxy the client's real IP arrives in X-Forwarded-For.
// Trust only the proxy hop(s) so c.ClientIP() — used by the auth rate
// limiter — reflects the real client and can't be spoofed by a forged
// header from a direct caller. An empty list trusts no proxy (ClientIP is
// the immediate peer).
if err := r.SetTrustedProxies(trustedProxies); err != nil {
return nil, fmt.Errorf("configure trusted proxies: %w", err)
}
// Health check — no auth required. // Health check — no auth required.
r.GET("/health", func(c *gin.Context) { r.GET("/health", func(c *gin.Context) {
c.JSON(http.StatusOK, gin.H{"status": "ok"}) c.JSON(http.StatusOK, gin.H{"status": "ok"})
@@ -69,9 +81,14 @@ func NewRouter(
files.GET("", fileHandler.List) files.GET("", fileHandler.List)
files.POST("", fileHandler.Upload) files.POST("", fileHandler.Upload)
// Bulk + import routes registered before /:id to prevent param collision. // Bulk + import + duplicates routes registered before /:id to prevent
// param collision (e.g. "duplicates" being captured as :id).
files.GET("/duplicates", duplicateHandler.List)
files.POST("/duplicates/dismiss", duplicateHandler.Dismiss)
files.POST("/duplicates/resolve", duplicateHandler.Resolve)
files.POST("/bulk/tags", fileHandler.BulkSetTags) files.POST("/bulk/tags", fileHandler.BulkSetTags)
files.POST("/bulk/delete", fileHandler.BulkDelete) files.POST("/bulk/delete", fileHandler.BulkDelete)
files.POST("/bulk/review", fileHandler.BulkReview)
files.POST("/bulk/common-tags", fileHandler.CommonTags) files.POST("/bulk/common-tags", fileHandler.CommonTags)
files.POST("/import", fileHandler.Import) files.POST("/import", fileHandler.Import)
@@ -80,8 +97,9 @@ func NewRouter(
files.PATCH("/:id", fileHandler.UpdateMeta) files.PATCH("/:id", fileHandler.UpdateMeta)
files.DELETE("/:id", fileHandler.SoftDelete) files.DELETE("/:id", fileHandler.SoftDelete)
files.GET("/:id/content", fileHandler.GetContent)
files.PUT("/:id/content", fileHandler.ReplaceContent) files.PUT("/:id/content", fileHandler.ReplaceContent)
// Mints a content token (strict auth) for the GET /:id/content route below.
files.POST("/:id/content-token", fileHandler.CreateContentToken)
files.GET("/:id/thumbnail", fileHandler.GetThumbnail) files.GET("/:id/thumbnail", fileHandler.GetThumbnail)
files.GET("/:id/preview", fileHandler.GetPreview) files.GET("/:id/preview", fileHandler.GetPreview)
files.POST("/:id/views", fileHandler.RecordView) files.POST("/:id/views", fileHandler.RecordView)
@@ -95,6 +113,15 @@ func NewRouter(
files.DELETE("/:id/tags/:tag_id", tagHandler.FileRemoveTag) files.DELETE("/:id/tags/:tag_id", tagHandler.FileRemoveTag)
} }
// Serving an original is the one read that can outlive a 15-minute access
// token — a long video streams via repeated Range requests over many minutes.
// So this route alone also accepts a file-scoped content token (see
// HandleContent), letting the media URL stay valid for the whole playback.
media := v1.Group("/files", auth.HandleContent())
{
media.GET("/:id/content", fileHandler.GetContent)
}
// ------------------------------------------------------------------------- // -------------------------------------------------------------------------
// Tags (all require auth) // Tags (all require auth)
// ------------------------------------------------------------------------- // -------------------------------------------------------------------------
@@ -141,6 +168,7 @@ func NewRouter(
pools.GET("/:pool_id", poolHandler.Get) pools.GET("/:pool_id", poolHandler.Get)
pools.PATCH("/:pool_id", poolHandler.Update) pools.PATCH("/:pool_id", poolHandler.Update)
pools.DELETE("/:pool_id", poolHandler.Delete) pools.DELETE("/:pool_id", poolHandler.Delete)
pools.POST("/:pool_id/views", poolHandler.RecordView)
// Sub-routes registered before /:pool_id/files to avoid param conflicts. // Sub-routes registered before /:pool_id/files to avoid param conflicts.
pools.POST("/:pool_id/files/remove", poolHandler.RemoveFiles) pools.POST("/:pool_id/files/remove", poolHandler.RemoveFiles)
@@ -188,5 +216,5 @@ func NewRouter(
r.NoRoute(spaHandler(staticDir)) r.NoRoute(spaHandler(staticDir))
} }
return r return r, nil
} }
+23
View File
@@ -0,0 +1,23 @@
package handler
import "testing"
// TestNewRouterRegisters builds the router with typed-nil dependencies to assert
// route registration itself succeeds. Gin panics on a route conflict (e.g. a
// duplicated method+path or an inconsistent wildcard name) during registration,
// before any handler runs — so this catches such mistakes without a database.
// Handlers are never invoked here; method values on nil pointers are fine.
func TestNewRouterRegisters(t *testing.T) {
r, err := NewRouter(
(*AuthMiddleware)(nil), (*AuthHandler)(nil),
(*FileHandler)(nil), (*DuplicateHandler)(nil), (*TagHandler)(nil), (*CategoryHandler)(nil), (*PoolHandler)(nil),
(*UserHandler)(nil), (*ACLHandler)(nil), (*AuditHandler)(nil),
"", nil,
)
if err != nil {
t.Fatalf("NewRouter: %v", err)
}
if r == nil {
t.Fatal("NewRouter returned nil engine")
}
}
+70
View File
@@ -0,0 +1,70 @@
// Package imagehash computes a 64-bit perceptual hash (dHash) of an image and
// compares two hashes by Hamming distance. It is used for near-duplicate
// detection: visually similar images (re-encoded, resized, recompressed) produce
// hashes a small distance apart, while unrelated images are far apart.
//
// dHash is chosen for its robustness and simplicity: the image is reduced to a
// 9×8 grayscale and each pixel is compared to its right-hand neighbour, yielding
// 64 gradient-direction bits. It tolerates scaling and brightness/contrast
// changes well, which is exactly what re-encoded duplicates exhibit.
package imagehash
import (
"bytes"
"image"
_ "image/gif" // register GIF decoder
_ "image/jpeg" // register JPEG decoder
_ "image/png" // register PNG decoder
"math/bits"
"github.com/disintegration/imaging"
_ "golang.org/x/image/webp" // register WebP decoder
)
// hashWidth/hashHeight define the reduced grayscale used for dHash. The extra
// column (width = height+1) provides the right-hand neighbour for the 64
// horizontal comparisons that make up the hash.
const (
hashHeight = 8
hashWidth = hashHeight + 1
)
// FromImage reduces img to a 9×8 grayscale and returns its 64-bit dHash. The
// uint64 of gradient bits is returned as int64 (a plain bit reinterpretation) so
// it fits PostgreSQL's bigint; equality and Distance are bitwise, so the signed
// interpretation never matters.
func FromImage(img image.Image) int64 {
small := imaging.Grayscale(imaging.Resize(img, hashWidth, hashHeight, imaging.Lanczos))
var hash uint64
bit := 0
for y := 0; y < hashHeight; y++ {
for x := 0; x < hashHeight; x++ {
// After Grayscale, R == G == B, so the red channel is the luminance.
left := small.Pix[small.PixOffset(x, y)]
right := small.Pix[small.PixOffset(x+1, y)]
if left < right {
hash |= 1 << uint(63-bit)
}
bit++
}
}
return int64(hash)
}
// FromBytes decodes data (JPEG/PNG/GIF/WebP) and returns its dHash. ok is false
// when the bytes are not a decodable image, so callers can simply skip hashing
// (e.g. leave phash NULL) rather than fail.
func FromBytes(data []byte) (hash int64, ok bool) {
img, _, err := image.Decode(bytes.NewReader(data))
if err != nil {
return 0, false
}
return FromImage(img), true
}
// Distance returns the Hamming distance (064) between two hashes: the number of
// differing bits. 0 means identical; small values mean near-duplicate.
func Distance(a, b int64) int {
return bits.OnesCount64(uint64(a) ^ uint64(b))
}
@@ -0,0 +1,99 @@
package imagehash
import (
"bytes"
"image"
"image/color"
"image/jpeg"
"image/png"
"math"
"testing"
)
// radial renders a smooth grayscale image whose brightness falls off with
// distance from (cx, cy). Smooth gradients are the realistic case for perceptual
// hashing and survive JPEG re-encoding well, so they make stable test fixtures.
func radial(w, h int, cx, cy float64) image.Image {
img := image.NewRGBA(image.Rect(0, 0, w, h))
maxD := math.Hypot(float64(w), float64(h))
for y := 0; y < h; y++ {
for x := 0; x < w; x++ {
d := math.Hypot(float64(x)-cx, float64(y)-cy)
v := uint8(255 * (1 - d/maxD))
img.Set(x, y, color.RGBA{v, v, v, 255})
}
}
return img
}
func encodePNG(t *testing.T, img image.Image) []byte {
t.Helper()
var buf bytes.Buffer
if err := png.Encode(&buf, img); err != nil {
t.Fatalf("png encode: %v", err)
}
return buf.Bytes()
}
func encodeJPEG(t *testing.T, img image.Image, quality int) []byte {
t.Helper()
var buf bytes.Buffer
if err := jpeg.Encode(&buf, img, &jpeg.Options{Quality: quality}); err != nil {
t.Fatalf("jpeg encode: %v", err)
}
return buf.Bytes()
}
// The same image re-encoded as PNG (lossless) and JPEG (lossy) must hash to a
// small Hamming distance — that is the whole point of a perceptual hash.
func TestFromBytes_SameImageAcrossEncodings(t *testing.T) {
img := radial(64, 64, 32, 32)
pngHash, ok := FromBytes(encodePNG(t, img))
if !ok {
t.Fatal("FromBytes(PNG): ok=false")
}
jpgHash, ok := FromBytes(encodeJPEG(t, img, 90))
if !ok {
t.Fatal("FromBytes(JPEG): ok=false")
}
if d := Distance(pngHash, jpgHash); d > 8 {
t.Errorf("same image, different encodings: distance = %d, want <= 8", d)
}
}
// Visually different images must be far apart, and clearly farther than the same
// image across encodings.
func TestDistance_DifferentImagesAreFarApart(t *testing.T) {
a := FromImage(radial(64, 64, 32, 32)) // centred
b := FromImage(radial(64, 64, 0, 0)) // corner
same, _ := FromBytes(encodeJPEG(t, radial(64, 64, 32, 32), 90))
d := Distance(a, b)
if d < 12 {
t.Errorf("different images: distance = %d, want >= 12", d)
}
if d <= Distance(a, same) {
t.Errorf("different images (%d) not farther than re-encoded same image (%d)", d, Distance(a, same))
}
}
func TestDistance_SymmetricAndZeroForEqual(t *testing.T) {
a := FromImage(radial(64, 64, 20, 40))
b := FromImage(radial(64, 64, 40, 20))
if Distance(a, a) != 0 {
t.Errorf("Distance(a, a) = %d, want 0", Distance(a, a))
}
if Distance(a, b) != Distance(b, a) {
t.Errorf("Distance not symmetric: %d vs %d", Distance(a, b), Distance(b, a))
}
}
func TestFromBytes_RejectsNonImage(t *testing.T) {
if _, ok := FromBytes([]byte("definitely not an image")); ok {
t.Error("FromBytes on garbage: ok=true, want false")
}
}
+516 -39
View File
@@ -16,13 +16,17 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"fmt" "fmt"
"image"
"image/jpeg"
"io" "io"
"mime/multipart" "mime/multipart"
"net" "net"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"net/url"
"os" "os"
"path/filepath" "path/filepath"
"sort"
"strings" "strings"
"testing" "testing"
"time" "time"
@@ -55,6 +59,9 @@ type harness struct {
client *http.Client client *http.Client
importDir string importDir string
pool *pgxpool.Pool pool *pgxpool.Pool
// dupSvc lets duplicate tests trigger a pairs rescan directly: rebuilding the
// pairs table is a CLI/maintenance action with no HTTP endpoint.
dupSvc *service.DuplicateService
} }
// setupSuite creates an ephemeral database, runs migrations, wires the full // setupSuite creates an ephemeral database, runs migrations, wires the full
@@ -110,7 +117,7 @@ func setupSuite(t *testing.T) *harness {
thumbsDir := t.TempDir() thumbsDir := t.TempDir()
importDir := t.TempDir() importDir := t.TempDir()
diskStorage, err := storage.NewDiskStorage(filesDir, thumbsDir, 160, 160, 1920, 1080) diskStorage, err := storage.NewDiskStorage(filesDir, thumbsDir, 160, 160, 1920, 1080, 0, 0)
require.NoError(t, err) require.NoError(t, err)
// --- Repositories -------------------------------------------------------- // --- Repositories --------------------------------------------------------
@@ -124,15 +131,18 @@ func setupSuite(t *testing.T) *harness {
tagRuleRepo := postgres.NewTagRuleRepo(pool) tagRuleRepo := postgres.NewTagRuleRepo(pool)
categoryRepo := postgres.NewCategoryRepo(pool) categoryRepo := postgres.NewCategoryRepo(pool)
poolRepo := postgres.NewPoolRepo(pool) poolRepo := postgres.NewPoolRepo(pool)
duplicatePairRepo := postgres.NewDuplicatePairRepo(pool)
dismissalRepo := postgres.NewDismissalRepo(pool)
transactor := postgres.NewTransactor(pool) transactor := postgres.NewTransactor(pool)
// --- Services ------------------------------------------------------------ // --- Services ------------------------------------------------------------
authSvc := service.NewAuthService(userRepo, sessionRepo, "test-secret", 15*time.Minute, 720*time.Hour) authSvc := service.NewAuthService(userRepo, sessionRepo, "test-secret", 15*time.Minute, 720*time.Hour, 6*time.Hour)
aclSvc := service.NewACLService(aclRepo, fileRepo, tagRepo, categoryRepo, poolRepo, transactor) aclSvc := service.NewACLService(aclRepo, fileRepo, tagRepo, categoryRepo, poolRepo, transactor)
auditSvc := service.NewAuditService(auditRepo) auditSvc := service.NewAuditService(auditRepo)
tagSvc := service.NewTagService(tagRepo, tagRuleRepo, aclSvc, auditSvc, transactor) tagSvc := service.NewTagService(tagRepo, tagRuleRepo, aclSvc, auditSvc, transactor)
categorySvc := service.NewCategoryService(categoryRepo, tagRepo, aclSvc, auditSvc) categorySvc := service.NewCategoryService(categoryRepo, tagRepo, aclSvc, auditSvc)
poolSvc := service.NewPoolService(poolRepo, aclSvc, auditSvc) poolSvc := service.NewPoolService(poolRepo, aclSvc, auditSvc)
duplicateSvc := service.NewDuplicateService(fileRepo, duplicatePairRepo, dismissalRepo, aclSvc, auditSvc, transactor, 10)
fileSvc := service.NewFileService(fileRepo, mimeRepo, diskStorage, aclSvc, auditSvc, tagSvc, transactor, importDir) fileSvc := service.NewFileService(fileRepo, mimeRepo, diskStorage, aclSvc, auditSvc, tagSvc, transactor, importDir)
userSvc := service.NewUserService(userRepo, sessionRepo, auditSvc) userSvc := service.NewUserService(userRepo, sessionRepo, auditSvc)
@@ -143,7 +153,8 @@ func setupSuite(t *testing.T) *harness {
// --- Handlers ------------------------------------------------------------ // --- Handlers ------------------------------------------------------------
authMiddleware := handler.NewAuthMiddleware(authSvc) authMiddleware := handler.NewAuthMiddleware(authSvc)
authHandler := handler.NewAuthHandler(authSvc) authHandler := handler.NewAuthHandler(authSvc)
fileHandler := handler.NewFileHandler(fileSvc, tagSvc, 500<<20) fileHandler := handler.NewFileHandler(fileSvc, tagSvc, authSvc, 500<<20)
duplicateHandler := handler.NewDuplicateHandler(duplicateSvc)
tagHandler := handler.NewTagHandler(tagSvc, fileSvc) tagHandler := handler.NewTagHandler(tagSvc, fileSvc)
categoryHandler := handler.NewCategoryHandler(categorySvc) categoryHandler := handler.NewCategoryHandler(categorySvc)
poolHandler := handler.NewPoolHandler(poolSvc) poolHandler := handler.NewPoolHandler(poolSvc)
@@ -151,12 +162,14 @@ func setupSuite(t *testing.T) *harness {
aclHandler := handler.NewACLHandler(aclSvc) aclHandler := handler.NewACLHandler(aclSvc)
auditHandler := handler.NewAuditHandler(auditSvc) auditHandler := handler.NewAuditHandler(auditSvc)
r := handler.NewRouter( r, err := handler.NewRouter(
authMiddleware, authHandler, authMiddleware, authHandler,
fileHandler, tagHandler, categoryHandler, poolHandler, fileHandler, duplicateHandler, tagHandler, categoryHandler, poolHandler,
userHandler, aclHandler, auditHandler, userHandler, aclHandler, auditHandler,
"", "",
nil,
) )
require.NoError(t, err)
srv := httptest.NewServer(r) srv := httptest.NewServer(r)
t.Cleanup(srv.Close) t.Cleanup(srv.Close)
@@ -167,6 +180,7 @@ func setupSuite(t *testing.T) *harness {
client: srv.Client(), client: srv.Client(),
importDir: importDir, importDir: importDir,
pool: pool, pool: pool,
dupSvc: duplicateSvc,
} }
} }
@@ -672,6 +686,94 @@ func TestTagRuleActivateApplyToExisting(t *testing.T) {
assert.ElementsMatch(t, []string{"animal", "living-thing", "organism"}, tagNames()) assert.ElementsMatch(t, []string{"animal", "living-thing", "organism"}, tagNames())
} }
// TestTagRuleCreateApplyToExisting verifies that *creating* a rule with
// apply_to_existing=true retroactively tags existing files — the same contract
// as activating one (TestTagRuleActivateApplyToExisting), exercised through the
// POST path. Also checks that apply_to_existing=false and an inactive rule both
// leave existing files untouched.
func TestTagRuleCreateApplyToExisting(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
h := setupSuite(t)
tok := h.login("admin", "admin")
mkTag := func(name string) string {
resp := h.doJSON("POST", "/tags", map[string]any{"name": name}, tok)
require.Equal(t, http.StatusCreated, resp.StatusCode, resp.String())
var obj map[string]any
resp.decode(t, &obj)
return obj["id"].(string)
}
tagA := mkTag("animal")
tagB := mkTag("living-thing")
tagC := mkTag("organism")
// Rule B→C: active, so it fires transitively once B lands on the file.
resp := h.doJSON("POST", "/tags/"+tagB+"/rules", map[string]any{
"then_tag_id": tagC,
"is_active": true,
}, tok)
require.Equal(t, http.StatusCreated, resp.StatusCode, resp.String())
// A file carrying only tag A — no A→B rule exists yet.
file := h.uploadJPEG(tok, "cat.jpg")
fileID := file["id"].(string)
resp = h.doJSON("PUT", "/files/"+fileID+"/tags", map[string]any{
"tag_ids": []string{tagA},
}, tok)
require.Equal(t, http.StatusOK, resp.StatusCode, resp.String())
tagNames := func() []string {
r := h.doJSON("GET", "/files/"+fileID+"/tags", nil, tok)
require.Equal(t, http.StatusOK, r.StatusCode)
var items []any
r.decode(t, &items)
names := make([]string, 0, len(items))
for _, it := range items {
names = append(names, it.(map[string]any)["name"].(string))
}
return names
}
assert.ElementsMatch(t, []string{"animal"}, tagNames())
// Creating an INACTIVE rule, even with apply_to_existing=true, must not tag.
resp = h.doJSON("POST", "/tags/"+tagA+"/rules", map[string]any{
"then_tag_id": tagB,
"is_active": false,
"apply_to_existing": true,
}, tok)
require.Equal(t, http.StatusCreated, resp.StatusCode, resp.String())
assert.ElementsMatch(t, []string{"animal"}, tagNames(), "inactive rule must not tag existing files")
// Drop it so we can recreate as active.
resp = h.doJSON("DELETE", "/tags/"+tagA+"/rules/"+tagB, nil, tok)
require.Equal(t, http.StatusNoContent, resp.StatusCode, resp.String())
// Creating an ACTIVE rule with apply_to_existing=false leaves the file alone.
resp = h.doJSON("POST", "/tags/"+tagA+"/rules", map[string]any{
"then_tag_id": tagB,
"is_active": true,
"apply_to_existing": false,
}, tok)
require.Equal(t, http.StatusCreated, resp.StatusCode, resp.String())
assert.ElementsMatch(t, []string{"animal"}, tagNames(), "apply_to_existing=false must not touch existing files")
resp = h.doJSON("DELETE", "/tags/"+tagA+"/rules/"+tagB, nil, tok)
require.Equal(t, http.StatusNoContent, resp.StatusCode, resp.String())
// Creating A→B active WITH apply_to_existing=true: the file gets B directly
// and C transitively via the already-active B→C rule.
resp = h.doJSON("POST", "/tags/"+tagA+"/rules", map[string]any{
"then_tag_id": tagB,
"is_active": true,
"apply_to_existing": true,
}, tok)
require.Equal(t, http.StatusCreated, resp.StatusCode, resp.String())
assert.ElementsMatch(t, []string{"animal", "living-thing", "organism"}, tagNames())
}
// TestTagAutoRule verifies that adding a tag automatically applies then_tags. // TestTagAutoRule verifies that adding a tag automatically applies then_tags.
func TestTagAutoRule(t *testing.T) { func TestTagAutoRule(t *testing.T) {
if testing.Short() { if testing.Short() {
@@ -857,6 +959,75 @@ func TestTagSortByCategoryThenName(t *testing.T) {
assert.Equal(t, []string{"ant", "zebra", "mid", "solo"}, names) assert.Equal(t, []string{"ant", "zebra", "mid", "solo"}, names)
} }
// TestRecordPoolView verifies that viewing a pool is logged (POST .../views),
// is repeatable (view history, not a unique flag), and 404s for unknown pools.
func TestRecordPoolView(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
h := setupSuite(t)
ctx := context.Background()
adminToken := h.login("admin", "admin")
resp := h.doJSON("POST", "/pools", map[string]any{"name": "trip"}, adminToken)
require.Equal(t, http.StatusCreated, resp.StatusCode, resp.String())
var pool map[string]any
resp.decode(t, &pool)
poolID := pool["id"].(string)
resp = h.doJSON("POST", "/pools/"+poolID+"/views", nil, adminToken)
require.Equal(t, http.StatusNoContent, resp.StatusCode, resp.String())
// Viewing again logs another history row, not a conflict.
resp = h.doJSON("POST", "/pools/"+poolID+"/views", nil, adminToken)
require.Equal(t, http.StatusNoContent, resp.StatusCode, resp.String())
var n int
require.NoError(t, h.pool.QueryRow(ctx,
`SELECT count(*) FROM activity.pool_views WHERE pool_id = $1`, poolID).Scan(&n))
assert.Equal(t, 2, n, "each view should add a history row")
// Unknown pool id → 404.
resp = h.doJSON("POST", "/pools/00000000-0000-0000-0000-000000000000/views", nil, adminToken)
require.Equal(t, http.StatusNotFound, resp.StatusCode, resp.String())
}
// TestTagColorOptional verifies a tag can be created without a colour (stored as
// NULL rather than the colour input's default) and that an existing colour can
// be cleared back to none.
func TestTagColorOptional(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
h := setupSuite(t)
adminToken := h.login("admin", "admin")
// Created without a colour → color is null.
resp := h.doJSON("POST", "/tags", map[string]any{"name": "plain"}, adminToken)
require.Equal(t, http.StatusCreated, resp.StatusCode, resp.String())
var plain map[string]any
resp.decode(t, &plain)
assert.Nil(t, plain["color"], "tag created without a colour should have null color")
// Created with a colour → kept verbatim.
resp = h.doJSON("POST", "/tags", map[string]any{"name": "red", "color": "aabbcc"}, adminToken)
require.Equal(t, http.StatusCreated, resp.StatusCode, resp.String())
var red map[string]any
resp.decode(t, &red)
assert.Equal(t, "aabbcc", red["color"])
redID := red["id"].(string)
// Clearing the colour (color: null) must store NULL — an empty string would
// violate the hex CHECK constraint and fail the update.
resp = h.doJSON("PATCH", "/tags/"+redID, map[string]any{"color": nil}, adminToken)
require.Equal(t, http.StatusOK, resp.StatusCode, resp.String())
var cleared map[string]any
resp.decode(t, &cleared)
assert.Nil(t, cleared["color"], "cleared colour should be null")
}
// TestBulkTagAutoRule verifies the bulk add path also applies then_tags. // TestBulkTagAutoRule verifies the bulk add path also applies then_tags.
func TestBulkTagAutoRule(t *testing.T) { func TestBulkTagAutoRule(t *testing.T) {
if testing.Short() { if testing.Short() {
@@ -1136,6 +1307,102 @@ func TestImportFromFolder(t *testing.T) {
assert.True(t, ct.Equal(mtime), "content_datetime %v should equal mtime %v", ct, mtime) assert.True(t, ct.Equal(mtime), "content_datetime %v should equal mtime %v", ct, mtime)
} }
// TestImportOrdersByMtime verifies that a folder import processes files in
// ascending mtime order, regardless of filename order. The three files are named
// so that alphabetical order (ReadDir's default) is the reverse of mtime order;
// the progress-event indices must follow mtime, oldest first.
func TestImportOrdersByMtime(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
h := setupSuite(t)
adminToken := h.login("admin", "admin")
// name → mtime; alphabetical order (a,b,c) is the reverse of chronological.
files := []struct {
name string
mtime time.Time
}{
{"a_newest.jpg", time.Date(2022, 1, 1, 0, 0, 0, 0, time.UTC)},
{"b_middle.jpg", time.Date(2021, 1, 1, 0, 0, 0, 0, time.UTC)},
{"c_oldest.jpg", time.Date(2019, 1, 1, 0, 0, 0, 0, time.UTC)},
}
for _, f := range files {
p := filepath.Join(h.importDir, f.name)
require.NoError(t, os.WriteFile(p, minimalJPEG(), 0o644))
require.NoError(t, os.Chtimes(p, f.mtime, f.mtime))
}
resp := h.doJSON("POST", "/files/import", map[string]any{}, adminToken)
require.Equal(t, http.StatusOK, resp.StatusCode, resp.String())
events := parseImportEvents(t, resp)
idx := map[string]int{}
for _, ev := range events {
if ev.Type == "file" {
require.Equal(t, "imported", ev.Status, "%s: %s", ev.Filename, ev.Reason)
idx[ev.Filename] = ev.Index
}
}
require.Len(t, idx, 3, resp.String())
// Oldest first: c (2019) → b (2021) → a (2022).
assert.Less(t, idx["c_oldest.jpg"], idx["b_middle.jpg"], "oldest should be processed before middle")
assert.Less(t, idx["b_middle.jpg"], idx["a_newest.jpg"], "middle should be processed before newest")
}
// TestFileReviewStatus verifies the per-file "needs review" flag: new uploads
// start as needs_review=true, POST /files/bulk/review clears it, and the DSL
// filter r=0/r=1 selects reviewed/unreviewed files (also combined with others).
func TestFileReviewStatus(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
h := setupSuite(t)
tok := h.login("admin", "admin")
file := h.uploadJPEG(tok, "review-me.jpg")
fileID := file["id"].(string)
assert.Equal(t, true, file["needs_review"], "new upload should need review")
getReview := func() bool {
r := h.doJSON("GET", "/files/"+fileID, nil, tok)
require.Equal(t, http.StatusOK, r.StatusCode, r.String())
var obj map[string]any
r.decode(t, &obj)
return obj["needs_review"].(bool)
}
listIDs := func(dsl string) []string {
r := h.doJSON("GET", "/files?filter="+url.QueryEscape(dsl), nil, tok)
require.Equal(t, http.StatusOK, r.StatusCode, r.String())
var page map[string]any
r.decode(t, &page)
ids := []string{}
for _, it := range page["items"].([]any) {
ids = append(ids, it.(map[string]any)["id"].(string))
}
return ids
}
// Before marking done: appears under r=1 (needs review), not under r=0.
assert.True(t, getReview())
assert.Contains(t, listIDs("{r=1}"), fileID)
assert.NotContains(t, listIDs("{r=0}"), fileID)
// Mark as reviewed (done) via the bulk endpoint (single-element list).
resp := h.doJSON("POST", "/files/bulk/review", map[string]any{
"file_ids": []string{fileID},
"needs_review": false,
}, tok)
require.Equal(t, http.StatusNoContent, resp.StatusCode, resp.String())
// Now reviewed: appears under r=0, not r=1; combines with a MIME predicate.
assert.False(t, getReview(), "file should no longer need review")
assert.Contains(t, listIDs("{r=0}"), fileID)
assert.NotContains(t, listIDs("{r=1}"), fileID)
assert.Contains(t, listIDs("{r=0,&,m~image/%}"), fileID)
}
// TestContentRangeRequests verifies the original-content endpoint answers a // TestContentRangeRequests verifies the original-content endpoint answers a
// byte-range request with 206 Partial Content (so the browser can seek within // byte-range request with 206 Partial Content (so the browser can seek within
// audio/video) rather than streaming the whole body. // audio/video) rather than streaming the whole body.
@@ -1311,41 +1578,25 @@ func TestPoolOperationsRequireACL(t *testing.T) {
// Test helpers // Test helpers
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// minimalJPEG returns the bytes of a 1×1 white JPEG image. // minimalJPEG returns the bytes of a small solid-white JPEG produced by the
// Generated offline; no external dependency needed. // standard library encoder. Encoding is deterministic, so every call yields
// byte-identical output — two uploads therefore hash to the same perceptual
// hash (a solid image dHashes to 0), which is what the duplicate test relies on.
//
// It is encoded (rather than a hand-written byte literal) so the bytes decode
// cleanly through image.Decode: that is the same path FileService.Upload takes
// to compute the perceptual hash, and a fixture that only passes MIME sniffing
// but fails to fully decode would leave phash NULL and silently break dedup.
func minimalJPEG() []byte { func minimalJPEG() []byte {
// This is a valid minimal JPEG: SOI + APP0 + DQT + SOF0 + DHT + SOS + EOI. img := image.NewGray(image.Rect(0, 0, 16, 16))
// 1×1 white pixel, quality ~50. Verified with `file` and browsers. for i := range img.Pix {
return []byte{ img.Pix[i] = 0xff // white
0xff, 0xd8, 0xff, 0xe0, 0x00, 0x10, 0x4a, 0x46, 0x49, 0x46, 0x00, 0x01,
0x01, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0xff, 0xdb, 0x00, 0x43,
0x00, 0x08, 0x06, 0x06, 0x07, 0x06, 0x05, 0x08, 0x07, 0x07, 0x07, 0x09,
0x09, 0x08, 0x0a, 0x0c, 0x14, 0x0d, 0x0c, 0x0b, 0x0b, 0x0c, 0x19, 0x12,
0x13, 0x0f, 0x14, 0x1d, 0x1a, 0x1f, 0x1e, 0x1d, 0x1a, 0x1c, 0x1c, 0x20,
0x24, 0x2e, 0x27, 0x20, 0x22, 0x2c, 0x23, 0x1c, 0x1c, 0x28, 0x37, 0x29,
0x2c, 0x30, 0x31, 0x34, 0x34, 0x34, 0x1f, 0x27, 0x39, 0x3d, 0x38, 0x32,
0x3c, 0x2e, 0x33, 0x34, 0x32, 0xff, 0xc0, 0x00, 0x0b, 0x08, 0x00, 0x01,
0x00, 0x01, 0x01, 0x01, 0x11, 0x00, 0xff, 0xc4, 0x00, 0x1f, 0x00, 0x00,
0x01, 0x05, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,
0x09, 0x0a, 0x0b, 0xff, 0xc4, 0x00, 0xb5, 0x10, 0x00, 0x02, 0x01, 0x03,
0x03, 0x02, 0x04, 0x03, 0x05, 0x05, 0x04, 0x04, 0x00, 0x00, 0x01, 0x7d,
0x01, 0x02, 0x03, 0x00, 0x04, 0x11, 0x05, 0x12, 0x21, 0x31, 0x41, 0x06,
0x13, 0x51, 0x61, 0x07, 0x22, 0x71, 0x14, 0x32, 0x81, 0x91, 0xa1, 0x08,
0x23, 0x42, 0xb1, 0xc1, 0x15, 0x52, 0xd1, 0xf0, 0x24, 0x33, 0x62, 0x72,
0x82, 0x09, 0x0a, 0x16, 0x17, 0x18, 0x19, 0x1a, 0x25, 0x26, 0x27, 0x28,
0x29, 0x2a, 0x34, 0x35, 0x36, 0x37, 0x38, 0x39, 0x3a, 0x43, 0x44, 0x45,
0x46, 0x47, 0x48, 0x49, 0x4a, 0x53, 0x54, 0x55, 0x56, 0x57, 0x58, 0x59,
0x5a, 0x63, 0x64, 0x65, 0x66, 0x67, 0x68, 0x69, 0x6a, 0x73, 0x74, 0x75,
0x76, 0x77, 0x78, 0x79, 0x7a, 0x83, 0x84, 0x85, 0x86, 0x87, 0x88, 0x89,
0x8a, 0x93, 0x94, 0x95, 0x96, 0x97, 0x98, 0x99, 0x9a, 0xa2, 0xa3, 0xa4,
0xa5, 0xa6, 0xa7, 0xa8, 0xa9, 0xaa, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7,
0xb8, 0xb9, 0xba, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7, 0xc8, 0xc9, 0xca,
0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7, 0xd8, 0xd9, 0xda, 0xe1, 0xe2, 0xe3,
0xe4, 0xe5, 0xe6, 0xe7, 0xe8, 0xe9, 0xea, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5,
0xf6, 0xf7, 0xf8, 0xf9, 0xfa, 0xff, 0xda, 0x00, 0x08, 0x01, 0x01, 0x00,
0x00, 0x3f, 0x00, 0xfb, 0xd3, 0xff, 0xd9,
} }
var buf bytes.Buffer
if err := jpeg.Encode(&buf, img, &jpeg.Options{Quality: 75}); err != nil {
panic("minimalJPEG: encode: " + err.Error())
}
return buf.Bytes()
} }
// replaceDSNDatabase returns a copy of dsn with the dbname parameter replaced. // replaceDSNDatabase returns a copy of dsn with the dbname parameter replaced.
@@ -1363,7 +1614,13 @@ func replaceDSNDatabase(dsn, newDB string) string {
} }
return dsn + " dbname=" + newDB return dsn + " dbname=" + newDB
} }
// URL style: not used in our defaults, but handled for completeness. // URL style (e.g. postgres://user:pass@host:port/dbname?opts): the database
// is the URL path. CI passes this form via TANABATA_TEST_ADMIN_DSN, so it
// must swap the path to point the suite at its per-run database.
if u, err := url.Parse(dsn); err == nil {
u.Path = "/" + newDB
return u.String()
}
return dsn return dsn
} }
@@ -1387,3 +1644,223 @@ var (
_ = freePort _ = freePort
_ = writeFile _ = writeFile
) )
// dupListResponse decodes GET /files/duplicates.
type dupListResponse struct {
Items []struct {
Files []struct {
ID string `json:"id"`
Tags []struct {
ID string `json:"id"`
} `json:"tags"`
} `json:"files"`
Distances []struct {
A string `json:"a"`
B string `json:"b"`
Distance int `json:"distance"`
} `json:"distances"`
} `json:"items"`
Total int `json:"total"`
}
// TestDuplicateDetection exercises the full duplicate lifecycle: perceptual hashes
// are computed on upload, a rescan builds the pairs table, the cluster surfaces,
// a field-by-field merge unions tags and trashes the discarded file, and a
// dismissal hides a pair permanently (surviving a re-rescan).
//
// minimalJPEG() is a 1×1 image, so every upload hashes identically — in a fresh
// database any two uploads form one duplicate pair, which keeps this deterministic.
func TestDuplicateDetection(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
h := setupSuite(t)
ctx := context.Background()
admin := h.login("admin", "admin")
// --- two uploads => one duplicate pair after a rescan ---------------------
f1 := h.uploadJPEG(admin, "a.jpg")["id"].(string)
f2 := h.uploadJPEG(admin, "b.jpg")["id"].(string)
// Tag f2 so the merge has something to union onto the survivor.
resp := h.doJSON("POST", "/tags", map[string]any{"name": "kept", "is_public": true}, admin)
require.Equal(t, http.StatusCreated, resp.StatusCode, resp.String())
var tag map[string]any
resp.decode(t, &tag)
tagID := tag["id"].(string)
resp = h.doJSON("PUT", "/files/"+f2+"/tags", map[string]any{"tag_ids": []string{tagID}}, admin)
require.Equal(t, http.StatusOK, resp.StatusCode, resp.String())
require.NoError(t, h.dupSvc.Rescan(ctx, nil))
resp = h.doJSON("GET", "/files/duplicates", nil, admin)
require.Equal(t, http.StatusOK, resp.StatusCode, resp.String())
var list dupListResponse
resp.decode(t, &list)
require.Equal(t, 1, list.Total, "expected one duplicate cluster: %s", resp)
require.Len(t, list.Items, 1)
require.Len(t, list.Items[0].Files, 2)
// The pair's stored distance rides along; identical 1×1 uploads are distance 0.
require.Len(t, list.Items[0].Distances, 1, "the pair's distance should be reported")
assert.Equal(t, 0, list.Items[0].Distances[0].Distance)
// --- resolve: keep f1, union tags from f2, trash f2 ----------------------
resp = h.doJSON("POST", "/files/duplicates/resolve", map[string]any{
"keep": f1,
"discard": f2,
"fields": map[string]any{"tags": "both"},
"delete_discarded": true,
}, admin)
require.Equal(t, http.StatusOK, resp.StatusCode, resp.String())
var survivor struct {
ID string `json:"id"`
Tags []struct {
ID string `json:"id"`
} `json:"tags"`
}
resp.decode(t, &survivor)
assert.Equal(t, f1, survivor.ID)
require.Len(t, survivor.Tags, 1, "survivor should have inherited the discarded file's tag")
assert.Equal(t, tagID, survivor.Tags[0].ID)
// f2 is now trashed, so the pair drops out of the duplicates view.
resp = h.doJSON("GET", "/files/duplicates", nil, admin)
resp.decode(t, &list)
assert.Equal(t, 0, list.Total, "resolved pair should no longer surface: %s", resp)
// --- dismiss: a new pair, hidden and staying hidden across a rescan -------
f3 := h.uploadJPEG(admin, "c.jpg")["id"].(string)
require.NoError(t, h.dupSvc.Rescan(ctx, nil))
resp = h.doJSON("GET", "/files/duplicates", nil, admin)
resp.decode(t, &list)
require.Equal(t, 1, list.Total, "f1 and f3 should now form a cluster: %s", resp)
resp = h.doJSON("POST", "/files/duplicates/dismiss", map[string]any{
"file_id_a": f1, "file_id_b": f3,
}, admin)
require.Equal(t, http.StatusNoContent, resp.StatusCode, resp.String())
resp = h.doJSON("GET", "/files/duplicates", nil, admin)
resp.decode(t, &list)
assert.Equal(t, 0, list.Total, "dismissed pair should be hidden")
// A rescan re-finds the pair but the dismissal still hides it.
require.NoError(t, h.dupSvc.Rescan(ctx, nil))
resp = h.doJSON("GET", "/files/duplicates", nil, admin)
resp.decode(t, &list)
assert.Equal(t, 0, list.Total, "dismissal must survive a rescan")
}
// TestPoolAutomaticSort exercises per-pool file ordering: the default manual
// order, switching to automatic sorts (by name and by creation), that an
// auto-sorted pool rejects manual reordering, and that keyset paging honours the
// active sort.
func TestPoolAutomaticSort(t *testing.T) {
if testing.Short() {
t.Skip("skipping integration test in short mode")
}
h := setupSuite(t)
admin := h.login("admin", "admin")
// Upload three files with distinct names in a non-alphabetical order.
idB := h.uploadJPEG(admin, "b.jpg")["id"].(string)
idA := h.uploadJPEG(admin, "a.jpg")["id"].(string)
idC := h.uploadJPEG(admin, "c.jpg")["id"].(string)
// "created" order is UUID (byte) order — lexicographic on the canonical string.
createdAsc := []string{idA, idB, idC}
sort.Strings(createdAsc)
createdDesc := []string{createdAsc[2], createdAsc[1], createdAsc[0]}
// New pool defaults to manual ordering.
resp := h.doJSON("POST", "/pools", map[string]any{"name": "sortpool"}, admin)
require.Equal(t, http.StatusCreated, resp.StatusCode, resp.String())
var pool map[string]any
resp.decode(t, &pool)
poolID := pool["id"].(string)
require.Equal(t, "manual", pool["sort_key"])
// Add in upload order (b, a, c) → that becomes the manual position order.
resp = h.doJSON("POST", "/pools/"+poolID+"/files",
map[string]any{"file_ids": []string{idB, idA, idC}}, admin)
require.Equal(t, http.StatusCreated, resp.StatusCode, resp.String())
listIDs := func() []string {
resp := h.doJSON("GET", "/pools/"+poolID+"/files", nil, admin)
require.Equal(t, http.StatusOK, resp.StatusCode, resp.String())
var page struct {
Items []struct {
ID string `json:"id"`
} `json:"items"`
}
resp.decode(t, &page)
ids := make([]string, len(page.Items))
for i, it := range page.Items {
ids[i] = it.ID
}
return ids
}
setSort := func(key, order string) {
body := map[string]any{"sort_key": key}
if order != "" {
body["sort_order"] = order
}
resp := h.doJSON("PATCH", "/pools/"+poolID, body, admin)
require.Equal(t, http.StatusOK, resp.StatusCode, resp.String())
}
// Manual = insertion order.
require.Equal(t, []string{idB, idA, idC}, listIDs())
// By name ascending → a, b, c.
setSort("original_name", "asc")
require.Equal(t, []string{idA, idB, idC}, listIDs())
// By creation descending → reverse UUID order.
setSort("created", "desc")
require.Equal(t, createdDesc, listIDs())
// Reordering an auto-sorted pool is rejected.
resp = h.doJSON("PUT", "/pools/"+poolID+"/files/reorder",
map[string]any{"file_ids": []string{idC, idB, idA}}, admin)
require.Equal(t, http.StatusBadRequest, resp.StatusCode, resp.String())
// Back to manual → reordering works again and sticks.
setSort("manual", "")
resp = h.doJSON("PUT", "/pools/"+poolID+"/files/reorder",
map[string]any{"file_ids": []string{idC, idB, idA}}, admin)
require.Equal(t, http.StatusNoContent, resp.StatusCode, resp.String())
require.Equal(t, []string{idC, idB, idA}, listIDs())
// Keyset paging (limit 1) under an automatic sort returns the full sequence.
setSort("original_name", "asc")
var paged []string
cursor := ""
for i := 0; i < 10; i++ {
url := "/pools/" + poolID + "/files?limit=1"
if cursor != "" {
url += "&cursor=" + cursor
}
resp := h.doJSON("GET", url, nil, admin)
require.Equal(t, http.StatusOK, resp.StatusCode, resp.String())
var page struct {
Items []struct {
ID string `json:"id"`
} `json:"items"`
NextCursor *string `json:"next_cursor"`
}
resp.decode(t, &page)
for _, it := range page.Items {
paged = append(paged, it.ID)
}
if page.NextCursor == nil {
break
}
cursor = *page.NextCursor
}
require.Equal(t, []string{idA, idB, idC}, paged)
}
+39
View File
@@ -36,6 +36,10 @@ type PoolFileListParams struct {
Cursor string Cursor string
Limit int Limit int
Filter string // filter DSL expression Filter string // filter DSL expression
// SortKey / SortOrder come from the pool itself (not the request) and select
// how files are ordered: manual position order or an automatic file-field sort.
SortKey string
SortOrder string
} }
// FileRepo is the persistence interface for file records. // FileRepo is the persistence interface for file records.
@@ -48,6 +52,19 @@ type FileRepo interface {
Create(ctx context.Context, f *domain.File) (*domain.File, error) Create(ctx context.Context, f *domain.File) (*domain.File, error)
// Update applies partial metadata changes and returns the updated record. // Update applies partial metadata changes and returns the updated record.
Update(ctx context.Context, id uuid.UUID, f *domain.File) (*domain.File, error) Update(ctx context.Context, id uuid.UUID, f *domain.File) (*domain.File, error)
// SetNeedsReview sets the review status on the given (non-trashed) files.
SetNeedsReview(ctx context.Context, ids []uuid.UUID, value bool) error
// SetPHash sets (or clears, when nil) the perceptual hash of a file.
SetPHash(ctx context.Context, id uuid.UUID, phash *int64) error
// ListMissingPHash returns live image/video files that have no perceptual
// hash yet (the dedup backfill work list).
ListMissingPHash(ctx context.Context) ([]domain.File, error)
// ListAllPHashes returns the id and perceptual hash of every live, hashed
// file (the global input to the dedup rescan; not ACL-filtered).
ListAllPHashes(ctx context.Context) ([]domain.PHashEntry, error)
// CopyPoolMemberships adds targetID to every pool sourceID belongs to,
// skipping pools target is already in (used by the duplicate merge).
CopyPoolMemberships(ctx context.Context, targetID, sourceID uuid.UUID) error
// SoftDelete moves a file to trash (sets is_deleted = true). // SoftDelete moves a file to trash (sets is_deleted = true).
SoftDelete(ctx context.Context, id uuid.UUID) error SoftDelete(ctx context.Context, id uuid.UUID) error
// Restore moves a file out of trash (sets is_deleted = false). // Restore moves a file out of trash (sets is_deleted = false).
@@ -68,6 +85,21 @@ type FileRepo interface {
RecordTagUses(ctx context.Context, userID int16, filterDSL string) error RecordTagUses(ctx context.Context, userID int16, filterDSL string) error
} }
// DuplicatePairRepo persists the precomputed near-duplicate candidate pairs.
type DuplicatePairRepo interface {
// ReplaceAll atomically replaces the whole pairs table (used by the rescan).
ReplaceAll(ctx context.Context, pairs []domain.DuplicatePair) error
// ListVisible returns pairs whose both files are live, not dismissed, and
// (for non-admins) visible to the viewer.
ListVisible(ctx context.Context, viewerID int16, isAdmin bool) ([]domain.DuplicatePair, error)
}
// DismissalRepo persists "not a duplicate" decisions.
type DismissalRepo interface {
// Add records a pair as dismissed (canonical order, idempotent).
Add(ctx context.Context, a, b uuid.UUID, userID int16) error
}
// TagRepo is the persistence interface for tags. // TagRepo is the persistence interface for tags.
type TagRepo interface { type TagRepo interface {
List(ctx context.Context, params OffsetParams) (*domain.TagOffsetPage, error) List(ctx context.Context, params OffsetParams) (*domain.TagOffsetPage, error)
@@ -101,6 +133,10 @@ type TagRuleRepo interface {
// are both true, the full transitive expansion of thenTagID is retroactively // are both true, the full transitive expansion of thenTagID is retroactively
// applied to all files that already carry whenTagID. // applied to all files that already carry whenTagID.
SetActive(ctx context.Context, whenTagID, thenTagID uuid.UUID, active, applyToExisting bool) error SetActive(ctx context.Context, whenTagID, thenTagID uuid.UUID, active, applyToExisting bool) error
// ApplyToExisting retroactively applies the full transitive expansion of
// thenTagID (following active rules) to every file that already carries
// whenTagID. Used when a rule is created or activated with apply_to_existing.
ApplyToExisting(ctx context.Context, whenTagID, thenTagID uuid.UUID) error
Delete(ctx context.Context, whenTagID, thenTagID uuid.UUID) error Delete(ctx context.Context, whenTagID, thenTagID uuid.UUID) error
} }
@@ -129,6 +165,9 @@ type PoolRepo interface {
RemoveFiles(ctx context.Context, poolID uuid.UUID, fileIDs []uuid.UUID) error RemoveFiles(ctx context.Context, poolID uuid.UUID, fileIDs []uuid.UUID) error
// Reorder sets the full ordered sequence of file IDs in the pool. // Reorder sets the full ordered sequence of file IDs in the pool.
Reorder(ctx context.Context, poolID uuid.UUID, fileIDs []uuid.UUID) error Reorder(ctx context.Context, poolID uuid.UUID, fileIDs []uuid.UUID) error
// RecordView appends a view-history row (activity.pool_views) for the user.
RecordView(ctx context.Context, poolID uuid.UUID, userID int16) error
} }
// UserRepo is the persistence interface for users. // UserRepo is the persistence interface for users.
+62
View File
@@ -20,6 +20,11 @@ import (
const ( const (
tokenTypeAccess = "access" tokenTypeAccess = "access"
tokenTypeRefresh = "refresh" tokenTypeRefresh = "refresh"
// tokenTypeContent is a file-scoped capability for reading one file's
// content by URL (originals / media streaming). It is not tied to a session,
// so it outlives the short access TTL and refresh rotation — letting a long
// video keep playing past access-token expiry.
tokenTypeContent = "content"
) )
// dummyPasswordHash is a valid bcrypt hash used to equalise the cost of a login // dummyPasswordHash is a valid bcrypt hash used to equalise the cost of a login
@@ -34,6 +39,8 @@ type Claims struct {
IsAdmin bool `json:"adm"` IsAdmin bool `json:"adm"`
SessionID int `json:"sid"` SessionID int `json:"sid"`
TokenType string `json:"typ"` TokenType string `json:"typ"`
// FileID scopes a content token to a single file; empty on access/refresh.
FileID string `json:"fid,omitempty"`
} }
// TokenPair holds an issued access/refresh token pair with the access TTL. // TokenPair holds an issued access/refresh token pair with the access TTL.
@@ -50,6 +57,7 @@ type AuthService struct {
secret []byte secret []byte
accessTTL time.Duration accessTTL time.Duration
refreshTTL time.Duration refreshTTL time.Duration
contentTTL time.Duration
} }
// NewAuthService creates an AuthService. // NewAuthService creates an AuthService.
@@ -59,6 +67,7 @@ func NewAuthService(
jwtSecret string, jwtSecret string,
accessTTL time.Duration, accessTTL time.Duration,
refreshTTL time.Duration, refreshTTL time.Duration,
contentTTL time.Duration,
) *AuthService { ) *AuthService {
return &AuthService{ return &AuthService{
users: users, users: users,
@@ -66,6 +75,7 @@ func NewAuthService(
secret: []byte(jwtSecret), secret: []byte(jwtSecret),
accessTTL: accessTTL, accessTTL: accessTTL,
refreshTTL: refreshTTL, refreshTTL: refreshTTL,
contentTTL: contentTTL,
} }
} }
@@ -233,6 +243,53 @@ func (s *AuthService) ValidateAccessToken(ctx context.Context, tokenStr string)
return claims, nil return claims, nil
} }
// GenerateContentToken issues a file-scoped capability token authorizing reads of
// one file's content (originals / media streaming) by URL. Unlike the access
// token it carries no session and is not validated against one, so it survives
// refresh rotation and outlives the short access TTL — which is what lets a long
// video keep playing. It is a bearer credential for that single file until
// ContentTokenTTL elapses. Returns the signed token and its lifetime in seconds.
func (s *AuthService) GenerateContentToken(fileID string, userID int16, isAdmin bool) (string, int, error) {
jti, err := randomJTI()
if err != nil {
return "", 0, err
}
now := time.Now()
claims := Claims{
RegisteredClaims: jwt.RegisteredClaims{
ID: jti,
IssuedAt: jwt.NewNumericDate(now),
ExpiresAt: jwt.NewNumericDate(now.Add(s.contentTTL)),
},
UserID: userID,
IsAdmin: isAdmin,
TokenType: tokenTypeContent,
FileID: fileID,
}
signed, err := s.signClaims(claims)
if err != nil {
return "", 0, err
}
return signed, int(s.contentTTL.Seconds()), nil
}
// ValidateContentToken parses a content token and checks it authorizes fileID.
// It verifies the signature and expiry (via parseToken), the content token type,
// and that the embedded file ID matches the requested file — so a token minted
// for one file cannot read another. It is intentionally session-independent (no
// session lookup), which is what lets it outlive access-token/session rotation.
// Per-file view permission is still enforced downstream against the token's user.
func (s *AuthService) ValidateContentToken(tokenStr, fileID string) (*Claims, error) {
claims, err := s.parseToken(tokenStr)
if err != nil {
return nil, domain.ErrUnauthorized
}
if claims.TokenType != tokenTypeContent || claims.FileID != fileID {
return nil, domain.ErrUnauthorized
}
return claims, nil
}
// issueToken signs a JWT with the given parameters. A random JWT ID guarantees // issueToken signs a JWT with the given parameters. A random JWT ID guarantees
// uniqueness even for tokens minted within the same second. // uniqueness even for tokens minted within the same second.
func (s *AuthService) issueToken(userID int16, isAdmin bool, sessionID int, ttl time.Duration, tokenType string) (string, error) { func (s *AuthService) issueToken(userID int16, isAdmin bool, sessionID int, ttl time.Duration, tokenType string) (string, error) {
@@ -252,6 +309,11 @@ func (s *AuthService) issueToken(userID int16, isAdmin bool, sessionID int, ttl
SessionID: sessionID, SessionID: sessionID,
TokenType: tokenType, TokenType: tokenType,
} }
return s.signClaims(claims)
}
// signClaims signs claims into an HS256 JWT with the service secret.
func (s *AuthService) signClaims(claims Claims) (string, error) {
token := jwt.NewWithClaims(jwt.SigningMethodHS256, claims) token := jwt.NewWithClaims(jwt.SigningMethodHS256, claims)
signed, err := token.SignedString(s.secret) signed, err := token.SignedString(s.secret)
if err != nil { if err != nil {
@@ -0,0 +1,80 @@
package service
import (
"testing"
"time"
)
// newContentTokenService builds an AuthService for content-token tests. The
// content-token methods never touch the user/session repos, so nil is fine.
func newContentTokenService(contentTTL time.Duration) *AuthService {
return NewAuthService(nil, nil, "test-secret", 15*time.Minute, 720*time.Hour, contentTTL)
}
func TestContentTokenRoundTrip(t *testing.T) {
s := newContentTokenService(time.Hour)
const fid = "11111111-1111-1111-1111-111111111111"
tok, expiresIn, err := s.GenerateContentToken(fid, 7, true)
if err != nil {
t.Fatalf("GenerateContentToken: %v", err)
}
if expiresIn != int(time.Hour.Seconds()) {
t.Fatalf("expires_in = %d, want %d", expiresIn, int(time.Hour.Seconds()))
}
claims, err := s.ValidateContentToken(tok, fid)
if err != nil {
t.Fatalf("ValidateContentToken: %v", err)
}
if claims.UserID != 7 || !claims.IsAdmin {
t.Fatalf("claims user mismatch: uid=%d adm=%v", claims.UserID, claims.IsAdmin)
}
if claims.FileID != fid || claims.TokenType != tokenTypeContent {
t.Fatalf("claims scope mismatch: fid=%q typ=%q", claims.FileID, claims.TokenType)
}
}
func TestContentTokenRejectsOtherFile(t *testing.T) {
s := newContentTokenService(time.Hour)
tok, _, err := s.GenerateContentToken("11111111-1111-1111-1111-111111111111", 7, false)
if err != nil {
t.Fatal(err)
}
// A token minted for one file must not authorize another.
if _, err := s.ValidateContentToken(tok, "22222222-2222-2222-2222-222222222222"); err == nil {
t.Fatal("expected rejection for a different file id")
}
}
func TestContentTokenRejectsAccessToken(t *testing.T) {
s := newContentTokenService(time.Hour)
// An ordinary access token must not pass as a content token (wrong type).
access, err := s.issueToken(7, false, 1, 15*time.Minute, tokenTypeAccess)
if err != nil {
t.Fatal(err)
}
if _, err := s.ValidateContentToken(access, ""); err == nil {
t.Fatal("expected rejection of an access token as a content token")
}
}
func TestContentTokenRejectsExpired(t *testing.T) {
// Negative TTL → the token is already expired when minted.
s := newContentTokenService(-time.Minute)
const fid = "11111111-1111-1111-1111-111111111111"
tok, _, err := s.GenerateContentToken(fid, 7, false)
if err != nil {
t.Fatal(err)
}
if _, err := s.ValidateContentToken(tok, fid); err == nil {
t.Fatal("expected rejection of an expired content token")
}
}
func TestContentTokenRejectsGarbage(t *testing.T) {
s := newContentTokenService(time.Hour)
if _, err := s.ValidateContentToken("not-a-jwt", "11111111-1111-1111-1111-111111111111"); err == nil {
t.Fatal("expected rejection of a malformed token")
}
}
+173
View File
@@ -0,0 +1,173 @@
package service
import (
"bytes"
"math/bits"
"sort"
"github.com/google/uuid"
"tanabata/backend/internal/domain"
)
// hamming returns the number of differing bits between two perceptual hashes.
func hamming(a, b uint64) int { return bits.OnesCount64(a ^ b) }
// bkNode is a node in a BK-tree over Hamming distance. Files that share the exact
// same hash are collected in ids (a distance-0 collision), so identical images
// don't degenerate the tree into a chain.
type bkNode struct {
hash uint64
ids []uuid.UUID
children map[int]*bkNode
}
// bkTree indexes perceptual hashes for sublinear radius queries. Building one and
// querying every element with a small radius is far cheaper than the O(N²) all-
// pairs comparison at 100k+ files.
type bkTree struct{ root *bkNode }
func (t *bkTree) insert(hash uint64, id uuid.UUID) {
if t.root == nil {
t.root = &bkNode{hash: hash, ids: []uuid.UUID{id}, children: map[int]*bkNode{}}
return
}
node := t.root
for {
d := hamming(hash, node.hash)
if d == 0 {
node.ids = append(node.ids, id)
return
}
child, ok := node.children[d]
if !ok {
node.children[d] = &bkNode{hash: hash, ids: []uuid.UUID{id}, children: map[int]*bkNode{}}
return
}
node = child
}
}
// query visits every node whose hash is within radius of target. The triangle
// inequality bounds which children can hold a match to [d-radius, d+radius].
func (t *bkTree) query(target uint64, radius int, visit func(node *bkNode, dist int)) {
if t.root == nil {
return
}
stack := []*bkNode{t.root}
for len(stack) > 0 {
node := stack[len(stack)-1]
stack = stack[:len(stack)-1]
d := hamming(target, node.hash)
if d <= radius {
visit(node, d)
}
lo, hi := d-radius, d+radius
for cd, child := range node.children {
if cd >= lo && cd <= hi {
stack = append(stack, child)
}
}
}
}
// buildPairs returns every unordered pair of files whose hashes are within
// threshold, each emitted exactly once with FileA < FileB (UUID byte order).
// onProgress, if set, is called periodically with (processed, total).
func buildPairs(entries []domain.PHashEntry, threshold int, onProgress func(done, total int)) []domain.DuplicatePair {
tree := &bkTree{}
for _, e := range entries {
tree.insert(uint64(e.PHash), e.ID)
}
var pairs []domain.DuplicatePair
total := len(entries)
for i := range entries {
e := entries[i]
tree.query(uint64(e.PHash), threshold, func(node *bkNode, dist int) {
for _, other := range node.ids {
// Emit each pair once, from the smaller id, which also skips self.
if bytes.Compare(e.ID[:], other[:]) < 0 {
pairs = append(pairs, domain.DuplicatePair{FileA: e.ID, FileB: other, Distance: dist})
}
}
})
if onProgress != nil && (i+1)%1000 == 0 {
onProgress(i+1, total)
}
}
if onProgress != nil {
onProgress(total, total)
}
return pairs
}
// orderedPair returns the two ids in canonical (a < b by UUID byte order) order,
// matching how the pairs table keys a distance so a lookup hits regardless of the
// argument order.
func orderedPair(a, b uuid.UUID) [2]uuid.UUID {
if bytes.Compare(a[:], b[:]) > 0 {
return [2]uuid.UUID{b, a}
}
return [2]uuid.UUID{a, b}
}
// clusterDistances returns the stored Hamming distance for every pair of files in
// the cluster that has one. Pairs present only transitively have no stored
// distance and are left out.
func clusterDistances(files []domain.File, distByPair map[[2]uuid.UUID]int) []PairDistance {
var out []PairDistance
for i := 0; i < len(files); i++ {
for j := i + 1; j < len(files); j++ {
if d, ok := distByPair[orderedPair(files[i].ID, files[j].ID)]; ok {
out = append(out, PairDistance{A: files[i].ID, B: files[j].ID, Distance: d})
}
}
}
return out
}
// clusterPairs groups pairs into connected components (transitive closure) via
// union-find. Every returned cluster has at least two files; clusters and the ids
// within them are sorted by UUID for stable pagination.
func clusterPairs(pairs []domain.DuplicatePair) [][]uuid.UUID {
parent := map[uuid.UUID]uuid.UUID{}
var find func(uuid.UUID) uuid.UUID
find = func(x uuid.UUID) uuid.UUID {
p, ok := parent[x]
if !ok {
parent[x] = x
return x
}
if p != x {
parent[x] = find(p)
}
return parent[x]
}
union := func(a, b uuid.UUID) {
ra, rb := find(a), find(b)
if ra != rb {
parent[ra] = rb
}
}
for _, p := range pairs {
union(p.FileA, p.FileB)
}
groups := map[uuid.UUID][]uuid.UUID{}
for node := range parent {
root := find(node)
groups[root] = append(groups[root], node)
}
clusters := make([][]uuid.UUID, 0, len(groups))
for _, ids := range groups {
sort.Slice(ids, func(i, j int) bool { return bytes.Compare(ids[i][:], ids[j][:]) < 0 })
clusters = append(clusters, ids)
}
sort.Slice(clusters, func(i, j int) bool {
return bytes.Compare(clusters[i][0][:], clusters[j][0][:]) < 0
})
return clusters
}
@@ -0,0 +1,384 @@
package service
import (
"context"
"encoding/json"
"errors"
"time"
"github.com/google/uuid"
"tanabata/backend/internal/domain"
"tanabata/backend/internal/port"
)
// Merge field source values.
const (
mergeKeep = "keep"
mergeDiscard = "discard"
mergeBoth = "both"
mergeMerge = "merge"
)
// MergeFields chooses, per field, which file supplies the survivor's value when
// resolving a duplicate. Scalars accept "keep"/"discard"; metadata also accepts
// "merge" (shallow object merge, survivor wins on key conflicts); relations
// (tags, pools) accept "keep"/"both" (union) — there is deliberately no option
// to drop the survivor's own tags/pools. An empty value defaults to "keep".
type MergeFields struct {
OriginalName string `json:"original_name"`
Notes string `json:"notes"`
ContentDatetime string `json:"content_datetime"`
IsPublic string `json:"is_public"`
Metadata string `json:"metadata"`
Tags string `json:"tags"`
Pools string `json:"pools"`
}
// MergeSpec is the input to a duplicate resolution: keep one file, fold chosen
// fields in from the other, and (usually) trash the other.
type MergeSpec struct {
Keep uuid.UUID
Discard uuid.UUID
Fields MergeFields
DeleteDiscarded bool
}
// normalize fills empty choices with "keep" and rejects unknown values.
func (m *MergeSpec) normalize() error {
scalar := func(v *string) error {
if *v == "" {
*v = mergeKeep
}
if *v != mergeKeep && *v != mergeDiscard {
return domain.ErrValidation
}
return nil
}
relation := func(v *string) error {
if *v == "" {
*v = mergeKeep
}
if *v != mergeKeep && *v != mergeBoth {
return domain.ErrValidation
}
return nil
}
f := &m.Fields
if err := scalar(&f.OriginalName); err != nil {
return err
}
if err := scalar(&f.Notes); err != nil {
return err
}
if err := scalar(&f.ContentDatetime); err != nil {
return err
}
if err := scalar(&f.IsPublic); err != nil {
return err
}
if f.Metadata == "" {
f.Metadata = mergeKeep
}
if f.Metadata != mergeKeep && f.Metadata != mergeDiscard && f.Metadata != mergeMerge {
return domain.ErrValidation
}
if err := relation(&f.Tags); err != nil {
return err
}
if err := relation(&f.Pools); err != nil {
return err
}
return nil
}
// DuplicateService finds near-duplicate clusters and resolves them.
type DuplicateService struct {
files port.FileRepo
pairs port.DuplicatePairRepo
dismissals port.DismissalRepo
acl *ACLService
audit *AuditService
tx port.Transactor
threshold int
}
// NewDuplicateService creates a DuplicateService. threshold is the maximum
// Hamming distance for two files to be treated as duplicate candidates.
func NewDuplicateService(
files port.FileRepo,
pairs port.DuplicatePairRepo,
dismissals port.DismissalRepo,
acl *ACLService,
audit *AuditService,
tx port.Transactor,
threshold int,
) *DuplicateService {
return &DuplicateService{
files: files,
pairs: pairs,
dismissals: dismissals,
acl: acl,
audit: audit,
tx: tx,
threshold: threshold,
}
}
// Cluster is a group of near-duplicate files together with the pairwise Hamming
// distances known between them. Distances are read from the stored pairs, so two
// files linked into the cluster only transitively (through an intermediate) may
// have no direct distance — that pair is simply omitted.
type Cluster struct {
Files []domain.File
Distances []PairDistance
}
// PairDistance is the stored Hamming distance between two files of a cluster.
type PairDistance struct {
A uuid.UUID
B uuid.UUID
Distance int
}
// Clusters returns a page of duplicate clusters visible to the caller. Pairs are
// read from the precomputed table (no all-pairs scan here) and grouped into
// connected components; pagination is over whole clusters. Each cluster carries
// the stored pairwise distances so callers can show how close the files are.
func (s *DuplicateService) Clusters(ctx context.Context, limit, offset int) (clusters []Cluster, total int, err error) {
userID, isAdmin, _ := domain.UserFromContext(ctx)
pairs, err := s.pairs.ListVisible(ctx, userID, isAdmin)
if err != nil {
return nil, 0, err
}
groups := clusterPairs(pairs)
total = len(groups)
if offset < 0 {
offset = 0
}
if offset >= len(groups) {
return []Cluster{}, total, nil
}
end := offset + limit
if end > len(groups) || limit <= 0 {
end = len(groups)
}
// Index the stored distances once; each page cluster looks up its own pairs.
distByPair := make(map[[2]uuid.UUID]int, len(pairs))
for _, p := range pairs {
distByPair[orderedPair(p.FileA, p.FileB)] = p.Distance
}
out := make([]Cluster, 0, end-offset)
for _, ids := range groups[offset:end] {
files := make([]domain.File, 0, len(ids))
for _, id := range ids {
f, err := s.files.GetByID(ctx, id)
if err != nil {
// A file deleted between the pair read and now just drops out.
if errors.Is(err, domain.ErrNotFound) {
continue
}
return nil, 0, err
}
files = append(files, *f)
}
if len(files) >= 2 {
out = append(out, Cluster{Files: files, Distances: clusterDistances(files, distByPair)})
}
}
return out, total, nil
}
// Rescan recomputes the entire duplicate_pairs table from the current set of
// perceptual hashes. It is the only thing that populates the table, so the
// duplicates view reflects state as of the last rescan. Called by the dedup CLI.
func (s *DuplicateService) Rescan(ctx context.Context, onProgress func(done, total int)) error {
entries, err := s.files.ListAllPHashes(ctx)
if err != nil {
return err
}
pairs := buildPairs(entries, s.threshold, onProgress)
return s.pairs.ReplaceAll(ctx, pairs)
}
// Dismiss records two files as "not a duplicate" so the pair stops surfacing.
// The caller must be able to view both files.
func (s *DuplicateService) Dismiss(ctx context.Context, a, b uuid.UUID) error {
if a == b {
return domain.ErrValidation
}
userID, isAdmin, _ := domain.UserFromContext(ctx)
for _, id := range []uuid.UUID{a, b} {
f, err := s.files.GetByID(ctx, id)
if err != nil {
return err
}
ok, err := s.acl.CanView(ctx, userID, isAdmin, f.CreatorID, f.IsPublic, fileObjectTypeID, id)
if err != nil {
return err
}
if !ok {
return domain.ErrForbidden
}
}
if err := s.dismissals.Add(ctx, a, b, userID); err != nil {
return err
}
objType := fileObjectType
_ = s.audit.Log(ctx, "duplicate_dismiss", &objType, &a, map[string]any{"other": b.String()})
return nil
}
// Resolve merges a duplicate pair: the survivor (keep) takes the chosen fields
// from the other (discard), and the other is trashed when DeleteDiscarded is set.
// The caller must be able to edit both files. Returns the updated survivor.
func (s *DuplicateService) Resolve(ctx context.Context, spec MergeSpec) (*domain.File, error) {
if spec.Keep == spec.Discard {
return nil, domain.ErrValidation
}
if err := spec.normalize(); err != nil {
return nil, err
}
keep, err := s.files.GetByID(ctx, spec.Keep)
if err != nil {
return nil, err
}
discard, err := s.files.GetByID(ctx, spec.Discard)
if err != nil {
return nil, err
}
userID, isAdmin, _ := domain.UserFromContext(ctx)
for _, f := range []*domain.File{keep, discard} {
ok, err := s.acl.CanEdit(ctx, userID, isAdmin, f.CreatorID, fileObjectTypeID, f.ID)
if err != nil {
return nil, err
}
if !ok {
return nil, domain.ErrForbidden
}
}
// FileRepo.Update rewrites all editable scalar columns, so build the complete
// resolved set (each field from keep or discard) rather than a sparse patch.
patch := &domain.File{
OriginalName: pickPtr(spec.Fields.OriginalName, keep.OriginalName, discard.OriginalName),
Notes: pickPtr(spec.Fields.Notes, keep.Notes, discard.Notes),
ContentDatetime: pickTime(spec.Fields.ContentDatetime, keep.ContentDatetime, discard.ContentDatetime),
IsPublic: pickBool(spec.Fields.IsPublic, keep.IsPublic, discard.IsPublic),
Metadata: pickMetadata(spec.Fields.Metadata, keep.Metadata, discard.Metadata),
}
var result *domain.File
txErr := s.tx.WithTx(ctx, func(ctx context.Context) error {
updated, err := s.files.Update(ctx, keep.ID, patch)
if err != nil {
return err
}
if spec.Fields.Tags == mergeBoth {
if err := s.files.SetTags(ctx, keep.ID, unionTagIDs(keep.Tags, discard.Tags)); err != nil {
return err
}
tags, err := s.files.ListTags(ctx, keep.ID)
if err != nil {
return err
}
updated.Tags = tags
}
if spec.Fields.Pools == mergeBoth {
if err := s.files.CopyPoolMemberships(ctx, keep.ID, discard.ID); err != nil {
return err
}
}
if spec.DeleteDiscarded {
if err := s.files.SoftDelete(ctx, discard.ID); err != nil {
return err
}
}
result = updated
return nil
})
if txErr != nil {
return nil, txErr
}
objType := fileObjectType
_ = s.audit.Log(ctx, "file_merge", &objType, &keep.ID, map[string]any{
"discard": spec.Discard.String(),
"fields": spec.Fields,
"deleted_discarded": spec.DeleteDiscarded,
})
return result, nil
}
// --- field pickers ---------------------------------------------------------
func pickPtr(choice string, keep, discard *string) *string {
if choice == mergeDiscard {
return discard
}
return keep
}
func pickBool(choice string, keep, discard bool) bool {
if choice == mergeDiscard {
return discard
}
return keep
}
func pickTime(choice string, keep, discard time.Time) time.Time {
if choice == mergeDiscard {
return discard
}
return keep
}
func unionTagIDs(a, b []domain.Tag) []uuid.UUID {
seen := make(map[uuid.UUID]bool, len(a)+len(b))
ids := make([]uuid.UUID, 0, len(a)+len(b))
for _, t := range append(append([]domain.Tag{}, a...), b...) {
if !seen[t.ID] {
seen[t.ID] = true
ids = append(ids, t.ID)
}
}
return ids
}
// pickMetadata returns keep's metadata, discard's, or a shallow merge in which
// the survivor's keys win on conflict.
func pickMetadata(choice string, keep, discard json.RawMessage) json.RawMessage {
switch choice {
case mergeDiscard:
return discard
case mergeMerge:
km := map[string]json.RawMessage{}
dm := map[string]json.RawMessage{}
_ = json.Unmarshal(keep, &km)
_ = json.Unmarshal(discard, &dm)
out := make(map[string]json.RawMessage, len(km)+len(dm))
for k, v := range dm {
out[k] = v
}
for k, v := range km { // survivor wins
out[k] = v
}
if len(out) == 0 {
return keep
}
b, err := json.Marshal(out)
if err != nil {
return keep
}
return b
default:
return keep
}
}
+152
View File
@@ -0,0 +1,152 @@
package service
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"sort"
"testing"
"github.com/google/uuid"
"tanabata/backend/internal/domain"
)
// id builds a deterministic UUID whose byte order matches n, so tests can reason
// about the canonical (FileA < FileB) ordering buildPairs produces.
func id(n int) uuid.UUID {
return uuid.MustParse(fmt.Sprintf("00000000-0000-0000-0000-%012d", n))
}
func entry(n int, hash uint64) domain.PHashEntry {
return domain.PHashEntry{ID: id(n), PHash: int64(hash)}
}
// pairKey canonicalises a pair for set comparison regardless of emission order.
func pairKey(p domain.DuplicatePair) string {
a, b := p.FileA, p.FileB
if bytes.Compare(a[:], b[:]) > 0 {
a, b = b, a
}
return fmt.Sprintf("%s|%s|%d", a, b, p.Distance)
}
func TestBuildPairs_ThresholdAndCanonicalOrder(t *testing.T) {
entries := []domain.PHashEntry{
entry(1, 0x0000000000000000),
entry(2, 0x0000000000000001), // distance 1 from #1
entry(3, 0x00000000000000FF), // distance 8 from #1, 7 from #2
entry(4, 0xFFFFFFFFFFFFFFFF), // distance 64 from #1
}
// Tight threshold: only the distance-1 pair qualifies.
got := buildPairs(entries, 2, nil)
if len(got) != 1 {
t.Fatalf("threshold 2: got %d pairs, want 1: %+v", len(got), got)
}
if got[0].FileA != id(1) || got[0].FileB != id(2) || got[0].Distance != 1 {
t.Errorf("threshold 2: unexpected pair %+v", got[0])
}
// Canonical order always FileA < FileB.
if bytes.Compare(got[0].FileA[:], got[0].FileB[:]) >= 0 {
t.Error("pair not in canonical FileA < FileB order")
}
// Looser threshold pulls in #3's pairs but never #4.
got8 := buildPairs(entries, 8, nil)
want := map[string]bool{
pairKey(domain.DuplicatePair{FileA: id(1), FileB: id(2), Distance: 1}): true,
pairKey(domain.DuplicatePair{FileA: id(1), FileB: id(3), Distance: 8}): true,
pairKey(domain.DuplicatePair{FileA: id(2), FileB: id(3), Distance: 7}): true,
}
if len(got8) != len(want) {
t.Fatalf("threshold 8: got %d pairs, want %d: %+v", len(got8), len(want), got8)
}
for _, p := range got8 {
if !want[pairKey(p)] {
t.Errorf("threshold 8: unexpected pair %+v", p)
}
}
}
func TestBuildPairs_IdenticalHashesPairAtDistanceZero(t *testing.T) {
entries := []domain.PHashEntry{
entry(1, 0xABCDABCDABCDABCD),
entry(2, 0xABCDABCDABCDABCD),
}
got := buildPairs(entries, 0, nil)
if len(got) != 1 || got[0].Distance != 0 || got[0].FileA != id(1) || got[0].FileB != id(2) {
t.Fatalf("identical hashes: got %+v, want one distance-0 pair (1,2)", got)
}
}
func TestClusterPairs_ConnectedComponents(t *testing.T) {
pairs := []domain.DuplicatePair{
{FileA: id(1), FileB: id(2)},
{FileA: id(2), FileB: id(3)}, // transitively joins 1-2-3
{FileA: id(5), FileB: id(6)},
}
clusters := clusterPairs(pairs)
if len(clusters) != 2 {
t.Fatalf("got %d clusters, want 2: %+v", len(clusters), clusters)
}
// Sorted by smallest id: {1,2,3} then {5,6}.
if len(clusters[0]) != 3 || clusters[0][0] != id(1) || clusters[0][2] != id(3) {
t.Errorf("cluster 0 = %v, want [1 2 3]", clusters[0])
}
if len(clusters[1]) != 2 || clusters[1][0] != id(5) {
t.Errorf("cluster 1 = %v, want [5 6]", clusters[1])
}
// Each cluster's ids are sorted.
for _, c := range clusters {
if !sort.SliceIsSorted(c, func(i, j int) bool { return bytes.Compare(c[i][:], c[j][:]) < 0 }) {
t.Errorf("cluster not sorted: %v", c)
}
}
}
func TestPickMetadata_Merge(t *testing.T) {
keep := json.RawMessage(`{"a":1,"b":2}`)
discard := json.RawMessage(`{"b":9,"c":3}`)
out := pickMetadata(mergeMerge, keep, discard)
var m map[string]int
if err := json.Unmarshal(out, &m); err != nil {
t.Fatalf("merge result not valid JSON: %v (%s)", err, out)
}
want := map[string]int{"a": 1, "b": 2, "c": 3} // survivor wins on "b"
if fmt.Sprint(m) != fmt.Sprint(want) {
t.Errorf("merge = %v, want %v", m, want)
}
if string(pickMetadata(mergeKeep, keep, discard)) != string(keep) {
t.Error("keep choice should return survivor metadata unchanged")
}
if string(pickMetadata(mergeDiscard, keep, discard)) != string(discard) {
t.Error("discard choice should return the other file's metadata")
}
}
func TestMergeSpec_Normalize(t *testing.T) {
// Empty fields default to "keep".
spec := MergeSpec{Keep: id(1), Discard: id(2)}
if err := spec.normalize(); err != nil {
t.Fatalf("normalize empty: %v", err)
}
if spec.Fields.OriginalName != mergeKeep || spec.Fields.Tags != mergeKeep || spec.Fields.Metadata != mergeKeep {
t.Errorf("empty fields not defaulted to keep: %+v", spec.Fields)
}
// "both" is invalid for a scalar field.
bad := MergeSpec{Keep: id(1), Discard: id(2), Fields: MergeFields{Notes: mergeBoth}}
if err := bad.normalize(); !errors.Is(err, domain.ErrValidation) {
t.Errorf("scalar=both: got %v, want ErrValidation", err)
}
// "discard" is invalid for a relation field.
badRel := MergeSpec{Keep: id(1), Discard: id(2), Fields: MergeFields{Tags: mergeDiscard}}
if err := badRel.normalize(); !errors.Is(err, domain.ErrValidation) {
t.Errorf("relation=discard: got %v, want ErrValidation", err)
}
}
+97 -4
View File
@@ -8,6 +8,7 @@ import (
"io" "io"
"os" "os"
"path/filepath" "path/filepath"
"sort"
"strings" "strings"
"time" "time"
@@ -15,6 +16,7 @@ import (
"github.com/google/uuid" "github.com/google/uuid"
"tanabata/backend/internal/domain" "tanabata/backend/internal/domain"
"tanabata/backend/internal/imagehash"
"tanabata/backend/internal/port" "tanabata/backend/internal/port"
) )
@@ -153,6 +155,17 @@ func (s *FileService) Upload(ctx context.Context, p UploadParams) (*domain.File,
} }
exifData, exifDatetime := extractMetadata(data, origName, p.ContentDatetimeFallback) exifData, exifDatetime := extractMetadata(data, origName, p.ContentDatetimeFallback)
// Compute a perceptual hash for images so duplicate detection can later match
// near-identical files. Best-effort: a decode failure just leaves phash unset
// (the dedup CLI backfills it). Video is hashed by that CLI, not inline, to keep
// ffmpeg off the upload path.
var phash *int64
if strings.HasPrefix(mime.Name, "image/") {
if h, ok := imagehash.FromBytes(data); ok {
phash = &h
}
}
// Resolve content datetime: explicit > metadata date > fallback (e.g. import mtime) > zero. // Resolve content datetime: explicit > metadata date > fallback (e.g. import mtime) > zero.
var contentDatetime time.Time var contentDatetime time.Time
if p.ContentDatetime != nil { if p.ContentDatetime != nil {
@@ -186,6 +199,7 @@ func (s *FileService) Upload(ctx context.Context, p UploadParams) (*domain.File,
Notes: p.Notes, Notes: p.Notes,
Metadata: p.Metadata, Metadata: p.Metadata,
EXIF: exifData, EXIF: exifData,
PHash: phash,
CreatorID: userID, CreatorID: userID,
IsPublic: p.IsPublic, IsPublic: p.IsPublic,
} }
@@ -272,7 +286,17 @@ func (s *FileService) Update(ctx context.Context, id uuid.UUID, p UpdateParams)
return nil, domain.ErrForbidden return nil, domain.ErrForbidden
} }
patch := &domain.File{} // The repo rewrites every editable column, so the patch must carry the final
// value for each. Seed the fields that have no always-present input in the
// editor (original_name, metadata) with their current values so a partial
// update that omits them leaves them untouched instead of clearing them. The
// remaining scalars are always supplied by the editor (notes uses an explicit
// null to clear). The merge path builds its own complete patch and calls the
// repo directly, so it is unaffected by this.
patch := &domain.File{
OriginalName: f.OriginalName,
Metadata: f.Metadata,
}
if p.OriginalName != nil { if p.OriginalName != nil {
patch.OriginalName = p.OriginalName patch.OriginalName = p.OriginalName
} }
@@ -452,6 +476,18 @@ func (s *FileService) Replace(ctx context.Context, id uuid.UUID, p UploadParams)
return nil, err return nil, err
} }
// Recompute the perceptual hash from the new content: images inline, anything
// else cleared to NULL so the old content's hash never lingers (the dedup CLI
// recomputes video). Best-effort, like on upload — phash is recomputable.
var phash *int64
if strings.HasPrefix(mime.Name, "image/") {
if h, ok := imagehash.FromBytes(data); ok {
phash = &h
}
}
_ = s.files.SetPHash(ctx, id, phash)
updated.PHash = phash
objType := fileObjectType objType := fileObjectType
_ = s.audit.Log(ctx, "file_replace", &objType, &id, nil) _ = s.audit.Log(ctx, "file_replace", &objType, &id, nil)
return updated, nil return updated, nil
@@ -557,6 +593,47 @@ func (s *FileService) BulkDelete(ctx context.Context, fileIDs []uuid.UUID) error
return nil return nil
} }
// SetNeedsReview sets the review status ("needs tagging" vs marked done) on the
// given files. Each file is checked against edit ACL; files the caller cannot
// edit, or that do not exist, are skipped (same forgiving semantics as
// BulkDelete). Authorized files are updated in a single statement and each is
// audit-logged. Works for one file (single-element slice) or many.
func (s *FileService) SetNeedsReview(ctx context.Context, ids []uuid.UUID, value bool) error {
userID, isAdmin, _ := domain.UserFromContext(ctx)
authorized := make([]uuid.UUID, 0, len(ids))
for _, id := range ids {
f, err := s.files.GetByID(ctx, id)
if err != nil {
if err == domain.ErrNotFound {
continue
}
return err
}
ok, err := s.acl.CanEdit(ctx, userID, isAdmin, f.CreatorID, fileObjectTypeID, id)
if err != nil {
return err
}
if ok {
authorized = append(authorized, id)
}
}
if len(authorized) == 0 {
return nil
}
if err := s.files.SetNeedsReview(ctx, authorized, value); err != nil {
return err
}
objType := fileObjectType
details := map[string]any{"needs_review": value}
for i := range authorized {
_ = s.audit.Log(ctx, "file_review", &objType, &authorized[i], details)
}
return nil
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Import // Import
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -589,6 +666,23 @@ func (s *FileService) Import(ctx context.Context, path string, onProgress func(I
return nil, fmt.Errorf("FileService.Import: read dir %q: %w", dir, err) return nil, fmt.Errorf("FileService.Import: read dir %q: %w", dir, err)
} }
// Import oldest-first: process entries in ascending mtime order so the
// resulting records' creation order matches the files' chronological order.
// mtimes are cached once (re-stat'ing per comparison would be wasteful) and
// reused below as the content_datetime fallback. Entries whose info can't be
// read get a zero time (sort first); they surface their error in the loop.
modTimes := make(map[string]time.Time, len(entries))
for _, e := range entries {
if info, infoErr := e.Info(); infoErr == nil {
modTimes[e.Name()] = info.ModTime()
}
}
// SliceStable preserves ReadDir's name order as a deterministic tiebreak for
// entries sharing an mtime.
sort.SliceStable(entries, func(a, b int) bool {
return modTimes[entries[a].Name()].Before(modTimes[entries[b].Name()])
})
emit := func(ev ImportEvent) { emit := func(ev ImportEvent) {
if onProgress != nil { if onProgress != nil {
onProgress(ev) onProgress(ev)
@@ -646,10 +740,9 @@ func (s *FileService) Import(ctx context.Context, path string, onProgress func(I
// Preserve the file's mtime as a content_datetime fallback (used only when // Preserve the file's mtime as a content_datetime fallback (used only when
// the file has no EXIF date) — once the source is removed below it's the // the file has no EXIF date) — once the source is removed below it's the
// only date left for non-photo files. // only date left for non-photo files. Reuses the mtime cached for sorting.
var mtime *time.Time var mtime *time.Time
if info, statErr := entry.Info(); statErr == nil { if t, ok := modTimes[name]; ok {
t := info.ModTime()
mtime = &t mtime = &t
} }
+62 -7
View File
@@ -13,12 +13,15 @@ import (
const poolObjectType = "pool" const poolObjectType = "pool"
const poolObjectTypeID int16 = 4 // fourth row in 007_seed_data.sql object_types const poolObjectTypeID int16 = 4 // fourth row in 007_seed_data.sql object_types
// PoolParams holds the fields for creating or patching a pool. // PoolParams holds the fields for creating or patching a pool. SortKey and
// SortOrder are pointers so a patch can leave them unchanged (nil) vs set them.
type PoolParams struct { type PoolParams struct {
Name string Name string
Notes *string Notes *string
Metadata json.RawMessage Metadata json.RawMessage
IsPublic *bool IsPublic *bool
SortKey *string
SortOrder *string
} }
// PoolService handles pool CRUD and poolfile management with ACL + audit. // PoolService handles pool CRUD and poolfile management with ACL + audit.
@@ -82,6 +85,16 @@ func (s *PoolService) authorizeView(ctx context.Context, poolID uuid.UUID) error
return nil return nil
} }
// RecordView appends a view-history entry for the current user, enforcing view
// ACL (you can only record a view of a pool you may see).
func (s *PoolService) RecordView(ctx context.Context, id uuid.UUID) error {
userID, _, _ := domain.UserFromContext(ctx)
if err := s.authorizeView(ctx, id); err != nil {
return err
}
return s.pools.RecordView(ctx, id, userID)
}
// authorizeEdit returns nil if the caller may edit the pool, else ErrForbidden // authorizeEdit returns nil if the caller may edit the pool, else ErrForbidden
// (or ErrNotFound if the pool does not exist). // (or ErrNotFound if the pool does not exist).
func (s *PoolService) authorizeEdit(ctx context.Context, poolID uuid.UUID) error { func (s *PoolService) authorizeEdit(ctx context.Context, poolID uuid.UUID) error {
@@ -154,6 +167,18 @@ func (s *PoolService) Update(ctx context.Context, id uuid.UUID, p PoolParams) (*
if p.IsPublic != nil { if p.IsPublic != nil {
patch.IsPublic = *p.IsPublic patch.IsPublic = *p.IsPublic
} }
if p.SortKey != nil {
if !domain.ValidPoolSortKey(*p.SortKey) {
return nil, domain.ErrValidation
}
patch.SortKey = *p.SortKey
}
if p.SortOrder != nil {
if !domain.ValidSortOrder(*p.SortOrder) {
return nil, domain.ErrValidation
}
patch.SortOrder = *p.SortOrder
}
updated, err := s.pools.Update(ctx, id, &patch) updated, err := s.pools.Update(ctx, id, &patch)
if err != nil { if err != nil {
@@ -195,12 +220,24 @@ func (s *PoolService) Delete(ctx context.Context, id uuid.UUID) error {
// Poolfile operations // Poolfile operations
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// ListFiles returns cursor-paginated files within a pool ordered by position, // ListFiles returns cursor-paginated files within a pool, enforcing view ACL.
// enforcing view ACL on the pool. // The ordering is the pool's own stored sort setting (manual position order or
// an automatic file-field sort), not a request parameter.
func (s *PoolService) ListFiles(ctx context.Context, poolID uuid.UUID, params port.PoolFileListParams) (*domain.PoolFilePage, error) { func (s *PoolService) ListFiles(ctx context.Context, poolID uuid.UUID, params port.PoolFileListParams) (*domain.PoolFilePage, error) {
if err := s.authorizeView(ctx, poolID); err != nil { userID, isAdmin, _ := domain.UserFromContext(ctx)
pool, err := s.pools.GetByID(ctx, poolID)
if err != nil {
return nil, err return nil, err
} }
ok, err := s.acl.CanView(ctx, userID, isAdmin, pool.CreatorID, pool.IsPublic, poolObjectTypeID, poolID)
if err != nil {
return nil, err
}
if !ok {
return nil, domain.ErrForbidden
}
params.SortKey = pool.SortKey
params.SortOrder = pool.SortOrder
return s.pools.ListFiles(ctx, poolID, params) return s.pools.ListFiles(ctx, poolID, params)
} }
@@ -232,10 +269,28 @@ func (s *PoolService) RemoveFiles(ctx context.Context, poolID uuid.UUID, fileIDs
} }
// Reorder sets the ordered sequence of file IDs within a pool, enforcing edit // Reorder sets the ordered sequence of file IDs within a pool, enforcing edit
// ACL on the pool. // ACL on the pool. Manual ordering only applies when the pool's sort key is
// "manual"; reordering an auto-sorted pool is rejected as a validation error.
func (s *PoolService) Reorder(ctx context.Context, poolID uuid.UUID, fileIDs []uuid.UUID) error { func (s *PoolService) Reorder(ctx context.Context, poolID uuid.UUID, fileIDs []uuid.UUID) error {
if err := s.authorizeEdit(ctx, poolID); err != nil { userID, isAdmin, _ := domain.UserFromContext(ctx)
pool, err := s.pools.GetByID(ctx, poolID)
if err != nil {
return err return err
} }
return s.pools.Reorder(ctx, poolID, fileIDs) ok, err := s.acl.CanEdit(ctx, userID, isAdmin, pool.CreatorID, poolObjectTypeID, poolID)
if err != nil {
return err
}
if !ok {
return domain.ErrForbidden
}
if pool.SortKey != domain.PoolSortManual {
return domain.ErrValidation
}
if err := s.pools.Reorder(ctx, poolID, fileIDs); err != nil {
return err
}
objType := poolObjectType
_ = s.audit.Log(ctx, "file_pool_reorder", &objType, &poolID, map[string]any{"count": len(fileIDs)})
return nil
} }
+22 -6
View File
@@ -192,16 +192,32 @@ func (s *TagService) ListRules(ctx context.Context, tagID uuid.UUID) ([]domain.T
return s.rules.ListByTag(ctx, tagID) return s.rules.ListByTag(ctx, tagID)
} }
// CreateRule adds a tag rule. If applyToExisting is true, the then_tag is // CreateRule adds a tag rule. When the rule is active and applyToExisting is
// retroactively applied to all files that already carry the when_tag. // true, the full transitive expansion of thenTagID is retroactively applied to
// Retroactive application requires a FileRepo; it is deferred until wired // every file already carrying whenTagID — same semantics as activating an
// in a future iteration (see port.FileRepo.ListByTag). // existing rule via SetRuleActive. The insert and retroactive apply run in one
func (s *TagService) CreateRule(ctx context.Context, whenTagID, thenTagID uuid.UUID, isActive, _ bool) (*domain.TagRule, error) { // transaction so a file is never left half-tagged.
return s.rules.Create(ctx, domain.TagRule{ func (s *TagService) CreateRule(ctx context.Context, whenTagID, thenTagID uuid.UUID, isActive, applyToExisting bool) (*domain.TagRule, error) {
var created *domain.TagRule
err := s.tx.WithTx(ctx, func(ctx context.Context) error {
rule, err := s.rules.Create(ctx, domain.TagRule{
WhenTagID: whenTagID, WhenTagID: whenTagID,
ThenTagID: thenTagID, ThenTagID: thenTagID,
IsActive: isActive, IsActive: isActive,
}) })
if err != nil {
return err
}
created = rule
if isActive && applyToExisting {
return s.rules.ApplyToExisting(ctx, whenTagID, thenTagID)
}
return nil
})
if err != nil {
return nil, err
}
return created, nil
} }
// SetRuleActive toggles a rule's is_active flag and returns the updated rule. // SetRuleActive toggles a rule's is_active flag and returns the updated rule.
+176 -25
View File
@@ -15,6 +15,9 @@ import (
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
"runtime"
"strconv"
"strings"
"time" "time"
"github.com/disintegration/imaging" "github.com/disintegration/imaging"
@@ -38,20 +41,36 @@ type DiskStorage struct {
thumbHeight int thumbHeight int
previewWidth int previewWidth int
previewHeight int previewHeight int
maxPixels int
// genSem bounds concurrent thumbnail/preview generation. Each resize already
// fans out across every core (imaging uses GOMAXPROCS), and large sources cost
// hundreds of MB to decode, so unbounded parallelism on a burst of big images
// pegs the CPU and can exhaust RAM. A buffered channel caps how many run at once.
genSem chan struct{}
} }
var _ port.FileStorage = (*DiskStorage)(nil) var _ port.FileStorage = (*DiskStorage)(nil)
// NewDiskStorage creates a DiskStorage and ensures both directories exist. // NewDiskStorage creates a DiskStorage and ensures both directories exist.
//
// maxPixels caps the source pixel count we will decode in-process (0 → a sane
// default). concurrency bounds simultaneous generation (≤0 → half the CPUs).
func NewDiskStorage( func NewDiskStorage(
filesPath, thumbsPath string, filesPath, thumbsPath string,
thumbW, thumbH, prevW, prevH int, thumbW, thumbH, prevW, prevH int,
maxPixels, concurrency int,
) (*DiskStorage, error) { ) (*DiskStorage, error) {
for _, p := range []string{filesPath, thumbsPath} { for _, p := range []string{filesPath, thumbsPath} {
if err := os.MkdirAll(p, 0o755); err != nil { if err := os.MkdirAll(p, 0o755); err != nil {
return nil, fmt.Errorf("storage: create directory %q: %w", p, err) return nil, fmt.Errorf("storage: create directory %q: %w", p, err)
} }
} }
if maxPixels <= 0 {
maxPixels = defaultMaxDecodePixels
}
if concurrency <= 0 {
concurrency = max(1, runtime.GOMAXPROCS(0)/2)
}
return &DiskStorage{ return &DiskStorage{
filesPath: filesPath, filesPath: filesPath,
thumbsPath: thumbsPath, thumbsPath: thumbsPath,
@@ -59,6 +78,8 @@ func NewDiskStorage(
thumbHeight: thumbH, thumbHeight: thumbH,
previewWidth: prevW, previewWidth: prevW,
previewHeight: prevH, previewHeight: prevH,
maxPixels: maxPixels,
genSem: make(chan struct{}, concurrency),
}, nil }, nil
} }
@@ -136,19 +157,39 @@ func (s *DiskStorage) Preview(ctx context.Context, id uuid.UUID) (io.ReadCloser,
return s.serveGenerated(ctx, id, s.previewCachePath(id), s.previewWidth, s.previewHeight) return s.serveGenerated(ctx, id, s.previewCachePath(id), s.previewWidth, s.previewHeight)
} }
// VideoFrameMiddle decodes a representative frame from the middle of a video
// (duration/2). The midpoint avoids the shared intros, title cards and black
// lead-in frames that make a fixed early offset collide across unrelated clips,
// so it is the right source for the video's perceptual (duplicate-detection)
// hash. The file must already exist in storage; ffmpeg/ffprobe must be installed.
// This is not part of port.FileStorage — only the dedup CLI needs it, with a
// concrete *DiskStorage — so the interface stays lean and ffmpeg stays out of the
// upload path.
func (s *DiskStorage) VideoFrameMiddle(ctx context.Context, id uuid.UUID) (image.Image, error) {
srcPath := s.originalPath(id)
if _, err := os.Stat(srcPath); err != nil {
if os.IsNotExist(err) {
return nil, domain.ErrNotFound
}
return nil, fmt.Errorf("storage: stat %q: %w", srcPath, err)
}
return extractVideoFrameMiddle(ctx, srcPath)
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Internal helpers // Internal helpers
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// serveGenerated is the shared implementation for Thumbnail and Preview. Both // serveGenerated is the shared implementation for Thumbnail and Preview. Both
// fit the source within maxW×maxH with imaging.Fit, preserving the aspect ratio // fit the source within maxW×maxH preserving the aspect ratio (no crop, no
// (no crop, no upscale); they differ only in the configured dimensions. // upscale); they differ only in the configured dimensions.
// //
// Resolution order: // Resolution order:
// 1. Return cached JPEG if present. // 1. Return cached JPEG if present.
// 2. Decode as still image (JPEG/PNG/GIF via imaging). // 2. vipsthumbnail (shrink-on-load; the primary still-image path).
// 3. Extract a frame with ffmpeg (video files). // 3. Pure-Go decode + imaging.Fit (fallback when vips is absent).
// 4. Solid-colour placeholder (archives, unrecognised formats, etc.). // 4. Extract a frame with ffmpeg (video files).
// 5. Solid-colour placeholder (archives, unrecognised formats, etc.).
func (s *DiskStorage) serveGenerated(ctx context.Context, id uuid.UUID, cachePath string, maxW, maxH int) (io.ReadCloser, error) { func (s *DiskStorage) serveGenerated(ctx context.Context, id uuid.UUID, cachePath string, maxW, maxH int) (io.ReadCloser, error) {
// Fast path: cache hit. // Fast path: cache hit.
if f, err := os.Open(cachePath); err == nil { if f, err := os.Open(cachePath); err == nil {
@@ -164,13 +205,39 @@ func (s *DiskStorage) serveGenerated(ctx context.Context, id uuid.UUID, cachePat
return nil, fmt.Errorf("storage: stat %q: %w", srcPath, err) return nil, fmt.Errorf("storage: stat %q: %w", srcPath, err)
} }
// 1. Try still-image decode (JPEG/PNG/GIF), rejecting decompression bombs. // Bound concurrent generation so a burst of large images can't peg every core
// 2. Try video frame extraction via ffmpeg. // or exhaust RAM. Queue here (respecting cancellation) rather than starting
// 3. Fall back to placeholder. // the heavy decode immediately.
select {
case s.genSem <- struct{}{}:
defer func() { <-s.genSem }()
case <-ctx.Done():
return nil, ctx.Err()
}
// Another request may have generated this while we waited on the semaphore.
if f, err := os.Open(cachePath); err == nil {
return f, nil
}
// Primary path: vipsthumbnail. It shrinks on load (e.g. JPEG DCT scaling), so
// even a 200+ Mpx photo is thumbnailed in a fraction of the memory and CPU of a
// full in-process decode, writing the final JPEG straight to the cache. Falls
// through when vips is absent or can't read the source (e.g. a video).
if vipsThumbnailPath != "" {
if rc, err := s.vipsThumbnail(ctx, srcPath, cachePath, maxW, maxH); err == nil {
return rc, nil
}
}
// Fallback pipeline (pure Go):
// 1. Still-image decode (JPEG/PNG/GIF), rejecting oversized rasters.
// 2. Video frame extraction via ffmpeg.
// 3. Solid-colour placeholder.
var img image.Image var img image.Image
if decoded, err := decodeImageLimited(srcPath); err == nil { if decoded, err := decodeImageLimited(srcPath, s.maxPixels); err == nil {
img = imaging.Fit(decoded, maxW, maxH, imaging.Lanczos) img = imaging.Fit(decoded, maxW, maxH, imaging.Lanczos)
} else if frame, err := extractVideoFrame(ctx, srcPath); err == nil { } else if frame, err := extractVideoFrameMiddle(ctx, srcPath); err == nil {
img = imaging.Fit(frame, maxW, maxH, imaging.Lanczos) img = imaging.Fit(frame, maxW, maxH, imaging.Lanczos)
} else { } else {
img = placeholder(maxW, maxH) img = placeholder(maxW, maxH)
@@ -221,15 +288,15 @@ func writeCache(cachePath string, img image.Image) (io.ReadCloser, error) {
return f, nil return f, nil
} }
// maxDecodePixels caps the pixel count of an image we are willing to decode // defaultMaxDecodePixels is the fallback cap when none is configured. It bounds
// into memory, bounding the cost of a decompression bomb (a tiny file that // the cost of a decompression bomb (a tiny file that expands to an enormous
// expands to an enormous raster). 64 Mpx is ~ an 8192×8192 image. // raster) and the per-image memory; ~300 Mpx covers e.g. a 13000×17000 photo.
const maxDecodePixels = 64 << 20 const defaultMaxDecodePixels = 300_000_000
// decodeImageLimited decodes the image at path after first inspecting its header // decodeImageLimited decodes the image at path after first inspecting its header
// dimensions via image.DecodeConfig (which does not allocate the raster), and // dimensions via image.DecodeConfig (which does not allocate the raster), and
// refuses images whose pixel count exceeds maxDecodePixels. // refuses images whose pixel count exceeds maxPixels.
func decodeImageLimited(path string) (image.Image, error) { func decodeImageLimited(path string, maxPixels int) (image.Image, error) {
f, err := os.Open(path) f, err := os.Open(path)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -240,7 +307,7 @@ func decodeImageLimited(path string) (image.Image, error) {
if err != nil { if err != nil {
return nil, err return nil, err
} }
if int64(cfg.Width)*int64(cfg.Height) > maxDecodePixels { if int64(cfg.Width)*int64(cfg.Height) > int64(maxPixels) {
return nil, fmt.Errorf("image too large to decode: %dx%d", cfg.Width, cfg.Height) return nil, fmt.Errorf("image too large to decode: %dx%d", cfg.Width, cfg.Height)
} }
if _, err := f.Seek(0, io.SeekStart); err != nil { if _, err := f.Seek(0, io.SeekStart); err != nil {
@@ -249,19 +316,80 @@ func decodeImageLimited(path string) (image.Image, error) {
return imaging.Decode(f, imaging.AutoOrientation(true)) return imaging.Decode(f, imaging.AutoOrientation(true))
} }
// extractVideoFrame uses ffmpeg to extract a single frame from a video file. // vipsThumbnailPath is the resolved path to the vipsthumbnail CLI, or "" when it
// It seeks 1 second in (keyframe-accurate fast seek) and pipes the frame out // isn't installed — in which case generation falls back to the pure-Go pipeline.
// as PNG. If the video is shorter than 1 s the seek is silently ignored by var vipsThumbnailPath, _ = exec.LookPath("vipsthumbnail")
// ffmpeg and the first available frame is returned instead.
// Returns an error if ffmpeg is not installed or produces no output. The run is // vipsThumbnail generates a JPEG thumbnail with the vipsthumbnail CLI, writing it
// bounded by a timeout so a malformed file cannot hang the request indefinitely. // straight to cachePath via an atomic temp→rename. vips decodes large images at a
func extractVideoFrame(ctx context.Context, srcPath string) (image.Image, error) { // reduced scale (shrink-on-load), so this costs a fraction of the memory and CPU
// of a full in-process decode. The result is fit within maxW×maxH and never
// upscaled (the ">" size modifier). Returns an error for inputs vips can't read
// (e.g. videos) so the caller can fall back.
func (s *DiskStorage) vipsThumbnail(ctx context.Context, srcPath, cachePath string, maxW, maxH int) (io.ReadCloser, error) {
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
tmp, err := os.CreateTemp(filepath.Dir(cachePath), ".vips-*.jpg")
if err != nil {
return nil, fmt.Errorf("storage: create temp file: %w", err)
}
tmpName := tmp.Name()
_ = tmp.Close()
cmd := exec.CommandContext(ctx, vipsThumbnailPath,
srcPath,
"--size", fmt.Sprintf("%dx%d>", maxW, maxH),
"--output", tmpName+"[Q=85]",
)
cmd.Stderr = io.Discard
if err := cmd.Run(); err != nil {
os.Remove(tmpName)
return nil, fmt.Errorf("vipsthumbnail: %w", err)
}
if fi, err := os.Stat(tmpName); err != nil || fi.Size() == 0 {
os.Remove(tmpName)
return nil, fmt.Errorf("vipsthumbnail: no output produced")
}
if err := os.Rename(tmpName, cachePath); err != nil {
os.Remove(tmpName)
return nil, fmt.Errorf("storage: rename cache file: %w", err)
}
f, err := os.Open(cachePath)
if err != nil {
return nil, fmt.Errorf("storage: open cache file: %w", err)
}
return f, nil
}
// extractVideoFrameMiddle extracts a single frame from the middle of the video
// (duration/2), falling back to a 1s offset when the duration can't be probed.
// The midpoint dodges shared intros, title cards and black lead-in frames, and
// matches the frame used for the perceptual hash so a video's thumbnail/preview
// shows the same representative frame dedup compared. See extractVideoFrameAt for
// the mechanics.
func extractVideoFrameMiddle(ctx context.Context, srcPath string) (image.Image, error) {
at := 1.0
if d, err := videoDurationSeconds(ctx, srcPath); err == nil && d > 0 {
at = d / 2
}
return extractVideoFrameAt(ctx, srcPath, at)
}
// extractVideoFrameAt uses ffmpeg to extract a single frame at atSec seconds into
// the video, piped out as PNG. The fast input seek (-ss before -i) is keyframe-
// accurate and cheap; if atSec is past the end the seek is silently ignored and
// the first available frame is returned instead. Returns an error if ffmpeg is
// not installed or produces no output. The run is bounded by a timeout so a
// malformed file cannot hang the caller indefinitely.
func extractVideoFrameAt(ctx context.Context, srcPath string, atSec float64) (image.Image, error) {
ctx, cancel := context.WithTimeout(ctx, 30*time.Second) ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel() defer cancel()
var out bytes.Buffer var out bytes.Buffer
cmd := exec.CommandContext(ctx, "ffmpeg", cmd := exec.CommandContext(ctx, "ffmpeg",
"-ss", "1", // fast input seek; ignored gracefully on short files "-ss", strconv.FormatFloat(atSec, 'f', 3, 64), // fast input seek; ignored gracefully past end
"-i", srcPath, "-i", srcPath,
"-vframes", "1", "-vframes", "1",
"-f", "image2", "-f", "image2",
@@ -277,6 +405,29 @@ func extractVideoFrame(ctx context.Context, srcPath string) (image.Image, error)
return imaging.Decode(&out) return imaging.Decode(&out)
} }
// videoDurationSeconds returns the container duration in seconds via ffprobe.
// Used to seek to the middle of a clip for perceptual hashing and thumbnails.
func videoDurationSeconds(ctx context.Context, srcPath string) (float64, error) {
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, "ffprobe",
"-v", "error",
"-show_entries", "format=duration",
"-of", "default=noprint_wrappers=1:nokey=1",
srcPath,
)
out, err := cmd.Output()
if err != nil {
return 0, fmt.Errorf("ffprobe duration: %w", err)
}
d, err := strconv.ParseFloat(strings.TrimSpace(string(out)), 64)
if err != nil {
return 0, fmt.Errorf("ffprobe duration parse %q: %w", out, err)
}
return d, nil
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Path helpers // Path helpers
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
+181
View File
@@ -0,0 +1,181 @@
package storage
import (
"bytes"
"context"
"image"
"image/color"
"image/png"
"io"
"os"
"path/filepath"
"testing"
"github.com/disintegration/imaging"
"github.com/google/uuid"
)
// writeTestImage writes a w×h PNG filled with a distinct (non-placeholder)
// colour so a generated thumbnail can be told apart from the grey placeholder.
func writeTestImage(t *testing.T, path string, w, h int) {
t.Helper()
img := image.NewNRGBA(image.Rect(0, 0, w, h))
for y := 0; y < h; y++ {
for x := 0; x < w; x++ {
img.Set(x, y, color.NRGBA{R: 0xC0, G: 0x10, B: 0x20, A: 0xFF})
}
}
f, err := os.Create(path)
if err != nil {
t.Fatal(err)
}
defer f.Close()
if err := png.Encode(f, img); err != nil {
t.Fatal(err)
}
}
func TestDecodeImageLimited(t *testing.T) {
dir := t.TempDir()
p := filepath.Join(dir, "img.png")
writeTestImage(t, p, 100, 80) // 8000 px
if _, err := decodeImageLimited(p, 4000); err == nil {
t.Fatal("expected rejection for an image over the pixel cap")
}
img, err := decodeImageLimited(p, 100000)
if err != nil {
t.Fatalf("expected decode within the cap, got %v", err)
}
if b := img.Bounds(); b.Dx() != 100 || b.Dy() != 80 {
t.Fatalf("unexpected decoded size %v", b.Size())
}
}
// TestThumbnailGeneratesAndCaches exercises the full generation path (semaphore
// acquire → decode → fit → encode → cache) and the cache fast path on re-request.
func TestThumbnailGeneratesAndCaches(t *testing.T) {
files := t.TempDir()
thumbs := t.TempDir()
id := uuid.New()
writeTestImage(t, filepath.Join(files, id.String()), 100, 80)
s, err := NewDiskStorage(files, thumbs, 160, 160, 1920, 1080, 0, 1)
if err != nil {
t.Fatal(err)
}
rc, err := s.Thumbnail(context.Background(), id)
if err != nil {
t.Fatalf("Thumbnail: %v", err)
}
data, _ := io.ReadAll(rc)
rc.Close()
out, err := imaging.Decode(bytes.NewReader(data))
if err != nil {
t.Fatalf("decode thumbnail: %v", err)
}
// The source fits within 160×160, so it is not upscaled.
if b := out.Bounds(); b.Dx() != 100 || b.Dy() != 80 {
t.Fatalf("unexpected thumbnail size %v", b.Size())
}
// Centre pixel should be the source's red, not the grey placeholder.
r, g, b, _ := out.At(50, 40).RGBA()
if !(r>>8 > g>>8+40 && r>>8 > b>>8+40) {
t.Fatalf("thumbnail is not the source image (got r=%d g=%d b=%d) — fell back to placeholder?", r>>8, g>>8, b>>8)
}
// The cache file must now exist, and a second request must serve it.
if _, err := os.Stat(s.thumbCachePath(id)); err != nil {
t.Fatalf("cache file not written: %v", err)
}
rc2, err := s.Thumbnail(context.Background(), id)
if err != nil {
t.Fatalf("Thumbnail (cached): %v", err)
}
rc2.Close()
}
// TestThumbnailFallbackWithoutVips forces the pure-Go pipeline (as if vips were
// not installed) and verifies generation still produces the source image.
func TestThumbnailFallbackWithoutVips(t *testing.T) {
orig := vipsThumbnailPath
vipsThumbnailPath = ""
t.Cleanup(func() { vipsThumbnailPath = orig })
files := t.TempDir()
thumbs := t.TempDir()
id := uuid.New()
writeTestImage(t, filepath.Join(files, id.String()), 100, 80)
s, err := NewDiskStorage(files, thumbs, 160, 160, 1920, 1080, 0, 1)
if err != nil {
t.Fatal(err)
}
rc, err := s.Thumbnail(context.Background(), id)
if err != nil {
t.Fatalf("Thumbnail: %v", err)
}
data, _ := io.ReadAll(rc)
rc.Close()
out, err := imaging.Decode(bytes.NewReader(data))
if err != nil {
t.Fatalf("decode thumbnail: %v", err)
}
if b := out.Bounds(); b.Dx() != 100 || b.Dy() != 80 {
t.Fatalf("unexpected thumbnail size %v", b.Size())
}
r, g, b, _ := out.At(50, 40).RGBA()
if !(r>>8 > g>>8+40 && r>>8 > b>>8+40) {
t.Fatalf("fallback produced a placeholder, not the source (r=%d g=%d b=%d)", r>>8, g>>8, b>>8)
}
}
// TestPreviewGeneratesAndCaches verifies Preview runs through the same pipeline
// with the preview dimensions and its own cache file (not the thumbnail's).
func TestPreviewGeneratesAndCaches(t *testing.T) {
files := t.TempDir()
thumbs := t.TempDir()
id := uuid.New()
// Larger than the thumbnail box but within the preview box, so the preview
// keeps full resolution where a thumbnail would shrink it.
writeTestImage(t, filepath.Join(files, id.String()), 400, 300)
s, err := NewDiskStorage(files, thumbs, 160, 160, 1920, 1080, 0, 1)
if err != nil {
t.Fatal(err)
}
rc, err := s.Preview(context.Background(), id)
if err != nil {
t.Fatalf("Preview: %v", err)
}
data, _ := io.ReadAll(rc)
rc.Close()
out, err := imaging.Decode(bytes.NewReader(data))
if err != nil {
t.Fatalf("decode preview: %v", err)
}
// 400×300 fits within 1920×1080, so the preview is not downscaled.
if b := out.Bounds(); b.Dx() != 400 || b.Dy() != 300 {
t.Fatalf("unexpected preview size %v", b.Size())
}
r, g, b, _ := out.At(200, 150).RGBA()
if !(r>>8 > g>>8+40 && r>>8 > b>>8+40) {
t.Fatalf("preview is not the source image (r=%d g=%d b=%d)", r>>8, g>>8, b>>8)
}
// The preview cache must be written, and the thumbnail cache must not — they
// are separate files served by the same code with different dimensions.
if _, err := os.Stat(s.previewCachePath(id)); err != nil {
t.Fatalf("preview cache not written: %v", err)
}
if _, err := os.Stat(s.thumbCachePath(id)); err == nil {
t.Fatal("thumbnail cache should not exist after a preview-only request")
}
}
+41 -2
View File
@@ -55,7 +55,8 @@ CREATE TABLE data.files (
creator_id smallint NOT NULL REFERENCES core.users(id) creator_id smallint NOT NULL REFERENCES core.users(id)
ON UPDATE CASCADE ON DELETE RESTRICT, ON UPDATE CASCADE ON DELETE RESTRICT,
is_public boolean NOT NULL DEFAULT false, is_public boolean NOT NULL DEFAULT false,
is_deleted boolean NOT NULL DEFAULT false -- soft delete (trash) is_deleted boolean NOT NULL DEFAULT false, -- soft delete (trash)
needs_review boolean NOT NULL DEFAULT true -- tagging not yet marked done; cleared explicitly
); );
CREATE TABLE data.file_tag ( CREATE TABLE data.file_tag (
@@ -75,8 +76,17 @@ CREATE TABLE data.pools (
creator_id smallint NOT NULL REFERENCES core.users(id) creator_id smallint NOT NULL REFERENCES core.users(id)
ON UPDATE CASCADE ON DELETE RESTRICT, ON UPDATE CASCADE ON DELETE RESTRICT,
is_public boolean NOT NULL DEFAULT false, is_public boolean NOT NULL DEFAULT false,
-- File ordering within the pool. 'manual' keeps the user-defined order in
-- data.file_pool.position (drag-to-reorder); any other key sorts the pool's
-- files automatically by that file field, in which case reordering is disabled.
sort_key varchar(32) NOT NULL DEFAULT 'manual',
sort_order varchar(4) NOT NULL DEFAULT 'asc',
CONSTRAINT uni__pools__name UNIQUE (name) CONSTRAINT uni__pools__name UNIQUE (name),
CONSTRAINT chk__pools__sort_key
CHECK (sort_key IN ('manual', 'content_datetime', 'created', 'original_name')),
CONSTRAINT chk__pools__sort_order
CHECK (sort_order IN ('asc', 'desc'))
); );
-- `position` uses integer with gaps (e.g. 1000, 2000, 3000) to allow -- `position` uses integer with gaps (e.g. 1000, 2000, 3000) to allow
@@ -91,6 +101,31 @@ CREATE TABLE data.file_pool (
PRIMARY KEY (file_id, pool_id) PRIMARY KEY (file_id, pool_id)
); );
-- Precomputed near-duplicate candidates (phash Hamming distance <= threshold),
-- (re)built in full by the dedup rescan. Stored once per unordered pair with a
-- canonical file_a < file_b ordering so a pair is never duplicated as (a,b)/(b,a).
CREATE TABLE data.duplicate_pairs (
file_a uuid NOT NULL REFERENCES data.files(id) ON UPDATE CASCADE ON DELETE CASCADE,
file_b uuid NOT NULL REFERENCES data.files(id) ON UPDATE CASCADE ON DELETE CASCADE,
distance smallint NOT NULL,
CONSTRAINT chk__duplicate_pairs__order CHECK (file_a < file_b),
PRIMARY KEY (file_a, file_b)
);
-- "Not a duplicate" decisions: a global overlay that hides a candidate pair from
-- the duplicates view. Survives rescans (the pair may be re-found but stays
-- hidden). Same canonical file_a < file_b ordering as data.duplicate_pairs.
CREATE TABLE data.duplicate_dismissals (
file_a uuid NOT NULL REFERENCES data.files(id) ON UPDATE CASCADE ON DELETE CASCADE,
file_b uuid NOT NULL REFERENCES data.files(id) ON UPDATE CASCADE ON DELETE CASCADE,
dismissed_by smallint NOT NULL REFERENCES core.users(id) ON UPDATE CASCADE ON DELETE RESTRICT,
dismissed_at timestamptz NOT NULL DEFAULT clock_timestamp(),
CONSTRAINT chk__duplicate_dismissals__order CHECK (file_a < file_b),
PRIMARY KEY (file_a, file_b)
);
COMMENT ON TABLE data.categories IS 'Logical grouping of tags'; COMMENT ON TABLE data.categories IS 'Logical grouping of tags';
COMMENT ON TABLE data.tags IS 'File labels/tags'; COMMENT ON TABLE data.tags IS 'File labels/tags';
COMMENT ON TABLE data.tag_rules IS 'Auto-tagging rules: when when_tag is assigned, then_tag follows'; COMMENT ON TABLE data.tag_rules IS 'Auto-tagging rules: when when_tag is assigned, then_tag follows';
@@ -98,6 +133,8 @@ COMMENT ON TABLE data.files IS 'Managed files; actual content stored on di
COMMENT ON TABLE data.file_tag IS 'Many-to-many: files <-> tags'; COMMENT ON TABLE data.file_tag IS 'Many-to-many: files <-> tags';
COMMENT ON TABLE data.pools IS 'Ordered collections of files'; COMMENT ON TABLE data.pools IS 'Ordered collections of files';
COMMENT ON TABLE data.file_pool IS 'Many-to-many: files <-> pools, with ordering'; COMMENT ON TABLE data.file_pool IS 'Many-to-many: files <-> pools, with ordering';
COMMENT ON TABLE data.duplicate_pairs IS 'Precomputed near-duplicate candidate pairs (perceptual-hash distance)';
COMMENT ON TABLE data.duplicate_dismissals IS 'Pairs marked "not a duplicate"; hidden from the duplicates view';
COMMENT ON COLUMN data.files.original_name IS 'Original filename at upload time'; COMMENT ON COLUMN data.files.original_name IS 'Original filename at upload time';
COMMENT ON COLUMN data.files.content_datetime IS 'Content datetime (e.g. when photo was taken); falls back to EXIF DateTimeOriginal'; COMMENT ON COLUMN data.files.content_datetime IS 'Content datetime (e.g. when photo was taken); falls back to EXIF DateTimeOriginal';
@@ -109,6 +146,8 @@ COMMENT ON COLUMN data.file_pool.position IS 'Manual ordering within pool; u
-- +goose Down -- +goose Down
DROP TABLE IF EXISTS data.duplicate_dismissals;
DROP TABLE IF EXISTS data.duplicate_pairs;
DROP TABLE IF EXISTS data.file_pool; DROP TABLE IF EXISTS data.file_pool;
DROP TABLE IF EXISTS data.pools; DROP TABLE IF EXISTS data.pools;
DROP TABLE IF EXISTS data.file_tag; DROP TABLE IF EXISTS data.file_tag;
+10
View File
@@ -20,11 +20,18 @@ CREATE INDEX idx__files__creator_id ON data.files USING hash (creator_id)
CREATE INDEX idx__files__content_datetime ON data.files USING btree (content_datetime DESC NULLS LAST); CREATE INDEX idx__files__content_datetime ON data.files USING btree (content_datetime DESC NULLS LAST);
CREATE INDEX idx__files__is_deleted ON data.files USING btree (is_deleted) WHERE is_deleted = true; CREATE INDEX idx__files__is_deleted ON data.files USING btree (is_deleted) WHERE is_deleted = true;
CREATE INDEX idx__files__phash ON data.files USING btree (phash) WHERE phash IS NOT NULL; CREATE INDEX idx__files__phash ON data.files USING btree (phash) WHERE phash IS NOT NULL;
CREATE INDEX idx__files__needs_review ON data.files USING btree (id) WHERE needs_review = true;
-- data.file_tag -- data.file_tag
CREATE INDEX idx__file_tag__tag_id ON data.file_tag USING hash (tag_id); CREATE INDEX idx__file_tag__tag_id ON data.file_tag USING hash (tag_id);
CREATE INDEX idx__file_tag__file_id ON data.file_tag USING hash (file_id); CREATE INDEX idx__file_tag__file_id ON data.file_tag USING hash (file_id);
-- data.duplicate_pairs / data.duplicate_dismissals
-- The composite primary keys cover lookups on file_a; these add the file_b side
-- (used by the ON DELETE CASCADE and by the visibility join on the second file).
CREATE INDEX idx__duplicate_pairs__file_b ON data.duplicate_pairs USING hash (file_b);
CREATE INDEX idx__duplicate_dismissals__file_b ON data.duplicate_dismissals USING hash (file_b);
-- data.pools -- data.pools
CREATE INDEX idx__pools__creator_id ON data.pools USING hash (creator_id); CREATE INDEX idx__pools__creator_id ON data.pools USING hash (creator_id);
@@ -69,11 +76,14 @@ DROP INDEX IF EXISTS activity.idx__sessions__token_hash;
DROP INDEX IF EXISTS activity.idx__sessions__user_id; DROP INDEX IF EXISTS activity.idx__sessions__user_id;
DROP INDEX IF EXISTS acl.idx__acl__user; DROP INDEX IF EXISTS acl.idx__acl__user;
DROP INDEX IF EXISTS acl.idx__acl__object; DROP INDEX IF EXISTS acl.idx__acl__object;
DROP INDEX IF EXISTS data.idx__duplicate_dismissals__file_b;
DROP INDEX IF EXISTS data.idx__duplicate_pairs__file_b;
DROP INDEX IF EXISTS data.idx__file_pool__file_id; DROP INDEX IF EXISTS data.idx__file_pool__file_id;
DROP INDEX IF EXISTS data.idx__file_pool__pool_id; DROP INDEX IF EXISTS data.idx__file_pool__pool_id;
DROP INDEX IF EXISTS data.idx__pools__creator_id; DROP INDEX IF EXISTS data.idx__pools__creator_id;
DROP INDEX IF EXISTS data.idx__file_tag__file_id; DROP INDEX IF EXISTS data.idx__file_tag__file_id;
DROP INDEX IF EXISTS data.idx__file_tag__tag_id; DROP INDEX IF EXISTS data.idx__file_tag__tag_id;
DROP INDEX IF EXISTS data.idx__files__needs_review;
DROP INDEX IF EXISTS data.idx__files__phash; DROP INDEX IF EXISTS data.idx__files__phash;
DROP INDEX IF EXISTS data.idx__files__is_deleted; DROP INDEX IF EXISTS data.idx__files__is_deleted;
DROP INDEX IF EXISTS data.idx__files__content_datetime; DROP INDEX IF EXISTS data.idx__files__content_datetime;
+3 -2
View File
@@ -20,7 +20,8 @@ INSERT INTO activity.action_types (name) VALUES
('user_login'), ('user_logout'), ('user_login'), ('user_logout'),
-- Files -- Files
('file_create'), ('file_edit'), ('file_delete'), ('file_restore'), ('file_create'), ('file_edit'), ('file_delete'), ('file_restore'),
('file_permanent_delete'), ('file_replace'), ('file_permanent_delete'), ('file_replace'), ('file_review'),
('file_merge'), ('duplicate_dismiss'),
-- Tags -- Tags
('tag_create'), ('tag_edit'), ('tag_delete'), ('tag_create'), ('tag_edit'), ('tag_delete'),
-- Categories -- Categories
@@ -29,7 +30,7 @@ INSERT INTO activity.action_types (name) VALUES
('pool_create'), ('pool_edit'), ('pool_delete'), ('pool_create'), ('pool_edit'), ('pool_delete'),
-- Relations -- Relations
('file_tag_add'), ('file_tag_remove'), ('file_tag_add'), ('file_tag_remove'),
('file_pool_add'), ('file_pool_remove'), ('file_pool_add'), ('file_pool_remove'), ('file_pool_reorder'),
-- ACL -- ACL
('acl_change'), ('acl_change'),
-- Admin -- Admin
+75 -3
View File
@@ -26,6 +26,12 @@ services:
container_name: tfm container_name: tfm
restart: unless-stopped restart: unless-stopped
# Give the app time to drain in-flight requests on stop before SIGKILL. Reads
# the same SHUTDOWN_TIMEOUT the app uses for its graceful-shutdown deadline
# (via env_file below), so the two never drift. Interpolated from .env at
# `docker compose` time; defaults to 15s if unset.
stop_grace_period: ${SHUTDOWN_TIMEOUT:-15s}
# All application config (secrets, DATABASE_URL, tunables) comes from .env. # All application config (secrets, DATABASE_URL, tunables) comes from .env.
env_file: .env env_file: .env
@@ -35,10 +41,23 @@ services:
environment: environment:
STATIC_DIR: /app/static STATIC_DIR: /app/static
# The container always listens on 42776 (Dockerfile default); APP_PORT only # Published on loopback only: a reverse proxy on the host (e.g. nginx) fronts
# changes the host-published port. # the app and proxies to 127.0.0.1:${APP_PORT}. Binding to 127.0.0.1 keeps the
# app off the LAN/WAN — a plain "PORT:42776" would publish on 0.0.0.0 and, since
# Docker's DNAT rules sit ahead of the host firewall, bypass ufw/firewalld. The
# container always listens on 42776 (Dockerfile default); APP_PORT only changes
# the host-published port. Drop the 127.0.0.1 prefix if exposing it directly.
ports: ports:
- "${APP_PORT:-42776}:42776" - "127.0.0.1:${APP_PORT:-42776}:42776"
# Two-tier networking. `web` is the app's public-facing bridge (reached via the
# published loopback port above; it also provides egress, e.g. to a host
# Postgres via host.docker.internal). `backend` is the private tier the app
# uses to reach the bundled DB. The DB sits only on `backend`, so nothing on
# the host-facing side can reach it.
networks:
- web
- backend
# Wait for the bundled DB when the with-db profile is active. When using a # Wait for the bundled DB when the with-db profile is active. When using a
# host Postgres the db service is disabled, and required:false keeps this # host Postgres the db service is disabled, and required:false keeps this
@@ -76,6 +95,10 @@ services:
# the app at a Postgres running on the host instead. # the app at a Postgres running on the host instead.
profiles: ["with-db"] profiles: ["with-db"]
# Private back-end tier only — never on `web`, never published.
networks:
- backend
environment: environment:
POSTGRES_DB: ${POSTGRES_DB:-tanabata} POSTGRES_DB: ${POSTGRES_DB:-tanabata}
POSTGRES_USER: ${POSTGRES_USER:-tanabata} POSTGRES_USER: ${POSTGRES_USER:-tanabata}
@@ -97,6 +120,55 @@ services:
timeout: 5s timeout: 5s
retries: 10 retries: 10
# One-shot maintenance task for duplicate detection: computes missing
# perceptual hashes (images + video) and rebuilds the duplicate-pairs table.
# It is NOT a daemon — the "tools" profile keeps it out of `docker compose up`;
# run it on demand, and it exits when done:
#
# docker compose run --rm dedup # hashes, then rebuild pairs
# docker compose run --rm dedup -pairs # only rebuild pairs (after uploads)
# docker compose run --rm dedup -hashes # only backfill hashes
#
# Reuses the app image, .env, volumes and networks; only the entrypoint differs
# (/app/dedup instead of the server). Connects to the same DB the app uses, so
# the app's DB (bundled or host) must be reachable when it runs.
dedup:
build:
context: .
dockerfile: Dockerfile
profiles: ["tools"]
env_file: .env
networks:
- web
- backend
depends_on:
db:
condition: service_healthy
required: false
extra_hosts:
- "host.docker.internal:host-gateway"
user: "${PUID:-42776}:${PGID:-42776}"
volumes:
- "${FILES_DIR:-app_files}:/data/files"
- "${THUMBS_DIR:-app_thumbs}:/data/thumbs"
entrypoint: ["/app/dedup"]
restart: "no"
networks:
# Public-facing bridge for this app. The explicit bridge name (instead of
# Docker's random br-<hash>) makes it identifiable on the host for tcpdump and
# firewall rules.
web:
driver_opts:
com.docker.network.bridge.name: dk-tanabata
# Private back-end tier (app ↔ DB). internal:true drops the gateway so the DB
# has no route off-host. Note: Linux caps interface names at 15 chars, and
# dk-tanabata-bnd is exactly 15 — a longer app name would need a shorter suffix.
backend:
internal: true
driver_opts:
com.docker.network.bridge.name: dk-tanabata-bnd
volumes: volumes:
app_files: app_files:
app_thumbs: app_thumbs:
+180
View File
@@ -0,0 +1,180 @@
# Tanabata File Manager — Architecture
System-level overview of Tanabata File Manager (TFM). For the product-level
requirements see [REQUIREMENTS.md](REQUIREMENTS.md); for per-side detail see
[GO_PROJECT_STRUCTURE.md](GO_PROJECT_STRUCTURE.md) (backend) and
[FRONTEND_STRUCTURE.md](FRONTEND_STRUCTURE.md) (frontend). The full HTTP contract
lives in [`openapi.yaml`](../openapi.yaml).
## System Context
TFM is a multi-user, tag-based web file manager for images and video. It is a
single deployable unit — one Docker image that serves both the REST API and the
built single-page app on one port — plus a PostgreSQL database.
```
┌───────────────────────────────────────────┐
Browser / installed │ Reverse proxy (nginx, TLS) │
PWA (desktop/mobile) │ host: 443 → 127.0.0.1:${APP_PORT} │
│ HTTPS └─────────────────────┬─────────────────────┘
└─────────────────────────────────────┼──────────────► 127.0.0.1:42776
┌─────────────────────────▼────────────────────────┐
│ Tanabata container (single image) │
│ │
│ Go server (Gin) │
│ ├─ /api/v1/* REST API │
│ ├─ /health liveness │
│ └─ /* static SPA + index.html │
│ fallback │
│ │
│ Disk: /data/files (originals, name = UUID) │
│ /data/thumbs (thumbnail/preview cache) │
│ /data/import (server-side import drop) │
└─────────────────────────┬────────────────────────┘
│ pgx (private network)
┌─────────▼─────────┐
│ PostgreSQL 14+ │
│ (bundled or host)│
└───────────────────┘
```
Optional companion process: a one-shot **dedup CLI** (same image, different
entrypoint) that backfills perceptual hashes and rebuilds the duplicate-pairs
table. It is not a daemon — it is run on demand.
## Components
| Component | Tech | Responsibility |
| -------------- | -------------------------------------------------------------------------- | -------------------------------------------------------- |
| Frontend (SPA) | SvelteKit (adapter-static, `ssr=false`), Svelte 5, Tailwind v4, TypeScript | UI, client routing, PWA/offline, calls the REST API |
| API server | Go + Gin, Clean Architecture | REST API, auth, ACL, business logic, thumbnailing, audit |
| Database | PostgreSQL 14+ (pgx v5, goose) | All structured data across 4 schemas / 19 tables |
| File storage | Local disk, flat, keyed by UUID | Originals + a regenerable thumbnail/preview cache |
| dedup CLI | Go (same image) | Offline perceptual-hash backfill + pairs rescan |
| Reverse proxy | nginx (host, not shipped) | TLS termination, large-body/streaming config |
## Backend Architecture (Clean Architecture)
Dependencies point inward; no layer imports a layer above it.
```
handler → service → port (interfaces) ← db/postgres, storage, imagehash
domain (entities, value objects, errors) — stdlib only
```
- **domain** — entities and errors, zero internal imports.
- **port** — interfaces (repositories, `FileStorage`, `Transactor`).
- **service** — use cases; the only place business rules live.
- **handler** — Gin HTTP layer; maps domain errors to HTTP status codes.
- **db/postgres**, **storage**, **imagehash** — adapters implementing the ports.
Wiring is manual in `cmd/server/main.go` (no DI framework). See
[GO_PROJECT_STRUCTURE.md](GO_PROJECT_STRUCTURE.md) for the file-by-file layout,
the transaction/context patterns, and the DI sketch.
## Request Flow (typical authenticated call)
1. The SPA sends `Authorization: Bearer <access token>` to `/api/v1/...`.
2. Gin middleware runs: security headers → (for `/auth`) per-IP rate limiter →
auth middleware validates the JWT and puts `(userID, isAdmin, sessionID)`
into the request context.
3. The handler parses/validates input and calls a service method
(`ctx` first arg).
4. The service enforces ACL via `ACLService`, performs the use case — composing
repository calls inside a `Transactor.WithTx` when several writes must be
atomic — and writes an audit entry.
5. Repositories run SQL through pgx (pool or the tx carried in `ctx`).
6. The handler serializes the result; domain errors are mapped to
`{ code, message, details? }` with the right HTTP status.
## Cross-Cutting Concerns
### Authentication & sessions
JWT bearer auth. A short-lived **access token** (15 min default) authorizes API
calls; a long-lived **refresh token** (30 days default) rotates on use and is
stored as a hash in `activity.sessions`. A separate **content token** (6 h
default) is a single-file capability embedded in media URLs so long video keeps
streaming past access-token expiry. The `/auth` endpoints are rate-limited per
client IP.
### Authorization (ACL)
Private-by-default. Admins see everything; otherwise access requires a `public`
flag, creator ownership, or an explicit grant in `acl.permissions` (read / edit).
All checks are centralized in `ACLService` and applied before reads and writes.
### File storage
Originals are stored flat under `FILES_PATH`, each named by its file UUID (no
directory tree, no original-name collisions). Thumbnails and previews are a
**regenerable cache** under `THUMBS_CACHE_PATH`: still images via vipsthumbnail
(shrink-on-load) with a pure-Go `imaging` fallback, video frames via ffmpeg;
metadata/EXIF via exiftool with a pure-Go fallback. Uploads are rejected unless
their sniffed MIME type is whitelisted in `core.mime_types`.
### Near-duplicate detection
Images are dHash-ed (64-bit perceptual hash) inline on upload; video hashes are
backfilled by the dedup CLI. A rescan rebuilds `data.duplicate_pairs` using a
BK-tree over Hamming distance (within `DUPLICATE_HASH_THRESHOLD`), and the API
groups pairs into connected-component clusters. Dismissed pairs are remembered so
they stop resurfacing. See the duplicate sections in
[GO_PROJECT_STRUCTURE.md](GO_PROJECT_STRUCTURE.md).
### Audit logging
User-visible actions (file/tag/category/pool CRUD, relations, ACL changes,
auth, session termination, admin user actions) are recorded in
`activity.audit_log` against a seeded set of action types.
### Frontend / PWA
Pure client-side SPA: static assets served by the Go binary, with `index.html`
as the fallback for client routes. Installable PWA with a service worker for
app-shell caching and optional offline viewing of pinned files. See
[FRONTEND_STRUCTURE.md](FRONTEND_STRUCTURE.md).
## Data Model
PostgreSQL, four schemas (see `backend/migrations/`):
- **core** — users, MIME whitelist, object types.
- **data** — categories, tags, tag rules, files, filetag, pools, filepool,
duplicate pairs, duplicate dismissals.
- **acl** — per-object permission grants.
- **activity** — sessions, file/pool views, tag uses, audit log, action types.
Migrations are goose files embedded via `go:embed` and applied automatically on
server startup, so a fresh database bootstraps itself.
## Deployment
- **One image, one port.** The multi-stage `Dockerfile` builds the SPA (Node
stage) and the static Go binary (Go stage), then ships an Alpine runtime with
vips-tools / ffmpeg / exiftool and a non-root user. The server serves both the
API and the SPA on port **42776** (the sum of the code points of 七夕).
- **Compose.** `docker-compose.yml` runs the app plus, optionally, a bundled
PostgreSQL (`with-db` profile); a host Postgres is supported by leaving the
profile empty. The app is published on loopback only and expects a host
reverse proxy; the DB sits on a private `internal` network with no route
off-host. The dedup CLI is a `tools`-profile, run-on-demand service.
- **Config.** All runtime config is environment variables, fully documented in
[`.env.example`](../.env.example) (1:1 with `config.Config`). Secrets
(`JWT_SECRET`, `ADMIN_PASSWORD`, `DATABASE_URL`) are never baked into the image.
- **First run.** Migrations auto-apply and the initial admin is bootstrapped
from `ADMIN_USERNAME` / `ADMIN_PASSWORD`, so setup is: fill `.env`,
`docker compose up`.
See [DEPLOY.md](DEPLOY.md) for the production deploy (Gitea Actions → host) and
the reverse-proxy notes in [README.md](../README.md).
## Design Constraints & Future Direction
- **DDD / Clean Architecture** on the server keeps business rules independent of
Gin and pgx.
- **PostgreSQL-specific adapters are isolated** behind the `port` interfaces (the
filter DSL → SQL translation lives in `db/postgres`), leaving room for other
database engines in a future version without touching the service layer.
+6 -3
View File
@@ -3,13 +3,16 @@
Tanabata is deployed by a [Gitea Actions](https://docs.gitea.com/usage/actions/overview) Tanabata is deployed by a [Gitea Actions](https://docs.gitea.com/usage/actions/overview)
workflow ([`.gitea/workflows/deploy.yml`](../.gitea/workflows/deploy.yml)) that workflow ([`.gitea/workflows/deploy.yml`](../.gitea/workflows/deploy.yml)) that
runs on the **production host itself**. On every push to `master` it updates the runs on the **production host itself**. On every push to `master` it updates the
git clone in `/opt/tanabata` and runs `docker compose up -d --build` there, so the git clone in `/opt/tanabata`, runs the test suite (backend + frontend, in
image is built from the freshly-pushed code and the stack is restarted. throwaway toolchain containers), and — only if it passes — runs
`docker compose up -d --build` there, so the image is built from the
freshly-pushed code and the stack is restarted.
``` ```
push master ──> Gitea (container) ──> act_runner (host, "host" label) push master ──> Gitea (container) ──> act_runner (host, "host" label)
│ git fetch + reset --hard (in /opt/tanabata) │ git fetch + reset --hard (in /opt/tanabata)
└ docker compose up -d --build │ run tests (go + node in containers; ephemeral Postgres)
└ docker compose up -d --build (only if tests pass)
``` ```
The Gitea server runs in a container, but the **runner runs directly on the host** The Gitea server runs in a container, but the **runner runs directly on the host**
+160 -306
View File
@@ -1,13 +1,21 @@
# Tanabata File Manager — Frontend Structure # Tanabata File Manager — Frontend Structure
> Frontend counterpart of [ARCHITECTURE.md](ARCHITECTURE.md). This document
> details the SvelteKit layout, the CSS approach, and the API client.
## Stack ## Stack
- **Framework**: SvelteKit (SPA mode, `ssr: false`) - **Framework**: SvelteKit in **SPA mode** (`adapter-static`, `ssr = false`),
- **Language**: TypeScript Svelte 5 (runes)
- **CSS**: Tailwind CSS + CSS custom properties (hybrid) - **Language**: TypeScript (strict)
- **API types**: Auto-generated via openapi-typescript - **Build**: Vite 7
- **PWA**: Service worker + web manifest - **CSS**: Tailwind CSS **v4** via `@tailwindcss/vite` + CSS custom properties
(`@theme` in `app.css`) — no `tailwind.config.*` / `postcss.config.*` file
- **API types**: auto-generated via `openapi-typescript` (`src/lib/api/schema.ts`)
- **PWA**: service worker + web manifest
- **Font**: Epilogue (variable weight) - **Font**: Epilogue (variable weight)
- **Dev**: `vite-mock-plugin.ts` serves a mock API so the UI can run without the
Go backend
- **Package manager**: npm - **Package manager**: npm
## SPA mode — why SvelteKit without the server ## SPA mode — why SvelteKit without the server
@@ -19,7 +27,7 @@ static assets, and the only backend is the Go API. SvelteKit is used here
purely as an SPA framework: file-based routing, the client router, and build purely as an SPA framework: file-based routing, the client router, and build
tooling. tooling.
**SvelteKit features we *do* use:** **SvelteKit features we _do_ use:**
- File-based routing with nested layouts (`admin/` has its own guard) and - File-based routing with nested layouts (`admin/` has its own guard) and
dynamic segments (`[id]`). dynamic segments (`[id]`).
@@ -30,11 +38,11 @@ tooling.
over the still-mounted list via shallow routing, so the browser back button over the still-mounted list via shallow routing, so the browser back button
dismisses it without reloading the grid. This is the single biggest reason we dismisses it without reloading the grid. This is the single biggest reason we
stay on SvelteKit rather than a plain router. stay on SvelteKit rather than a plain router.
- `load` functions, used *only* as client-side route guards (auth redirect, - `load` functions, used _only_ as client-side route guards (auth redirect,
admin redirect, `/``/files`). admin redirect, `/``/files`).
- `$lib` alias, generated `./$types`, Vite/HMR integration. - `$lib` alias, generated `./$types`, Vite/HMR integration.
**SvelteKit features we deliberately do *not* use** (the "server half"): **SvelteKit features we deliberately do _not_ use** (the "server half"):
- SSR / hydration. - SSR / hydration.
- `+page.server.ts`, `+server.ts` endpoints, form actions — all data goes - `+page.server.ts`, `+server.ts` endpoints, form actions — all data goes
@@ -43,7 +51,7 @@ tooling.
`hooks.client.ts` files exist. `hooks.client.ts` files exist.
**Decision: stay on SvelteKit, do not migrate to a bare Svelte + router SPA.** **Decision: stay on SvelteKit, do not migrate to a bare Svelte + router SPA.**
The project already *is* an SPA, so there is no runtime gain from switching The project already _is_ an SPA, so there is no runtime gain from switching
(adapter-static tree-shakes the unused server bits; the client-runtime size (adapter-static tree-shakes the unused server bits; the client-runtime size
difference is negligible). A migration would mean re-implementing nested difference is negligible). A migration would mean re-implementing nested
layouts, guards, dynamic params, and — most painfully — shallow routing / layouts, guards, dynamic params, and — most painfully — shallow routing /
@@ -55,15 +63,7 @@ expect SSR, endpoints, or hooks to do anything here; that is intentional.
``` ```
tanabata/ tanabata/
├── backend/ ← Go project (go.mod in here) ├── backend/ ← Go project (go.mod in here)
│ ├── cmd/
│ ├── internal/
│ ├── migrations/
│ ├── go.mod
│ └── go.sum
├── frontend/ ← SvelteKit project (package.json in here) ├── frontend/ ← SvelteKit project (package.json in here)
│ └── (see below)
├── openapi.yaml ← Shared API contract (root level) ├── openapi.yaml ← Shared API contract (root level)
├── docker-compose.yml ├── docker-compose.yml
├── Dockerfile ├── Dockerfile
@@ -71,306 +71,173 @@ tanabata/
└── README.md └── README.md
``` ```
`openapi.yaml` lives at repository root — both backend and frontend `openapi.yaml` lives at repository root — both backend and frontend reference
reference it. The frontend generates types from it; the backend it. The frontend generates types from it; the backend implements it.
validates its handlers against it.
## Frontend Directory Layout ## Frontend Directory Layout
``` ```
frontend/ frontend/
├── package.json ├── package.json
├── svelte.config.js ├── svelte.config.js # adapter-static, fallback: index.html
├── vite.config.ts ├── vite.config.ts # plugins: tailwindcss(), sveltekit(), mockApiPlugin()
├── vite-mock-plugin.ts # dev-only mock API (run the UI without the Go backend)
├── tsconfig.json ├── tsconfig.json
├── tailwind.config.ts
├── postcss.config.js
├── src/ ├── static/ # Copied verbatim into the build
│ ├── app.html # Shell HTML (PWA meta, font preload) │ ├── manifest.webmanifest # PWA manifest
│ ├── app.css # Tailwind directives + CSS custom properties │ ├── browserconfig.xml
│ # (no hooks.* — see "SPA mode" above) ├── robots.txt
├── favicon.ico
│ ├── lib/ # Shared code ($lib/ alias) │ ├── fonts/
│ │ │ │ └── Epilogue-VariableFont_wght.ttf
│ ├── api/ # API client layer └── images/ # PWA icons, section icons (svg), login decorations
│ │ │ ├── client.ts # Base fetch wrapper: auth headers, token refresh,
│ │ │ │ # error parsing, base URL
│ │ │ ├── files.ts # listFiles, getFile, uploadFile, deleteFile, etc.
│ │ │ ├── tags.ts # listTags, createTag, getTag, updateTag, etc.
│ │ │ ├── categories.ts # Category API functions
│ │ │ ├── pools.ts # Pool API functions
│ │ │ ├── auth.ts # login, logout, refresh, listSessions
│ │ │ ├── acl.ts # getPermissions, setPermissions
│ │ │ ├── users.ts # getMe, updateMe, admin user CRUD
│ │ │ ├── audit.ts # queryAuditLog
│ │ │ ├── schema.ts # AUTO-GENERATED from openapi.yaml (do not edit)
│ │ │ └── types.ts # Friendly type aliases:
│ │ │ # export type File = components["schemas"]["File"]
│ │ │ # export type Tag = components["schemas"]["Tag"]
│ │ │
│ │ ├── components/ # Reusable UI components
│ │ │ │
│ │ │ ├── layout/ # App shell
│ │ │ │ ├── Navbar.svelte # Bottom navigation bar (mobile-first)
│ │ │ │ ├── Header.svelte # Section header with sorting controls
│ │ │ │ ├── SelectionBar.svelte # Floating bar for multi-select actions
│ │ │ │ └── Loader.svelte # Full-screen loading overlay
│ │ │ │
│ │ │ ├── file/ # File-related components
│ │ │ │ ├── FileGrid.svelte # Thumbnail grid with infinite scroll
│ │ │ │ ├── FileCard.svelte # Single thumbnail (160×160, selectable)
│ │ │ │ ├── FileViewer.svelte # Full-screen preview with prev/next navigation
│ │ │ │ ├── FileUpload.svelte # Upload form + drag-and-drop zone
│ │ │ │ ├── FileDetail.svelte # Metadata editor (notes, datetime, tags)
│ │ │ │ └── FilterBar.svelte # DSL filter builder UI
│ │ │ │
│ │ │ ├── tag/ # Tag-related components
│ │ │ │ ├── TagBadge.svelte # Colored pill with tag name
│ │ │ │ ├── TagPicker.svelte # Searchable tag selector (add/remove)
│ │ │ │ ├── TagList.svelte # Tag grid for section view
│ │ │ │ └── TagRuleEditor.svelte # Auto-tag rule management
│ │ │ │
│ │ │ ├── pool/ # Pool-related components
│ │ │ │ ├── PoolCard.svelte # Pool preview card
│ │ │ │ ├── PoolFileList.svelte # Ordered file list with drag reorder
│ │ │ │ └── PoolDetail.svelte # Pool metadata editor
│ │ │ │
│ │ │ ├── acl/ # Access control components
│ │ │ │ └── PermissionEditor.svelte # User permission grid
│ │ │ │
│ │ │ └── common/ # Shared primitives
│ │ │ ├── Button.svelte
│ │ │ ├── Modal.svelte
│ │ │ ├── ConfirmDialog.svelte
│ │ │ ├── Toast.svelte
│ │ │ ├── InfiniteScroll.svelte
│ │ │ ├── Pagination.svelte
│ │ │ ├── SortDropdown.svelte
│ │ │ ├── SearchInput.svelte
│ │ │ ├── ColorPicker.svelte
│ │ │ ├── Checkbox.svelte # Three-state: checked, unchecked, partial
│ │ │ └── EmptyState.svelte
│ │ │
│ │ ├── stores/ # Svelte stores (global state)
│ │ │ ├── auth.ts # Current user, JWT tokens, isAuthenticated
│ │ │ ├── selection.ts # Selected item IDs, selection mode toggle
│ │ │ ├── sorting.ts # Per-section sort key + order (persisted to localStorage)
│ │ │ ├── theme.ts # Dark/light mode (persisted, respects prefers-color-scheme)
│ │ │ └── toast.ts # Notification queue (success, error, info)
│ │ │
│ │ └── utils/ # Pure helper functions
│ │ ├── format.ts # formatDate, formatFileSize, formatDuration
│ │ ├── dsl.ts # Filter DSL builder: UI state → query string
│ │ ├── pwa.ts # PWA reset, cache clear, update prompt
│ │ └── keyboard.ts # Keyboard shortcut helpers (Ctrl+A, Escape, etc.)
│ │
│ ├── routes/ # SvelteKit file-based routing
│ │ │
│ │ ├── +layout.svelte # Root layout: Navbar, theme wrapper, toast container
│ │ ├── +layout.ts # Root load: auth guard → redirect to /login if no token
│ │ │
│ │ ├── +page.svelte # / → redirect to /files
│ │ │
│ │ ├── login/
│ │ │ └── +page.svelte # Login form (decorative Tanabata images)
│ │ │
│ │ ├── files/
│ │ │ ├── +page.svelte # File grid: filter bar, sort, multi-select, upload
│ │ │ ├── +page.ts # Load: initial file list (cursor page)
│ │ │ ├── [id]/
│ │ │ │ ├── +page.svelte # File view: preview, metadata, tags, ACL
│ │ │ │ └── +page.ts # Load: file detail + tags
│ │ │ └── trash/
│ │ │ ├── +page.svelte # Trash: restore / permanent delete
│ │ │ └── +page.ts
│ │ │
│ │ ├── tags/
│ │ │ ├── +page.svelte # Tag list: search, sort, multi-select
│ │ │ ├── +page.ts
│ │ │ ├── new/
│ │ │ │ └── +page.svelte # Create tag form
│ │ │ └── [id]/
│ │ │ ├── +page.svelte # Tag detail: edit, category, rules, parent tags
│ │ │ └── +page.ts
│ │ │
│ │ ├── categories/
│ │ │ ├── +page.svelte # Category list
│ │ │ ├── +page.ts
│ │ │ ├── new/
│ │ │ │ └── +page.svelte
│ │ │ └── [id]/
│ │ │ ├── +page.svelte # Category detail: edit, view tags
│ │ │ └── +page.ts
│ │ │
│ │ ├── pools/
│ │ │ ├── +page.svelte # Pool list
│ │ │ ├── +page.ts
│ │ │ ├── new/
│ │ │ │ └── +page.svelte
│ │ │ └── [id]/
│ │ │ ├── +page.svelte # Pool detail: files (reorderable), filter, edit
│ │ │ └── +page.ts
│ │ │
│ │ ├── settings/
│ │ │ ├── +page.svelte # Profile: name, password, active sessions
│ │ │ └── +page.ts
│ │ │
│ │ └── admin/
│ │ ├── +layout.svelte # Admin layout: restrict to is_admin
│ │ ├── users/
│ │ │ ├── +page.svelte # User management list
│ │ │ ├── +page.ts
│ │ │ └── [id]/
│ │ │ ├── +page.svelte # User detail: role, block/unblock
│ │ │ └── +page.ts
│ │ └── audit/
│ │ ├── +page.svelte # Audit log with filters
│ │ └── +page.ts
│ │
│ └── service-worker.ts # PWA: offline cache for pinned files, app shell caching
└── static/ └── src/
├── favicon.png ├── app.html # Shell HTML (PWA meta, font preload)
├── favicon.ico ├── app.css # `@import 'tailwindcss'` + `@theme` custom properties
├── manifest.webmanifest # PWA manifest (name, icons, theme_color) ├── app.d.ts # Ambient types
├── images/ ├── service-worker.ts # PWA: app-shell + pinned-file offline cache
├── tanabata-left.png # Login page decorations (from current design)
│ ├── tanabata-right.png ├── lib/ # Shared code ($lib alias)
── icons/ # PWA icons (192×192, 512×512, etc.) ── index.ts
└── fonts/ │ │
── Epilogue-VariableFont_wght.ttf ── api/ # API client layer
│ │ ├── client.ts # fetch wrapper: bearer auth, 401 refresh+retry, error parsing,
│ │ │ # upload-with-progress (XHR), NDJSON streaming; exports `api`
│ │ ├── auth.ts # login, refresh, logout, sessions
│ │ ├── tags.ts # tag + tag-rule calls
│ │ ├── categories.ts # category calls
│ │ ├── duplicates.ts # duplicate list / dismiss / resolve
│ │ ├── schema.ts # AUTO-GENERATED from openapi.yaml (gitignored; do not edit)
│ │ └── types.ts # Friendly aliases: components['schemas'][...]
│ │
│ ├── components/
│ │ ├── layout/
│ │ │ ├── Header.svelte # Section header with sorting controls
│ │ │ ├── SelectionBar.svelte # Floating bar for multi-select actions
│ │ │ └── KeyboardHelp.svelte # Keyboard-shortcut overlay
│ │ │
│ │ ├── file/
│ │ │ ├── FileCard.svelte # Single thumbnail (160×160, selectable)
│ │ │ ├── Thumb.svelte # Lazy-loaded thumbnail (IntersectionObserver)
│ │ │ ├── FileViewer.svelte # Full-screen viewer with prev/next
│ │ │ ├── FileUpload.svelte # Upload form + drag-and-drop
│ │ │ ├── FilterBar.svelte # DSL filter builder UI
│ │ │ ├── MetadataEditor.svelte # Notes / datetime / metadata (nested) editor
│ │ │ ├── TagPicker.svelte # Searchable tag selector (add/remove)
│ │ │ ├── PoolPicker.svelte # Add-to-pool dialog
│ │ │ ├── BulkTagEditor.svelte # Multi-select tag add/remove
│ │ │ └── DuplicateMergeDialog.svelte # Field-by-field duplicate resolution
│ │ │
│ │ ├── tag/
│ │ │ ├── TagBadge.svelte # Colored pill
│ │ │ └── TagRuleEditor.svelte # Auto-tag rule management
│ │ │
│ │ └── common/
│ │ ├── ConfirmDialog.svelte
│ │ └── InfiniteScroll.svelte # Below-the-fold lazy loading on scroll
│ │
│ ├── stores/ # Svelte stores (global state)
│ │ ├── auth.ts # Current user + JWT tokens (persisted, cross-tab sync)
│ │ ├── selection.ts # Selected item IDs, selection mode
│ │ ├── sorting.ts # Per-section sort key + order (persisted)
│ │ ├── theme.ts # Dark/light theme (persisted)
│ │ ├── appSettings.ts # Misc client-side settings
│ │ ├── listScroll.ts # Restore list scroll position after overlay/back
│ │ └── sectionCache.ts # Cached list snapshots, invalidated on mutation
│ │
│ └── utils/ # Pure helpers
│ ├── dsl.ts # Filter DSL builder: UI state → query string
│ ├── metadata.ts # Nested metadata <-> editor rows
│ ├── pwa.ts # PWA reset / update prompt
│ └── rovingGrid.svelte.ts # Roving-tabindex keyboard grid navigation
└── routes/ # SvelteKit file-based routing (guards only in load)
├── +layout.svelte # Root layout: nav, theme
├── +layout.ts # ssr=false; root auth guard
├── +page.svelte / +page.ts # / → redirect to /files
├── login/+page.svelte
├── files/
│ ├── +page.svelte / +page.ts # Grid: filter, sort, multi-select, upload
│ ├── [id]/+page.svelte # File view (also opened as shallow-routing overlay)
│ ├── duplicates/+page.svelte # Duplicate clusters
│ └── trash/+page.svelte # Trash: restore / permanent delete
├── tags/ { +page.svelte, new/, [id]/ }
├── categories/ { +page.svelte, new/, [id]/ }
├── pools/ { +page.svelte, new/, [id]/ }
├── settings/+page.svelte # Profile: name, password, sessions, import path
└── admin/
├── +layout.svelte / +layout.ts # Restrict to admins
├── users/{ +page.svelte, [id]/ }
└── audit/+page.svelte
``` ```
## Key Architecture Decisions ## Key Architecture Decisions
### CSS Hybrid: Tailwind + Custom Properties ### CSS: Tailwind v4 + Custom Properties
Theme colors defined as CSS custom properties in `app.css`: Tailwind v4 is configured **in CSS**, not in a JS config file. `app.css` imports
Tailwind and declares the theme tokens as CSS custom properties inside `@theme`;
Tailwind then generates utilities (`bg-bg-primary`, `text-text-primary`,
`font-sans`, …) from those tokens automatically.
```css ```css
@tailwind base; /* src/app.css */
@tailwind components; @import "tailwindcss";
@tailwind utilities;
:root { @theme {
--color-bg-primary: #312F45; --color-bg-primary: #312f45;
--color-bg-secondary: #181721; --color-bg-secondary: #181721;
--color-bg-elevated: #111118; --color-bg-elevated: #111118;
--color-accent: #9592B5; --color-accent: #9592b5;
--color-accent-hover: #7D7AA4; --color-accent-hover: #7d7aa4;
--color-text-primary: #f0f0f0; --color-text-primary: #f0f0f0;
--color-text-muted: #9999AD;
--color-danger: #DB6060;
--color-info: #4DC7ED;
--color-warning: #F5E872;
--color-tag-default: #444455; --color-tag-default: #444455;
} /* … info / danger / warning / success / nav tokens … */
:root[data-theme="light"] { --font-sans: "Epilogue", sans-serif;
--color-bg-primary: #f5f5f5;
--color-bg-secondary: #ffffff;
/* ... */
} }
``` ```
Tailwind references them in `tailwind.config.ts`: Dark theme is primary; the light theme overrides the same custom properties.
```ts
export default {
theme: {
extend: {
colors: {
bg: {
primary: 'var(--color-bg-primary)',
secondary: 'var(--color-bg-secondary)',
elevated: 'var(--color-bg-elevated)',
},
accent: {
DEFAULT: 'var(--color-accent)',
hover: 'var(--color-accent-hover)',
},
// ...
},
fontFamily: {
sans: ['Epilogue', 'sans-serif'],
},
},
},
darkMode: 'class', // controlled via data-theme attribute
};
```
Usage in components: `<div class="bg-bg-primary text-text-primary rounded-xl p-4">`. Usage in components: `<div class="bg-bg-primary text-text-primary rounded-xl p-4">`.
Complex cases use scoped `<style>` inside `.svelte` files. Complex cases use scoped `<style>` inside `.svelte` files.
### API Client Pattern ### API Client Pattern
`client.ts` thin wrapper around fetch: `src/lib/api/client.ts` is a thin fetch wrapper exporting a generic `api`
object. It attaches the bearer token, transparently handles a single `401`
refresh-and-retry (deduplicating concurrent refreshes and syncing rotated
tokens across tabs), parses `{ code, message, details }` errors into `ApiError`,
and invalidates cached list snapshots on mutation. It also provides
`uploadWithProgress` (XHR, for upload progress) and `postStream` (NDJSON, for
the live import progress).
```ts ```ts
// $lib/api/client.ts // $lib/api/client.ts (shape)
import { authStore } from '$lib/stores/auth';
const BASE = '/api/v1';
async function request<T>(path: string, init?: RequestInit): Promise<T> {
const token = get(authStore).accessToken;
const res = await fetch(BASE + path, {
...init,
headers: {
'Content-Type': 'application/json',
...(token && { Authorization: `Bearer ${token}` }),
...init?.headers,
},
});
if (res.status === 401) {
// attempt refresh, retry once
}
if (!res.ok) {
const err = await res.json();
throw new ApiError(res.status, err.code, err.message, err.details);
}
if (res.status === 204) return undefined as T;
return res.json();
}
export const api = { export const api = {
get: <T>(path: string) => request<T>(path), get: <T>(path) => request<T>(path),
post: <T>(path: string, body?: unknown) => post: <T>(path, body?) =>
request<T>(path, { method: 'POST', body: JSON.stringify(body) }), request<T>(path, { method: "POST", body: JSON.stringify(body) }),
patch: <T>(path: string, body?: unknown) => patch: <T>(path, body?) =>
request<T>(path, { method: 'PATCH', body: JSON.stringify(body) }), request<T>(path, { method: "PATCH", body: JSON.stringify(body) }),
put: <T>(path: string, body?: unknown) => put: <T>(path, body?) =>
request<T>(path, { method: 'PUT', body: JSON.stringify(body) }), request<T>(path, { method: "PUT", body: JSON.stringify(body) }),
delete: <T>(path: string) => request<T>(path, { method: 'DELETE' }), delete: <T>(path) => request<T>(path, { method: "DELETE" }),
upload: <T>(path: string, formData: FormData) => upload: <T>(path, fd) => request<T>(path, { method: "POST", body: fd }),
request<T>(path, { method: 'POST', body: formData, headers: {} }),
}; };
``` ```
Domain-specific modules use it: Resource modules (`auth.ts`, `tags.ts`, `categories.ts`, `duplicates.ts`) wrap
`api` with typed helpers. Endpoints without a dedicated module (files, pools,
```ts users, acl, audit) are called through `api.*` directly from their route
// $lib/api/files.ts components.
import { api } from './client';
import type { File, FileCursorPage } from './types';
export function listFiles(params: Record<string, string>) {
const qs = new URLSearchParams(params).toString();
return api.get<FileCursorPage>(`/files?${qs}`);
}
export function uploadFile(formData: FormData) {
return api.upload<File>('/files', formData);
}
```
### Type Generation ### Type Generation
Script in `package.json`:
```json ```json
// package.json
{ {
"scripts": { "scripts": {
"generate:types": "openapi-typescript ../openapi.yaml -o src/lib/api/schema.ts", "generate:types": "openapi-typescript ../openapi.yaml -o src/lib/api/schema.ts",
@@ -380,43 +247,30 @@ Script in `package.json`:
} }
``` ```
Friendly aliases in `types.ts`: `schema.ts` is generated (and gitignored) — never edit it by hand. `types.ts`
re-exports friendly aliases:
```ts ```ts
import type { components } from './schema'; import type { components } from "./schema";
export type File = components["schemas"]["File"];
export type File = components['schemas']['File']; export type Tag = components["schemas"]["Tag"];
export type Tag = components['schemas']['Tag']; // …
export type Category = components['schemas']['Category'];
export type Pool = components['schemas']['Pool'];
export type FileCursorPage = components['schemas']['FileCursorPage'];
export type TagOffsetPage = components['schemas']['TagOffsetPage'];
export type Error = components['schemas']['Error'];
// ...
``` ```
### SPA Mode ### SPA Mode (build)
`svelte.config.js`:
```js ```js
import adapter from '@sveltejs/adapter-static'; // svelte.config.js
import adapter from "@sveltejs/adapter-static";
export default { export default { kit: { adapter: adapter({ fallback: "index.html" }) } };
kit: {
adapter: adapter({ fallback: 'index.html' }),
// SPA: all routes handled client-side
},
};
``` ```
The Go backend serves `index.html` for all non-API routes (SPA fallback). The Go backend serves `index.html` for all non-API routes (SPA fallback, see
In development, Vite dev server proxies `/api` to the Go backend. `handler/static.go`). In development the Vite dev server serves the UI and the
mock plugin (or a proxied Go backend) answers `/api`.
### PWA ### PWA
`service-worker.ts` handles: `service-worker.ts` handles app-shell caching (HTML/CSS/JS/fonts) and optional
- App shell caching (HTML, CSS, JS, fonts) user-pinned file caching for offline viewing; `utils/pwa.ts` exposes the reset /
- User-pinned file caching (explicit, via UI button) update flow (clear caches and reload from the server, keeping pinned files).
- Cache versioning and cleanup on update
- Reset function (clear all caches except pinned files)
+156 -144
View File
@@ -1,18 +1,28 @@
# Tanabata File Manager — Go Project Structure # Tanabata File Manager — Go Project Structure
> Backend counterpart of [ARCHITECTURE.md](ARCHITECTURE.md). This document
> details the Go layout, the layer rules, and the key backend decisions.
## Stack ## Stack
- **Router**: Gin - **Router**: Gin
- **Database**: pgx v5 (pgxpool) - **Database**: pgx v5 (pgxpool)
- **Migrations**: goose v3 + go:embed (auto-migrate on startup) - **Migrations**: goose v3 + `go:embed` (auto-applied on startup)
- **Auth**: JWT (golang-jwt/jwt/v5) - **Auth**: JWT (golang-jwt/jwt/v5), Bearer access tokens + rotating refresh tokens
- **Config**: environment variables via .env (joho/godotenv) - **Config**: environment variables via `.env` (joho/godotenv)
- **Logging**: slog (stdlib, Go 1.21+) - **Logging**: slog (stdlib)
- **Validation**: go-playground/validator/v10 - **Metadata**: exiftool (external, preferred) with a pure-Go EXIF fallback
- **EXIF**: rwcarlsen/goexif or dsoprea/go-exif (rwcarlsen/goexif)
- **Image processing**: disintegration/imaging (thumbnails, previews) - **Thumbnails / previews**: vipsthumbnail (external, shrink-on-load) and ffmpeg
(video frames), with a pure-Go fallback (disintegration/imaging)
- **Near-duplicate detection**: 64-bit dHash perceptual hashing + a BK-tree /
Hamming-distance pairing (`internal/imagehash`, `internal/service/duplicate_*`)
- **Architecture**: Clean Architecture (domain → service → repository/handler) - **Architecture**: Clean Architecture (domain → service → repository/handler)
The binary is fully static (`CGO_ENABLED=0`). External tools are invoked as
subprocesses when present and are optional — the pure-Go paths keep the server
working without them.
## Monorepo Layout ## Monorepo Layout
``` ```
@@ -31,81 +41,98 @@ tanabata/
``` ```
backend/ backend/
├── cmd/ ├── cmd/
── server/ ── server/
└── main.go # Entrypoint: config → DB → migrate → wire → run └── main.go # Entrypoint: config → DB → migrate → bootstrap admin → wire → serve
│ └── dedup/
│ └── main.go # Offline maintenance CLI: perceptual-hash backfill + duplicate-pairs rescan
├── internal/ ├── internal/
│ │ │ │
│ ├── domain/ # Pure business entities & value objects │ ├── domain/ # Pure business entities & value objects (stdlib only)
│ │ ├── file.go # File, FileFilter, FilePage │ │ ├── file.go # File, FileFilter, FileListParams, FilePage
│ │ ├── tag.go # Tag, TagRule │ │ ├── tag.go # Tag, TagRule
│ │ ├── category.go # Category │ │ ├── category.go # Category
│ │ ├── pool.go # Pool, PoolFile │ │ ├── pool.go # Pool, PoolFile
│ │ ├── user.go # User, Session │ │ ├── user.go # User, Session
│ │ ├── acl.go # Permission, ObjectType │ │ ├── acl.go # Permission, ObjectType
│ │ ├── audit.go # AuditEntry, ActionType │ │ ├── audit.go # AuditEntry, ActionType
│ │ ── errors.go # Domain error types (ErrNotFound, ErrForbidden, etc.) │ │ ── duplicate.go # DuplicatePair, PHashEntry
│ │ ├── context.go # WithUser / UserFromContext (identity + session in ctx)
│ │ └── errors.go # Domain error types (ErrNotFound, ErrForbidden, …)
│ │ │ │
│ ├── port/ # Interfaces (ports) — contracts between layers │ ├── port/ # Interfaces (ports) — contracts between layers
│ │ ├── repository.go # FileRepo, TagRepo, CategoryRepo, PoolRepo, │ │ ├── repository.go # Transactor, FileRepo, TagRepo, TagRuleRepo, CategoryRepo,
│ │ │ # UserRepo, SessionRepo, ACLRepo, AuditRepo, │ │ │ # PoolRepo, UserRepo, SessionRepo, ACLRepo, AuditRepo,
│ │ │ # MimeRepo, TagRuleRepo │ │ │ # MimeRepo, DuplicatePairRepo, DismissalRepo
│ │ └── storage.go # FileStorage interface (disk operations) │ │ └── storage.go # FileStorage (originals + thumbnail/preview cache)
│ │ │ │
│ ├── service/ # Business logic (use cases) │ ├── service/ # Business logic (use cases)
│ │ ├── file_service.go # Upload, update, delete, trash/restore, replace, │ │ ├── file_service.go # Upload, update, delete, trash/restore, replace, import, filter/list
│ │ │ # import, filter/list, duplicate detection │ │ ├── tag_service.go # CRUD + auto-tag (rule) application
│ │ ├── tag_service.go # CRUD + auto-tag application logic │ │ ├── category_service.go # CRUD (thin: repo + ACL + audit)
│ │ ├── category_service.go # CRUD (thin, delegates to repo + ACL + audit)
│ │ ├── pool_service.go # CRUD + file ordering, add/remove files │ │ ├── pool_service.go # CRUD + file ordering, add/remove files
│ │ ├── auth_service.go # Login, logout, JWT issue/refresh, session management │ │ ├── auth_service.go # Login, logout, JWT issue/refresh, content tokens, sessions
│ │ ├── acl_service.go # Permission checks, grant/revoke │ │ ├── acl_service.go # Permission checks, grant/revoke
│ │ ├── audit_service.go # Log actions, query audit log │ │ ├── audit_service.go # Log actions, query audit log
│ │ ── user_service.go # Profile update, admin CRUD, block/unblock │ │ ── user_service.go # Profile update, admin CRUD, block/unblock, EnsureAdmin
│ │ ├── duplicate_service.go # Cluster / resolve (merge) / dismiss + rescan orchestration
│ │ ├── duplicate_index.go # BK-tree, Hamming pairing, connected-component clustering
│ │ └── metadata.go # EXIF / media metadata extraction (exiftool + pure-Go fallback)
│ │ │ │
│ ├── handler/ # HTTP layer (Gin handlers) │ ├── handler/ # HTTP layer (Gin handlers)
│ │ ├── router.go # Route registration, middleware wiring │ │ ├── router.go # Route registration, middleware, security headers, SPA fallback
│ │ ├── middleware.go # Auth middleware (JWT extraction → context) │ │ ├── middleware.go # Auth middleware (JWT / content token → context)
│ │ ├── request.go # Common request parsing helpers │ │ ├── ratelimit.go # Per-IP token-bucket limiter for /auth
│ │ ├── response.go # Error/success response builders │ │ ├── response.go # Error/success builders, domain-error → HTTP mapping
│ │ ├── static.go # Built SPA serving + index.html fallback
│ │ ├── file_handler.go # /files endpoints │ │ ├── file_handler.go # /files endpoints
│ │ ├── tag_handler.go # /tags endpoints │ │ ├── duplicate_handler.go # /files/duplicates endpoints
│ │ ├── tag_handler.go # /tags endpoints (+ filetag relations)
│ │ ├── category_handler.go # /categories endpoints │ │ ├── category_handler.go # /categories endpoints
│ │ ├── pool_handler.go # /pools endpoints │ │ ├── pool_handler.go # /pools endpoints
│ │ ├── auth_handler.go # /auth endpoints │ │ ├── auth_handler.go # /auth endpoints
│ │ ├── acl_handler.go # /acl endpoints │ │ ├── acl_handler.go # /acl endpoints
│ │ ├── user_handler.go # /users endpoints │ │ ├── user_handler.go # /users endpoints
│ │ └── audit_handler.go # /audit endpoints │ │ └── audit_handler.go # /audit endpoint
│ │ │ │
│ ├── db/ # Database adapters │ ├── db/ # Database adapters
│ │ ├── db.go # Common helpers: pagination, repo factory, transactor base │ │ ├── db.go # Shared helpers: Querier, tx-from-context, ScanRow, limit/offset clamps
│ │ └── postgres/ # PostgreSQL implementation │ │ └── postgres/ # PostgreSQL implementation
│ │ ├── postgres.go # pgxpool init, tx-from-context helpers │ │ ├── postgres.go # pgxpool init, Transactor, conn-or-tx helper
│ │ ├── file_repo.go # FileRepo implementation │ │ ├── file_repo.go # FileRepo (incl. perceptual-hash projections)
│ │ ├── tag_repo.go # TagRepo + TagRuleRepo implementation │ │ ├── tag_repo.go # TagRepo + TagRuleRepo
│ │ ├── category_repo.go # CategoryRepo implementation │ │ ├── category_repo.go # CategoryRepo
│ │ ├── pool_repo.go # PoolRepo implementation │ │ ├── pool_repo.go # PoolRepo
│ │ ├── user_repo.go # UserRepo implementation │ │ ├── user_repo.go # UserRepo
│ │ ├── session_repo.go # SessionRepo implementation │ │ ├── session_repo.go # SessionRepo
│ │ ├── acl_repo.go # ACLRepo implementation │ │ ├── acl_repo.go # ACLRepo
│ │ ├── audit_repo.go # AuditRepo implementation │ │ ├── audit_repo.go # AuditRepo
│ │ ├── mime_repo.go # MimeRepo implementation │ │ ├── mime_repo.go # MimeRepo
│ │ ── filter_parser.go # DSL → SQL WHERE clause builder │ │ ── duplicate_repo.go # DuplicatePairRepo + DismissalRepo
│ │ └── filter_parser.go # Filter DSL → SQL WHERE clause builder
│ │ │ │
│ ├── storage/ # File storage adapter │ ├── storage/ # File storage adapter
│ │ └── disk.go # FileStorage implementation (read/write/delete on disk) │ │ └── disk.go # FileStorage on disk: originals + thumbnail/preview cache
│ │ # (vipsthumbnail / ffmpeg / pure-Go imaging)
│ │
│ ├── imagehash/ # Perceptual hashing (64-bit dHash) for near-duplicate detection
│ │ └── imagehash.go
│ │
│ ├── integration/ # End-to-end HTTP tests against a disposable Postgres
│ │ └── server_test.go
│ │ │ │
│ └── config/ # Configuration │ └── config/ # Configuration
│ └── config.go # Struct + loader from env vars │ └── config.go # Config struct + loader from env vars
├── migrations/ # SQL migration files (goose format) ├── migrations/ # SQL migration files (goose format), embedded via go:embed
│ ├── 001_init_schemas.sql │ ├── 001_init_schemas.sql
│ ├── 002_core_tables.sql │ ├── 002_core_tables.sql
│ ├── 003_data_tables.sql │ ├── 003_data_tables.sql
│ ├── 004_acl_tables.sql │ ├── 004_acl_tables.sql
│ ├── 005_activity_tables.sql │ ├── 005_activity_tables.sql
│ ├── 006_indexes.sql │ ├── 006_indexes.sql
── 007_seed_data.sql ── 007_seed_data.sql
│ └── embed.go # //go:embed *.sql → migrations.FS
├── go.mod ├── go.mod
└── go.sum └── go.sum
@@ -126,6 +153,7 @@ handler → service → port (interfaces) ← db/postgres / storage
- **db/postgres/**: imports domain/, port/, and db/ (common helpers). Implements port interfaces. - **db/postgres/**: imports domain/, port/, and db/ (common helpers). Implements port interfaces.
- **db/**: imports domain/ and port/. Shared utilities for all DB adapters. - **db/**: imports domain/ and port/. Shared utilities for all DB adapters.
- **storage/**: imports domain/ and port/. Implements FileStorage. - **storage/**: imports domain/ and port/. Implements FileStorage.
- **imagehash/**: leaf package (stdlib + image libs); used by service/ and storage/.
No layer may import a layer above it. No circular dependencies. No layer may import a layer above it. No circular dependencies.
@@ -133,104 +161,102 @@ No layer may import a layer above it. No circular dependencies.
### Dependency Injection (Wiring) ### Dependency Injection (Wiring)
Manual wiring in `cmd/server/main.go`. No DI frameworks. Manual wiring in `cmd/server/main.go`. No DI frameworks. Constructors take their
collaborators explicitly; the shape below matches the real signatures.
```go ```go
// Pseudocode // Pseudocode — see cmd/server/main.go for the exact calls.
pool := postgres.NewPool(cfg.DatabaseURL) pool := postgres.NewPool(ctx, cfg.DatabaseURL)
goose.Up(pool, migrations) goose.Up(stdlib.OpenDBFromPool(pool), ".") // migrations.FS embedded
// Storage
diskStorage := storage.NewDiskStorage(
cfg.FilesPath, cfg.ThumbsCachePath,
cfg.ThumbWidth, cfg.ThumbHeight, cfg.PreviewWidth, cfg.PreviewHeight,
cfg.ThumbMaxPixels, cfg.ThumbConcurrency,
)
// Repos (all from internal/db/postgres/) // Repos (all from internal/db/postgres/)
fileRepo := postgres.NewFileRepo(pool) fileRepo := postgres.NewFileRepo(pool)
tagRepo := postgres.NewTagRepo(pool) // … tag, tagRule, category, pool, user, session, acl, audit, mime,
// ... // duplicatePair, dismissal repos + transactor
// Storage
diskStore := storage.NewDiskStorage(cfg.FilesPath)
// Services // Services
aclSvc := service.NewACLService(aclRepo, objectTypeRepo) authSvc := service.NewAuthService(userRepo, sessionRepo,
auditSvc := service.NewAuditService(auditRepo, actionTypeRepo) cfg.JWTSecret, cfg.JWTAccessTTL, cfg.JWTRefreshTTL, cfg.ContentTokenTTL)
fileSvc := service.NewFileService(fileRepo, mimeRepo, tagRepo, diskStore, aclSvc, auditSvc) aclSvc := service.NewACLService(aclRepo, fileRepo, tagRepo, categoryRepo, poolRepo, transactor)
tagSvc := service.NewTagService(tagRepo, tagRuleRepo, aclSvc, auditSvc) auditSvc := service.NewAuditService(auditRepo)
// ... tagSvc := service.NewTagService(tagRepo, tagRuleRepo, aclSvc, auditSvc, transactor)
dupSvc := service.NewDuplicateService(fileRepo, duplicatePairRepo, dismissalRepo,
aclSvc, auditSvc, transactor, cfg.DuplicateHashThreshold)
fileSvc := service.NewFileService(fileRepo, mimeRepo, diskStorage,
aclSvc, auditSvc, tagSvc, transactor, cfg.ImportPath)
// … category, pool, user services
// Handlers // Bootstrap the initial admin from env (idempotent).
fileHandler := handler.NewFileHandler(fileSvc, tagSvc) userSvc.EnsureAdmin(ctx, cfg.AdminUsername, cfg.AdminPassword)
// ...
router := handler.NewRouter(cfg, fileHandler, tagHandler, ...) // Handlers → router (also wires trusted proxies + optional static SPA dir)
router.Run(cfg.ListenAddr) router, _ := handler.NewRouter(authMiddleware, authHandler, fileHandler,
duplicateHandler, tagHandler, categoryHandler, poolHandler,
userHandler, aclHandler, auditHandler, cfg.StaticDir, cfg.TrustedProxies)
srv.ListenAndServe()
``` ```
### Context Propagation ### Context Propagation
Every service method receives `context.Context` as the first argument. Every service method receives `context.Context` as the first argument.
The handler extracts user info from JWT (via middleware) and puts it The auth middleware parses the JWT and puts the caller's identity (user id,
into context. Services read the current user from context for ACL checks admin flag, session id) into the context. Services read it for ACL checks and
and audit logging. audit logging.
```go ```go
// middleware.go // handler/middleware.go
func (m *AuthMiddleware) Handle(c *gin.Context) { claims := parseJWT(c.GetHeader("Authorization"))
claims := parseJWT(c.GetHeader("Authorization")) ctx := domain.WithUser(c.Request.Context(), claims.UserID, claims.IsAdmin, claims.SessionID)
ctx := domain.WithUser(c.Request.Context(), claims.UserID, claims.IsAdmin) c.Request = c.Request.WithContext(ctx)
c.Request = c.Request.WithContext(ctx)
c.Next()
}
// domain/context.go // domain/context.go
type ctxKey int func WithUser(ctx context.Context, userID int16, isAdmin bool, sessionID int) context.Context
const userKey ctxKey = iota func UserFromContext(ctx context.Context) (userID int16, isAdmin bool, sessionID int)
func WithUser(ctx context.Context, userID int16, isAdmin bool) context.Context { ... }
func UserFromContext(ctx context.Context) (userID int16, isAdmin bool) { ... }
``` ```
### Transaction Management ### Transaction Management
Repository interfaces include a `Transactor`: The `Transactor` port lets services compose multiple repo calls atomically.
The postgres implementation stores the active `pgx.Tx` in the context; repo
methods pick it up via a conn-or-tx helper, so the same method works inside or
outside a transaction.
```go ```go
// port/repository.go // port/repository.go
type Transactor interface { type Transactor interface {
WithTx(ctx context.Context, fn func(ctx context.Context) error) error WithTx(ctx context.Context, fn func(ctx context.Context) error) error
} }
```
The postgres implementation wraps `pgxpool.Pool.BeginTx`. Inside `fn`, // service/file_service.go (sketch)
all repo calls use the transaction from context. This allows services func (s *FileService) Upload(ctx context.Context, p UploadParams) (*domain.File, error) {
to compose multiple repo calls in a single transaction:
```go
// service/file_service.go
func (s *FileService) Upload(ctx context.Context, input UploadInput) (*domain.File, error) {
return s.tx.WithTx(ctx, func(ctx context.Context) error { return s.tx.WithTx(ctx, func(ctx context.Context) error {
file, err := s.fileRepo.Create(ctx, ...) // uses tx created, err := s.files.Create(ctx, f) // uses tx from ctx
if err != nil { return err } // apply initial tags, etc., in the same tx
for _, tagID := range input.TagIDs { return err
s.tagRepo.AddFileTag(ctx, file.ID, tagID) // same tx
}
s.auditRepo.Log(ctx, ...) // same tx
return nil
}) })
} }
``` ```
### ACL Check Pattern ### ACL Check Pattern
ACL logic is centralized in `ACLService`. Other services call it before ACL logic is centralized in `ACLService`. Other services call it before any
any data mutation or retrieval: mutation or retrieval. The model is private-by-default: admins see everything;
otherwise a `public` flag, creator ownership, or an explicit `acl.permissions`
grant is required.
```go ```go
// service/acl_service.go // service/acl_service.go (shape)
func (s *ACLService) CanView(ctx context.Context, objectType string, objectID uuid.UUID) error { func (s *ACLService) CanView(ctx context.Context, userID int16, isAdmin bool,
userID, isAdmin := domain.UserFromContext(ctx) creatorID int16, isPublic bool, objectType int16, objectID uuid.UUID) (bool, error)
if isAdmin { return nil } func (s *ACLService) CanEdit(ctx context.Context, userID int16, isAdmin bool,
// Check is_public on the object creatorID int16, objectType int16, objectID uuid.UUID) (bool, error)
// If not public, check creator_id == userID
// If not creator, check acl.permissions
// Return domain.ErrForbidden if none match
}
``` ```
### Error Mapping ### Error Mapping
@@ -238,7 +264,7 @@ func (s *ACLService) CanView(ctx context.Context, objectType string, objectID uu
Domain errors → HTTP status codes (handled in handler/response.go): Domain errors → HTTP status codes (handled in handler/response.go):
| Domain Error | HTTP Status | Error Code | | Domain Error | HTTP Status | Error Code |
|-----------------------|-------------|-------------------| | ------------------ | ----------- | ---------------- |
| ErrNotFound | 404 | not_found | | ErrNotFound | 404 | not_found |
| ErrForbidden | 403 | forbidden | | ErrForbidden | 403 | forbidden |
| ErrUnauthorized | 401 | unauthorized | | ErrUnauthorized | 401 | unauthorized |
@@ -249,22 +275,12 @@ Domain errors → HTTP status codes (handled in handler/response.go):
### Filter DSL ### Filter DSL
The DSL parser lives in `db/postgres/filter_parser.go` because it produces The DSL parser lives in `db/postgres/filter_parser.go` because it produces SQL
SQL WHERE clauses — it is a PostgreSQL-specific adapter concern. WHERE clauses — a PostgreSQL-specific adapter concern. The service layer passes
The service layer passes the raw DSL string to the repository; the the raw DSL string down; the repository parses it and builds the query. For a
repository parses it and builds the query. different DBMS, a corresponding parser would live in `db/<dbms>/filter_parser.go`.
For a different DBMS, a corresponding parser would live in
`db/<dbms>/filter_parser.go`.
The interface:
```go ```go
// port/repository.go
type FileRepo interface {
List(ctx context.Context, params FileListParams) (*domain.FilePage, error)
// ...
}
// domain/file.go // domain/file.go
type FileListParams struct { type FileListParams struct {
Filter string // raw DSL string Filter string // raw DSL string
@@ -279,6 +295,9 @@ type FileListParams struct {
} }
``` ```
The DSL grammar itself is documented in `openapi.yaml` (the `filter` query
parameter), so the contract stays in one place.
### JWT Structure ### JWT Structure
```go ```go
@@ -290,31 +309,24 @@ type Claims struct {
} }
``` ```
Access token: short-lived (15 min). Refresh token: long-lived (30 days), Access token: short-lived (15 min default). Refresh token: long-lived (30 days
stored as hash in `activity.sessions.token_hash`. default), rotated on use, stored as a hash in `activity.sessions`. A separate
**content token** (default 6 h) is a single-file capability minted for media
URLs, so a long video keeps streaming past access-token expiry — see
`CONTENT_TOKEN_TTL` in `.env.example`.
### Perceptual Duplicate Detection
Images are dHash-ed inline on upload (`internal/imagehash`); video hashes are
backfilled by the `dedup` CLI (ffmpeg stays off the upload path). A rescan
rebuilds `data.duplicate_pairs` by inserting every pair within
`DUPLICATE_HASH_THRESHOLD` Hamming distance (BK-tree lookups, not O(N²)); the
duplicates API then groups pairs into connected-component clusters. See
`service/duplicate_service.go` and `service/duplicate_index.go`.
### Configuration (.env) ### Configuration (.env)
```env Every variable the server reads is documented in `.env.example` (1:1 with
# Server `config.Config`). Required at startup: `JWT_SECRET`, `ADMIN_PASSWORD`,
LISTEN_ADDR=:42776 `DATABASE_URL`, `FILES_PATH`, `THUMBS_CACHE_PATH`, `IMPORT_PATH`. Everything
JWT_SECRET=<random-32-bytes> else has a sensible default (see `config.go`).
JWT_ACCESS_TTL=15m
JWT_REFRESH_TTL=720h
# Database
DATABASE_URL=postgres://user:pass@host:5432/tanabata?sslmode=disable
# Storage
FILES_PATH=/data/files
THUMBS_CACHE_PATH=/data/thumbs
# Thumbnails
THUMB_WIDTH=160
THUMB_HEIGHT=160
PREVIEW_WIDTH=1920
PREVIEW_HEIGHT=1080
# Import
IMPORT_PATH=/data/import
```
+179
View File
@@ -0,0 +1,179 @@
# Tanabata File Manager — Requirements
> Product requirements for Tanabata File Manager (TFM). Architecture and code
> layout are described separately in [ARCHITECTURE.md](ARCHITECTURE.md),
> [GO_PROJECT_STRUCTURE.md](GO_PROJECT_STRUCTURE.md) and
> [FRONTEND_STRUCTURE.md](FRONTEND_STRUCTURE.md).
## Overview
Tanabata File Manager (TFM) is a multi-user, tag-based web file manager. It runs
on a clientserver architecture and is operated entirely through a web
interface. Its goal is centralized, server-side storage of files with access and
management from both desktop and mobile browsers. The application is primarily
oriented toward **images and video**.
The app is a PWA that can be installed on a desktop or a phone. Files managed by
Tanabata are stored flat in a single directory; each file's on-disk name equals
its UUID in the database.
Support for additional database engines is planned for future versions.
## Core Concepts
- **File** — a single file on the server. It may carry any number of tags and
belong to any number of pools. It has a creator and optional access settings
(a user — which may be null, making the file public — plus read and edit
permission flags). It has an original name and metadata (keyvalue, including
all EXIF data).
- **Tag** — a label on a file. It may be attached to any number of files and
belong to at most one category. It has a name, a description, keyvalue
metadata, and may define auto-tag rules.
- **Auto-tag (tag rule)** — a rule stating that when tag A is attached to a file,
tag B is attached to the same file automatically.
- **Category** — an entity that logically groups several tags. It has a name, a
description, and keyvalue metadata.
- **Pool** — a logical grouping of files. It has a name, a description, and
keyvalue metadata. Files in a pool can be sorted automatically or arranged in
a user-defined manual order.
## Functional Requirements
### 1. File management
1. Browse the file list (lazy load, pagination).
2. Filter files by tags and metadata.
3. View and edit sort settings (persisted per user).
4. Multi-select files (Ctrl, Shift) and act on the selection:
1. Attach / detach tags.
2. Copy / paste tags.
3. Add to a pool.
4. View and edit access settings.
5. Delete (with a confirmation prompt).
5. View a single file.
6. Single-file actions:
1. Attach / detach tags.
2. Copy / paste tags.
3. Add to a pool.
4. View and edit access settings.
5. Replace the file (upload new content under the same ID).
6. Delete (with a confirmation prompt).
7. Browse files gallery-style (prev/next paging through the viewer).
8. Upload new files through the web UI (form or drag-and-drop onto the list).
9. Import new files from a folder on the server.
10. Near-duplicate detection for images and video:
1. Show groups (clusters) of duplicates.
2. Dismiss false duplicates (the app remembers that file A is _not_ a
duplicate of file B).
3. Choose which duplicate to keep and which to delete.
4. Choose, per field, which duplicate the surviving file inherits it from.
11. Trash:
1. Browse trashed files.
2. Restore from trash.
3. Delete permanently.
### 2. Tag management
1. Browse the tag list (lazy load, pagination).
2. Search by name.
3. View and edit sort settings (persisted per user).
4. Multi-select tags (Ctrl, Shift) and act on the selection:
1. Assign auto-tag rules.
2. Change category.
3. Delete (with a confirmation prompt).
5. View a single tag.
6. Single-tag actions:
1. Edit name, description, and metadata (keyvalue).
2. Change category.
3. Assign auto-tag rules.
4. Delete (with a confirmation prompt).
7. Create a tag:
1. Enter name, description, and metadata (keyvalue).
2. Assign a category (optional).
3. Assign auto-tag rules.
### 3. Category management
1. Browse the category list (lazy load, pagination).
2. Search by name.
3. View and edit sort settings (persisted per user).
4. Multi-select categories (Ctrl, Shift) and act on the selection:
1. View shared tags and tags attached to some (but not all) of them.
2. Attach / detach tags.
3. Delete (with a confirmation prompt).
5. View a single category.
6. Single-category actions:
1. Edit name, description, and metadata (keyvalue).
2. View attached tags.
3. Attach / detach tags.
4. Delete (with a confirmation prompt).
7. Create a category:
1. Enter name, description, and metadata (keyvalue).
2. Attach tags.
### 4. Pool management
1. Browse the pool list (lazy load, pagination).
2. Search by name.
3. View and edit sort settings (persisted per user).
4. Multi-select pools (Ctrl, Shift) and act on the selection:
1. View and edit access settings.
2. Delete (with a confirmation prompt).
5. View a single pool.
6. Single-pool actions:
1. Edit name, description, and metadata (keyvalue).
2. View and edit access settings.
3. View all files in the pool.
4. Filter the pool's files by tags.
5. Change the file sort setting (including disabling automatic sorting).
6. Reorder files manually (when automatic sorting is disabled).
7. Delete (with a confirmation prompt).
7. Create a pool:
1. Enter name, description, and metadata (keyvalue).
2. Attach files.
### 5. User settings
1. Username.
2. Password.
3. Sessions:
1. Terminate a session.
4. Path to the server folder scanned during file import.
### 6. Server administration (admin panel)
1. Users:
1. Browse the list.
2. View a single user.
3. Create.
4. Delete.
5. Block / unblock.
6. Set role (reader / editor).
### 7. Audit logging (in the database)
Log the following user actions:
1. File views.
2. Changes to file access settings.
3. Create / edit / delete of a file, tag, category, pool, or filetag relation.
4. Create / block / unblock / delete of a user.
5. User role changes.
6. User login / logout.
7. Session termination.
## Non-Functional Requirements
1. The interface must be as simple and convenient as possible: everything needed
should be at hand, reachable in the fewest possible actions.
2. The interface must adapt to both desktop and mobile devices.
3. The interface must offer dark and light themes.
4. Use PWA technology, including a button that fully resets the PWA (except the
cache) and reloads it from the server.
5. Allow selected files to be cached and viewed offline in the installed PWA.
6. First-run setup must require minimal effort: automatic database migration, a
ready-made Docker Compose file, and a `.env` file with the configurable
installation parameters.
7. Use a Domain-Driven Design approach on the API server.
8. Reject files whose MIME type is not present in the database (no DB entry — no
support).
-374
View File
@@ -1,374 +0,0 @@
from configparser import ConfigParser
from psycopg2.pool import ThreadedConnectionPool
from psycopg2.extras import RealDictCursor
from contextlib import contextmanager
from os import access, W_OK, makedirs, chmod, system
from os.path import isfile, join, basename
from shutil import move
from magic import Magic
from preview_generator.manager import PreviewManager
conf = None
mage = None
previewer = None
db_pool = None
DEFAULT_SORTING = {
"files": {
"key": "created",
"asc": False
},
"tags": {
"key": "created",
"asc": False
},
"categories": {
"key": "created",
"asc": False
},
"pools": {
"key": "created",
"asc": False
},
}
def Initialize(conf_path="/etc/tfm/tfm.conf"):
global mage, previewer
load_config(conf_path)
mage = Magic(mime=True)
previewer = PreviewManager(conf["Paths"]["Thumbs"])
db_connect(conf["DB.limits"]["MinimumConnections"], conf["DB.limits"]["MaximumConnections"], **conf["DB.params"])
def load_config(path):
global conf
conf = ConfigParser()
conf.read(path)
def db_connect(minconn, maxconn, **kwargs):
global db_pool
db_pool = ThreadedConnectionPool(minconn, maxconn, **kwargs)
@contextmanager
def _db_cursor():
global db_pool
try:
conn = db_pool.getconn()
except:
raise RuntimeError("Database not connected")
try:
with conn.cursor(cursor_factory=RealDictCursor) as cur:
yield cur
conn.commit()
except:
conn.rollback()
raise
finally:
db_pool.putconn(conn)
def _validate_column_name(cur, table, column):
cur.execute("SELECT get_column_names(%s) AS name", (table,))
if all([column!=col["name"] for col in cur.fetchall()]):
raise RuntimeError("Invalid column name")
def authorize(username, password, useragent):
with _db_cursor() as cur:
cur.execute("SELECT tfm_session_request(tfm_user_auth(%s, %s), %s) AS sid", (username, password, useragent))
sid = cur.fetchone()["sid"]
return TSession(sid)
class TSession:
sid = None
def __init__(self, sid):
with _db_cursor() as cur:
cur.execute("SELECT tfm_session_validate(%s) IS NOT NULL AS valid", (sid,))
if not cur.fetchone()["valid"]:
raise RuntimeError("Invalid sid")
self.sid = sid
def terminate(self):
with _db_cursor() as cur:
cur.execute("CALL tfm_session_terminate(%s)", (self.sid,))
del self
@property
def username(self):
with _db_cursor() as cur:
cur.execute("SELECT tfm_session_username(%s) AS name", (self.sid,))
return cur.fetchone()["name"]
@property
def is_admin(self):
with _db_cursor() as cur:
cur.execute("SELECT * FROM tfm_user_get_info(%s)", (self.sid,))
return cur.fetchone()["can_edit"]
def get_files(self, order_key=DEFAULT_SORTING["files"]["key"], order_asc=DEFAULT_SORTING["files"]["asc"], offset=0, limit=None):
with _db_cursor() as cur:
_validate_column_name(cur, "v_files", order_key)
cur.execute("SELECT * FROM tfm_get_files(%%s) ORDER BY %s %s OFFSET %s LIMIT %s" % (
order_key,
"ASC" if order_asc else "DESC",
int(offset),
int(limit) if limit is not None else "ALL"
), (self.sid,))
return list(map(dict, cur.fetchall()))
def get_files_by_filter(self, philter=None, order_key=DEFAULT_SORTING["files"]["key"], order_asc=DEFAULT_SORTING["files"]["asc"], offset=0, limit=None):
with _db_cursor() as cur:
_validate_column_name(cur, "v_files", order_key)
cur.execute("SELECT * FROM tfm_get_files_by_filter(%%s, %%s) ORDER BY %s %s OFFSET %s LIMIT %s" % (
order_key,
"ASC" if order_asc else "DESC",
int(offset),
int(limit) if limit is not None else "ALL"
), (self.sid, philter))
return list(map(dict, cur.fetchall()))
def get_tags(self, order_key=DEFAULT_SORTING["tags"]["key"], order_asc=DEFAULT_SORTING["tags"]["asc"], offset=0, limit=None):
with _db_cursor() as cur:
_validate_column_name(cur, "v_tags", order_key)
cur.execute("SELECT * FROM tfm_get_tags(%%s) ORDER BY %s %s, name ASC OFFSET %s LIMIT %s" % (
order_key,
"ASC" if order_asc else "DESC",
int(offset),
int(limit) if limit is not None else "ALL"
), (self.sid,))
return list(map(dict, cur.fetchall()))
def get_categories(self, order_key=DEFAULT_SORTING["categories"]["key"], order_asc=DEFAULT_SORTING["categories"]["asc"], offset=0, limit=None):
with _db_cursor() as cur:
_validate_column_name(cur, "v_categories", order_key)
cur.execute("SELECT * FROM tfm_get_categories(%%s) ORDER BY %s %s OFFSET %s LIMIT %s" % (
order_key,
"ASC" if order_asc else "DESC",
int(offset),
int(limit) if limit is not None else "ALL"
), (self.sid,))
return list(map(dict, cur.fetchall()))
def get_pools(self, order_key=DEFAULT_SORTING["pools"]["key"], order_asc=DEFAULT_SORTING["pools"]["asc"], offset=0, limit=None):
with _db_cursor() as cur:
_validate_column_name(cur, "v_pools", order_key)
cur.execute("SELECT * FROM tfm_get_pools(%%s) ORDER BY %s %s OFFSET %s LIMIT %s" % (
order_key,
"ASC" if order_asc else "DESC",
int(offset),
int(limit) if limit is not None else "ALL"
), (self.sid,))
return list(map(dict, cur.fetchall()))
def get_autotags(self, order_key="child_id", order_asc=True, offset=0, limit=None):
with _db_cursor() as cur:
_validate_column_name(cur, "v_autotags", order_key)
cur.execute("SELECT * FROM tfm_get_autotags(%%s) ORDER BY %s %s OFFSET %s LIMIT %s" % (
order_key,
"ASC" if order_asc else "DESC",
int(offset),
int(limit) if limit is not None else "ALL"
), (self.sid,))
return list(map(dict, cur.fetchall()))
def get_my_sessions(self, order_key="started", order_asc=False, offset=0, limit=None):
with _db_cursor() as cur:
_validate_column_name(cur, "v_sessions", order_key)
cur.execute("SELECT * FROM tfm_get_my_sessions(%%s) ORDER BY %s %s OFFSET %s LIMIT %s" % (
order_key,
"ASC" if order_asc else "DESC",
int(offset),
int(limit) if limit is not None else "ALL"
), (self.sid,))
return list(map(dict, cur.fetchall()))
def get_tags_by_file(self, file_id, order_key=DEFAULT_SORTING["tags"]["key"], order_asc=DEFAULT_SORTING["tags"]["asc"], offset=0, limit=None):
with _db_cursor() as cur:
_validate_column_name(cur, "v_tags", order_key)
cur.execute("SELECT * FROM tfm_get_tags_by_file(%%s, %%s) ORDER BY %s %s OFFSET %s LIMIT %s" % (
order_key,
"ASC" if order_asc else "DESC",
int(offset),
int(limit) if limit is not None else "ALL"
), (self.sid, file_id))
return list(map(dict, cur.fetchall()))
def get_files_by_tag(self, tag_id, order_key=DEFAULT_SORTING["files"]["key"], order_asc=DEFAULT_SORTING["files"]["asc"], offset=0, limit=None):
with _db_cursor() as cur:
_validate_column_name(cur, "v_files", order_key)
cur.execute("SELECT * FROM tfm_get_files_by_tag(%%s, %%s) ORDER BY %s %s OFFSET %s LIMIT %s" % (
order_key,
"ASC" if order_asc else "DESC",
int(offset),
int(limit) if limit is not None else "ALL"
), (self.sid, tag_id))
return list(map(dict, cur.fetchall()))
def get_files_by_pool(self, pool_id, order_key=DEFAULT_SORTING["files"]["key"], order_asc=DEFAULT_SORTING["files"]["asc"], offset=0, limit=None):
with _db_cursor() as cur:
_validate_column_name(cur, "v_files", order_key)
cur.execute("SELECT * FROM tfm_get_files_by_pool(%%s, %%s) ORDER BY %s %s OFFSET %s LIMIT %s" % (
order_key,
"ASC" if order_asc else "DESC",
int(offset),
int(limit) if limit is not None else "ALL"
), (self.sid, pool_id))
return list(map(dict, cur.fetchall()))
def get_parent_tags(self, tag_id, order_key=DEFAULT_SORTING["tags"]["key"], order_asc=DEFAULT_SORTING["tags"]["asc"], offset=0, limit=None):
with _db_cursor() as cur:
_validate_column_name(cur, "v_tags", order_key)
cur.execute("SELECT * FROM tfm_get_parent_tags(%%s, %%s) ORDER BY %s %s OFFSET %s LIMIT %s" % (
order_key,
"ASC" if order_asc else "DESC",
int(offset),
int(limit) if limit is not None else "ALL"
), (self.sid, tag_id))
return list(map(dict, cur.fetchall()))
def get_my_file_views(self, file_id=None, order_key="datetime", order_asc=False, offset=0, limit=None):
with _db_cursor() as cur:
_validate_column_name(cur, "v_files", order_key)
cur.execute("SELECT * FROM tfm_get_my_file_views(%%s, %%s) ORDER BY %s %s OFFSET %s LIMIT %s" % (
order_key,
"ASC" if order_asc else "DESC",
int(offset),
int(limit) if limit is not None else "ALL"
), (self.sid, file_id))
return list(map(dict, cur.fetchall()))
def get_file(self, file_id):
with _db_cursor() as cur:
cur.execute("SELECT * FROM tfm_get_files(%s) WHERE id=%s", (self.sid, file_id))
return cur.fetchone()
def get_tag(self, tag_id):
with _db_cursor() as cur:
cur.execute("SELECT * FROM tfm_get_tags(%s) WHERE id=%s", (self.sid, tag_id))
return cur.fetchone()
def get_category(self, category_id):
with _db_cursor() as cur:
cur.execute("SELECT * FROM tfm_get_categories(%s) WHERE id=%s", (self.sid, category_id))
return cur.fetchone()
def view_file(self, file_id):
with _db_cursor() as cur:
cur.execute("CALL tfm_view_file(%s, %s)", (self.sid, file_id))
def add_file(self, path, datetime=None, notes=None, is_private=None, orig_name=True):
if not isfile(path):
raise FileNotFoundError("No such file '%s'" % path)
if not access(conf["Paths"]["Files"], W_OK) or not access(conf["Paths"]["Thumbs"], W_OK):
raise PermissionError("Invalid directories for files and thumbs")
mime = mage.from_file(path)
if orig_name == True:
orig_name = basename(path)
with _db_cursor() as cur:
cur.execute("SELECT * FROM tfm_add_file(%s, %s, %s, %s, %s, %s)", (self.sid, mime, datetime, notes, is_private, orig_name))
res = cur.fetchone()
file_id = res["f_id"]
ext = res["ext"]
file_path = join(conf["Paths"]["Files"], file_id)
move(path, file_path)
thumb_path = previewer.get_jpeg_preview(file_path, height=160, width=160)
preview_path = previewer.get_jpeg_preview(file_path, height=1080, width=1920)
chmod(file_path, 0o664)
chmod(thumb_path, 0o664)
chmod(preview_path, 0o664)
return file_id, ext
def add_tag(self, name, notes=None, color=None, category_id=None, is_private=None):
if color is not None:
color = color.replace('#', '')
if not category_id:
category_id = None
with _db_cursor() as cur:
cur.execute("SELECT tfm_add_tag(%s, %s, %s, %s, %s, %s) AS id", (self.sid, name, notes, color, category_id, is_private))
return cur.fetchone()["id"]
def add_category(self, name, notes=None, color=None, is_private=None):
if color is not None:
color = color.replace('#', '')
with _db_cursor() as cur:
cur.execute("SELECT tfm_add_category(%s, %s, %s, %s, %s) AS id", (self.sid, name, notes, color, is_private))
return cur.fetchone()["id"]
def add_pool(self, name, notes=None, parent_id=None, is_private=None):
with _db_cursor() as cur:
cur.execute("SELECT tfm_add_pool(%s, %s, %s, %s, %s) AS id", (self.sid, name, notes, parent_id, is_private))
return cur.fetchone()["id"]
def add_autotag(self, child_id, parent_id, is_active=None, apply_to_existing=None):
with _db_cursor() as cur:
cur.execute("SELECT tfm_add_autotag(%s, %s, %s, %s, %s) AS added", (self.sid, child_id, parent_id, is_active, apply_to_existing))
return cur.fetchone()["added"]
def add_file_to_tag(self, file_id, tag_id):
with _db_cursor() as cur:
cur.execute("SELECT tfm_add_file_to_tag(%s, %s, %s) AS id", (self.sid, file_id, tag_id))
return list(map(lambda t: t["id"], cur.fetchall()))
def add_file_to_pool(self, file_id, pool_id):
with _db_cursor() as cur:
cur.execute("SELECT tfm_add_file_to_pool(%s, %s, %s) AS added", (self.sid, file_id, pool_id))
return cur.fetchone()["added"]
def edit_file(self, file_id, mime=None, datetime=None, notes=None, is_private=None):
with _db_cursor() as cur:
cur.execute("CALL tfm_edit_file(%s, %s, %s, %s, %s, %s)", (self.sid, file_id, mime, datetime, notes, is_private))
def edit_tag(self, tag_id, name=None, notes=None, color=None, category_id=None, is_private=None):
if color is not None:
color = color.replace('#', '')
if not category_id:
category_id = None
with _db_cursor() as cur:
cur.execute("CALL tfm_edit_tag(%s, %s, %s, %s, %s, %s, %s)", (self.sid, tag_id, name, notes, color, category_id, is_private))
def edit_category(self, category_id, name=None, notes=None, color=None, is_private=None):
if color is not None:
color = color.replace('#', '')
with _db_cursor() as cur:
cur.execute("CALL tfm_edit_category(%s, %s, %s, %s, %s, %s)", (self.sid, category_id, name, notes, color, is_private))
def edit_pool(self, pool_id, name=None, notes=None, parent_id=None, is_private=None):
with _db_cursor() as cur:
cur.execute("CALL tfm_edit_pool(%s, %s, %s, %s, %s, %s)", (self.sid, pool_id, name, notes, parent_id, is_private))
def remove_file(self, file_id):
with _db_cursor() as cur:
cur.execute("CALL tfm_remove_file(%s, %s)", (self.sid, file_id))
if system("rm %s/%s*" % (conf["Paths"]["Files"], file_id)):
raise RuntimeError("Failed to remove file '%s'" % file_id)
def remove_tag(self, tag_id):
with _db_cursor() as cur:
cur.execute("CALL tfm_remove_tag(%s, %s)", (self.sid, tag_id))
def remove_category(self, category_id):
with _db_cursor() as cur:
cur.execute("CALL tfm_remove_category(%s, %s)", (self.sid, category_id))
def remove_pool(self, pool_id):
with _db_cursor() as cur:
cur.execute("CALL tfm_remove_pool(%s, %s)", (self.sid, pool_id))
def remove_autotag(self, child_id, parent_id):
with _db_cursor() as cur:
cur.execute("CALL tfm_remove_autotag(%s, %s, %s)", (self.sid, child_id, parent_id))
def remove_file_to_tag(self, file_id, tag_id):
with _db_cursor() as cur:
cur.execute("CALL tfm_remove_file_to_tag(%s, %s, %s)", (self.sid, file_id, tag_id))
def remove_file_to_pool(self, file_id, pool_id):
with _db_cursor() as cur:
cur.execute("CALL tfm_remove_file_to_pool(%s, %s, %s)", (self.sid, file_id, pool_id))
-22
View File
@@ -1,22 +0,0 @@
package main
import (
"tanabata/internal/storage/postgres"
)
func main() {
postgres.InitDB("postgres://hiko:taikibansei@192.168.0.25/Tanabata_new?application_name=Tanabata%20testing")
// test_json := json.RawMessage([]byte("{\"valery\": \"ponosoff\"}"))
// data, statusCode, err := db.FileGetSlice(1, "", "+2", -2, 0)
// data, statusCode, err := db.FileGet(1, "0197d056-cfb0-76b5-97e0-bd588826393c")
// data, statusCode, err := db.FileAdd(1, "ABOBA.png", "image/png", time.Now(), "slkdfjsldkflsdkfj;sldkf", test_json)
// statusCode, err := db.FileUpdate(2, "0197d159-bf3a-7617-a3a8-a4a9fc39eca6", map[string]interface{}{
// "name": "ponos.png",
// })
// statusCode, err := db.FileDelete(1, "0197d155-848f-7221-ba4a-4660f257c7d5")
// v, e, err := postgres.FileGetAccess(1, "0197d15a-57f9-712c-991e-c512290e774f")
// fmt.Printf("V: %s, E: %s\n", v, e)
// fmt.Printf("Status: %d\n", statusCode)
// fmt.Printf("Error: %s\n", err)
// fmt.Printf("%+v\n", data)
}
@@ -1,22 +0,0 @@
package main
import (
"fmt"
"tanabata/db"
)
func main() {
db.InitDB("postgres://hiko:taikibansei@192.168.0.25/Tanabata_new?application_name=Tanabata%20testing")
// test_json := json.RawMessage([]byte("{\"valery\": \"ponosoff\"}"))
// data, statusCode, err := db.FileGetSlice(2, "", "+2", -2, 0)
// data, statusCode, err := db.FileGet(1, "0197d056-cfb0-76b5-97e0-bd588826393c")
// data, statusCode, err := db.FileAdd(1, "ABOBA.png", "image/png", time.Now(), "slkdfjsldkflsdkfj;sldkf", test_json)
// statusCode, err := db.FileUpdate(2, "0197d159-bf3a-7617-a3a8-a4a9fc39eca6", map[string]interface{}{
// "name": "ponos.png",
// })
statusCode, err := db.FileDelete(1, "0197d155-848f-7221-ba4a-4660f257c7d5")
fmt.Printf("Status: %d\n", statusCode)
fmt.Printf("Error: %s\n", err)
// fmt.Printf("%+v\n", data)
}
-79
View File
@@ -1,79 +0,0 @@
package db
import (
"context"
"errors"
"fmt"
"net/http"
"time"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgconn"
"github.com/jackc/pgx/v5/pgxpool"
)
var connPool *pgxpool.Pool
func InitDB(connString string) error {
poolConfig, err := pgxpool.ParseConfig(connString)
if err != nil {
return fmt.Errorf("error while parsing connection string: %w", err)
}
poolConfig.MaxConns = 100
poolConfig.MinConns = 0
poolConfig.MaxConnLifetime = time.Hour
poolConfig.HealthCheckPeriod = 30 * time.Second
connPool, err = pgxpool.NewWithConfig(context.Background(), poolConfig)
if err != nil {
return fmt.Errorf("error while initializing DB connections pool: %w", err)
}
return nil
}
func transaction(handler func(context.Context, pgx.Tx) (statusCode int, err error)) (statusCode int, err error) {
ctx := context.Background()
tx, err := connPool.Begin(ctx)
if err != nil {
statusCode = http.StatusInternalServerError
return
}
statusCode, err = handler(ctx, tx)
if err != nil {
tx.Rollback(ctx)
return
}
err = tx.Commit(ctx)
if err != nil {
statusCode = http.StatusInternalServerError
}
return
}
// Handle database error
func handleDBError(errIn error) (statusCode int, err error) {
if errIn == nil {
statusCode = http.StatusOK
return
}
if errors.Is(errIn, pgx.ErrNoRows) {
err = fmt.Errorf("not found")
statusCode = http.StatusNotFound
return
}
var pgErr *pgconn.PgError
if errors.As(errIn, &pgErr) {
switch pgErr.Code {
case "22P02", "22007": // Invalid data format
err = fmt.Errorf("%s", pgErr.Message)
statusCode = http.StatusBadRequest
return
case "23505": // Unique constraint violation
err = fmt.Errorf("already exists")
statusCode = http.StatusConflict
return
}
}
return http.StatusInternalServerError, errIn
}
-53
View File
@@ -1,53 +0,0 @@
package db
import (
"fmt"
"net/http"
"strconv"
"strings"
)
// Convert "filter" URL param to SQL "WHERE" condition
func filterToSQL(filter string) (sql string, statusCode int, err error) {
// filterTokens := strings.Split(string(filter), ";")
sql = "(true)"
return
}
// Convert "sort" URL param to SQL "ORDER BY"
func sortToSQL(sort string) (sql string, statusCode int, err error) {
if sort == "" {
return
}
sortOptions := strings.Split(sort, ",")
sql = " ORDER BY "
for i, sortOption := range sortOptions {
sortOrder := sortOption[:1]
sortColumn := sortOption[1:]
// parse sorting order marker
switch sortOrder {
case "+":
sortOrder = "ASC"
case "-":
sortOrder = "DESC"
default:
err = fmt.Errorf("invalid sorting order mark: %q", sortOrder)
statusCode = http.StatusBadRequest
return
}
// validate sorting column
var n int
n, err = strconv.Atoi(sortColumn)
if err != nil || n < 0 {
err = fmt.Errorf("invalid sorting column: %q", sortColumn)
statusCode = http.StatusBadRequest
return
}
// add sorting option to query
if i > 0 {
sql += ","
}
sql += fmt.Sprintf("%s %s NULLS LAST", sortColumn, sortOrder)
}
return
}
-19
View File
@@ -1,19 +0,0 @@
module tanabata
go 1.23.0
toolchain go1.23.10
require github.com/jackc/pgx/v5 v5.7.5
require (
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/pgx v3.6.2+incompatible // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/stretchr/testify v1.9.0 // indirect
golang.org/x/crypto v0.37.0 // indirect
golang.org/x/sync v0.13.0 // indirect
golang.org/x/text v0.24.0 // indirect
)
-32
View File
@@ -1,32 +0,0 @@
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx v3.6.2+incompatible h1:2zP5OD7kiyR3xzRYMhOcXVvkDZsImVXfj+yIyTQf3/o=
github.com/jackc/pgx v3.6.2+incompatible/go.mod h1:0ZGrqGqkRlliWnWB4zKnWtjbSWbGkVEFm4TeybAXq+I=
github.com/jackc/pgx/v5 v5.7.5 h1:JHGfMnQY+IEtGM63d+NGMjoRpysB2JBwDr5fsngwmJs=
github.com/jackc/pgx/v5 v5.7.5/go.mod h1:aruU7o91Tc2q2cFp5h4uP3f6ztExVpyVv88Xl/8Vl8M=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
golang.org/x/crypto v0.37.0 h1:kJNSjF/Xp7kU0iB2Z+9viTPMW4EqqsrywMXLJOOsXSE=
golang.org/x/crypto v0.37.0/go.mod h1:vg+k43peMZ0pUMhYmVAWysMK35e6ioLh3wB8ZCAfbVc=
golang.org/x/sync v0.13.0 h1:AauUjRAJ9OSnvULf/ARrrVywoJDy0YS2AwQ98I37610=
golang.org/x/sync v0.13.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
golang.org/x/text v0.24.0 h1:dd5Bzh4yt5KYA8f9CJHCP4FB4D51c2c6JvN37xJJkJ0=
golang.org/x/text v0.24.0/go.mod h1:L8rBsPeo2pSS+xqN0d5u2ikmjtmoJbDBT1b7nHvFCdU=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
@@ -1,122 +0,0 @@
package domain
import (
"encoding/json"
"time"
"github.com/jackc/pgx/v5/pgtype"
)
type User struct {
Name string `json:"name"`
IsAdmin bool `json:"isAdmin"`
CanCreate bool `json:"canCreate"`
}
type MIME struct {
Name string `json:"name"`
Extension string `json:"extension"`
}
type (
CategoryCore struct {
ID string `json:"id"`
Name string `json:"name"`
Color pgtype.Text `json:"color"`
}
CategoryItem struct {
CategoryCore
}
CategoryFull struct {
CategoryCore
CreatedAt time.Time `json:"createdAt"`
Creator User `json:"creator"`
Notes pgtype.Text `json:"notes"`
}
)
type (
FileCore struct {
ID string `json:"id"`
Name pgtype.Text `json:"name"`
MIME MIME `json:"mime"`
}
FileItem struct {
FileCore
CreatedAt time.Time `json:"createdAt"`
Creator User `json:"creator"`
}
FileFull struct {
FileCore
CreatedAt time.Time `json:"createdAt"`
Creator User `json:"creator"`
Notes pgtype.Text `json:"notes"`
Metadata json.RawMessage `json:"metadata"`
Tags []TagCore `json:"tags"`
Viewed int `json:"viewed"`
}
)
type (
TagCore struct {
ID string `json:"id"`
Name string `json:"name"`
Color pgtype.Text `json:"color"`
}
TagItem struct {
TagCore
Category CategoryCore `json:"category"`
}
TagFull struct {
TagCore
Category CategoryCore `json:"category"`
CreatedAt time.Time `json:"createdAt"`
Creator User `json:"creator"`
Notes pgtype.Text `json:"notes"`
UsedIncl int `json:"usedIncl"`
UsedExcl int `json:"usedExcl"`
}
)
type Autotag struct {
TriggerTag TagCore `json:"triggerTag"`
AddTag TagCore `json:"addTag"`
IsActive bool `json:"isActive"`
}
type (
PoolCore struct {
ID string `json:"id"`
Name string `json:"name"`
}
PoolItem struct {
PoolCore
}
PoolFull struct {
PoolCore
CreatedAt time.Time `json:"createdAt"`
Creator User `json:"creator"`
Notes pgtype.Text `json:"notes"`
Viewed int `json:"viewed"`
}
)
type Session struct {
ID int `json:"id"`
UserAgent string `json:"userAgent"`
StartedAt time.Time `json:"startedAt"`
ExpiresAt time.Time `json:"expiresAt"`
LastActivity time.Time `json:"lastActivity"`
}
type Pagination struct {
Total int `json:"total"`
Offset int `json:"offset"`
Limit int `json:"limit"`
Count int `json:"count"`
}
type Slice[T any] struct {
Pagination Pagination `json:"pagination"`
Data []T `json:"data"`
}
@@ -1,16 +0,0 @@
package postgres
import "context"
func UserLogin(ctx context.Context, name, password string) (user_id int, err error) {
row := connPool.QueryRow(ctx, "SELECT id FROM users WHERE name=$1 AND password=crypt($2, password)", name, password)
err = row.Scan(&user_id)
return
}
func UserAuth(ctx context.Context, user_id int) (ok, isAdmin bool) {
row := connPool.QueryRow(ctx, "SELECT is_admin FROM users WHERE id=$1", user_id)
err := row.Scan(&isAdmin)
ok = (err == nil)
return
}
@@ -1,268 +0,0 @@
package postgres
import (
"context"
"encoding/json"
"fmt"
"net/http"
"time"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
"tanabata/internal/domain"
)
type FileStore struct {
db *pgxpool.Pool
}
func NewFileStore(db *pgxpool.Pool) *FileStore {
return &FileStore{db: db}
}
// Get user's access rights to file
func (s *FileStore) getAccess(user_id int, file_id string) (canView, canEdit bool, err error) {
ctx := context.Background()
row := connPool.QueryRow(ctx, `
SELECT
COALESCE(a.view, FALSE) OR f.creator_id=$1 OR COALESCE(u.is_admin, FALSE),
COALESCE(a.edit, FALSE) OR f.creator_id=$1 OR COALESCE(u.is_admin, FALSE)
FROM data.files f
LEFT JOIN acl.files a ON a.file_id=f.id AND a.user_id=$1
LEFT JOIN system.users u ON u.id=$1
WHERE f.id=$2
`, user_id, file_id)
err = row.Scan(&canView, &canEdit)
return
}
// Get a set of files
func (s *FileStore) GetSlice(user_id int, filter, sort string, limit, offset int) (files domain.Slice[domain.FileItem], statusCode int, err error) {
filterCond, statusCode, err := filterToSQL(filter)
if err != nil {
return
}
sortExpr, statusCode, err := sortToSQL(sort)
if err != nil {
return
}
// prepare query
query := `
SELECT
f.id,
f.name,
m.name,
m.extension,
uuid_extract_timestamp(f.id),
u.name,
u.is_admin
FROM data.files f
JOIN system.mime m ON m.id=f.mime_id
JOIN system.users u ON u.id=f.creator_id
WHERE NOT f.is_deleted AND (f.creator_id=$1 OR (SELECT view FROM acl.files WHERE file_id=f.id AND user_id=$1) OR (SELECT is_admin FROM system.users WHERE id=$1)) AND
`
query += filterCond
queryCount := query
query += sortExpr
if limit >= 0 {
query += fmt.Sprintf(" LIMIT %d", limit)
}
if offset > 0 {
query += fmt.Sprintf(" OFFSET %d", offset)
}
// execute query
statusCode, err = transaction(func(ctx context.Context, tx pgx.Tx) (statusCode int, err error) {
rows, err := tx.Query(ctx, query, user_id)
if err != nil {
statusCode, err = handleDBError(err)
return
}
defer rows.Close()
count := 0
for rows.Next() {
var file domain.FileItem
err = rows.Scan(&file.ID, &file.Name, &file.MIME.Name, &file.MIME.Extension, &file.CreatedAt, &file.Creator.Name, &file.Creator.IsAdmin)
if err != nil {
statusCode = http.StatusInternalServerError
return
}
files.Data = append(files.Data, file)
count++
}
err = rows.Err()
if err != nil {
statusCode = http.StatusInternalServerError
return
}
files.Pagination.Limit = limit
files.Pagination.Offset = offset
files.Pagination.Count = count
row := tx.QueryRow(ctx, fmt.Sprintf("SELECT COUNT(*) FROM (%s) tmp", queryCount), user_id)
err = row.Scan(&files.Pagination.Total)
if err != nil {
statusCode = http.StatusInternalServerError
}
return
})
if err == nil {
statusCode = http.StatusOK
}
return
}
// Get file
func (s *FileStore) Get(user_id int, file_id string) (file domain.FileFull, statusCode int, err error) {
ctx := context.Background()
row := connPool.QueryRow(ctx, `
SELECT
f.id,
f.name,
m.name,
m.extension,
uuid_extract_timestamp(f.id),
u.name,
u.is_admin,
f.notes,
f.metadata,
(SELECT COUNT(*) FROM activity.file_views fv WHERE fv.file_id=$2 AND fv.user_id=$1)
FROM data.files f
JOIN system.mime m ON m.id=f.mime_id
JOIN system.users u ON u.id=f.creator_id
WHERE NOT f.is_deleted AND f.id=$2 AND (f.creator_id=$1 OR (SELECT view FROM acl.files WHERE file_id=$2 AND user_id=$1) OR (SELECT is_admin FROM system.users WHERE id=$1))
`, user_id, file_id)
err = row.Scan(&file.ID, &file.Name, &file.MIME.Name, &file.MIME.Extension, &file.CreatedAt, &file.Creator.Name, &file.Creator.IsAdmin, &file.Notes, &file.Metadata, &file.Viewed)
if err != nil {
statusCode, err = handleDBError(err)
return
}
rows, err := connPool.Query(ctx, `
SELECT
t.id,
t.name,
COALESCE(t.color, c.color)
FROM data.tags t
LEFT JOIN data.categories c ON c.id=t.category_id
JOIN data.file_tag ft ON ft.tag_id=t.id
WHERE ft.file_id=$1
`, file_id)
if err != nil {
statusCode, err = handleDBError(err)
return
}
defer rows.Close()
for rows.Next() {
var tag domain.TagCore
err = rows.Scan(&tag.ID, &tag.Name, &tag.Color)
if err != nil {
statusCode = http.StatusInternalServerError
return
}
file.Tags = append(file.Tags, tag)
}
err = rows.Err()
if err != nil {
statusCode = http.StatusInternalServerError
return
}
statusCode = http.StatusOK
return
}
// Add file
func (s *FileStore) Add(user_id int, name, mime string, datetime time.Time, notes string, metadata json.RawMessage) (file domain.FileCore, statusCode int, err error) {
ctx := context.Background()
var mime_id int
var extension string
row := connPool.QueryRow(ctx, "SELECT id, extension FROM system.mime WHERE name=$1", mime)
err = row.Scan(&mime_id, &extension)
if err != nil {
if err == pgx.ErrNoRows {
err = fmt.Errorf("unsupported file type: %q", mime)
statusCode = http.StatusBadRequest
} else {
statusCode, err = handleDBError(err)
}
return
}
row = connPool.QueryRow(ctx, `
INSERT INTO data.files (name, mime_id, datetime, creator_id, notes, metadata)
VALUES (NULLIF($1, ''), $2, $3, $4, NULLIF($5 ,''), $6)
RETURNING id
`, name, mime_id, datetime, user_id, notes, metadata)
err = row.Scan(&file.ID)
if err != nil {
statusCode, err = handleDBError(err)
return
}
file.Name.String = name
file.Name.Valid = (name != "")
file.MIME.Name = mime
file.MIME.Extension = extension
statusCode = http.StatusOK
return
}
// Update file
func (s *FileStore) Update(user_id int, file_id string, updates map[string]interface{}) (statusCode int, err error) {
if len(updates) == 0 {
err = fmt.Errorf("no fields provided for update")
statusCode = http.StatusBadRequest
return
}
writableFields := map[string]bool{
"name": true,
"datetime": true,
"notes": true,
"metadata": true,
}
query := "UPDATE data.files SET"
newValues := []interface{}{user_id}
count := 2
for field, value := range updates {
if !writableFields[field] {
err = fmt.Errorf("invalid field: %q", field)
statusCode = http.StatusBadRequest
return
}
query += fmt.Sprintf(" %s=NULLIF($%d, '')", field, count)
newValues = append(newValues, value)
count++
}
query += fmt.Sprintf(
" WHERE id=$%d AND (creator_id=$1 OR (SELECT edit FROM acl.files WHERE file_id=$%d AND user_id=$1) OR (SELECT is_admin FROM system.users WHERE id=$1))",
count, count)
newValues = append(newValues, file_id)
ctx := context.Background()
commandTag, err := connPool.Exec(ctx, query, newValues...)
if err != nil {
statusCode, err = handleDBError(err)
return
}
if commandTag.RowsAffected() == 0 {
err = fmt.Errorf("not found")
statusCode = http.StatusNotFound
return
}
statusCode = http.StatusNoContent
return
}
// Delete file
func (s *FileStore) Delete(user_id int, file_id string) (statusCode int, err error) {
ctx := context.Background()
commandTag, err := connPool.Exec(ctx,
"DELETE FROM data.files WHERE id=$2 AND (creator_id=$1 OR (SELECT edit FROM acl.files WHERE file_id=$2 AND user_id=$1) OR (SELECT is_admin FROM system.users WHERE id=$1))",
user_id, file_id)
if err != nil {
statusCode, err = handleDBError(err)
return
}
if commandTag.RowsAffected() == 0 {
err = fmt.Errorf("not found")
statusCode = http.StatusNotFound
return
}
statusCode = http.StatusNoContent
return
}
@@ -1,92 +0,0 @@
package postgres
import (
"context"
"errors"
"fmt"
"net/http"
"time"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgconn"
"github.com/jackc/pgx/v5/pgxpool"
)
type Storage struct {
db *pgxpool.Pool
}
var connPool *pgxpool.Pool
// Initialize new database storage
func New(dbURL string) (*Storage, error) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
config, err := pgxpool.ParseConfig(dbURL)
if err != nil {
return nil, fmt.Errorf("failed to parse DB URL: %w", err)
}
config.MaxConns = 10
config.MinConns = 2
config.HealthCheckPeriod = time.Minute
db, err := pgxpool.NewWithConfig(ctx, config)
if err != nil {
return nil, fmt.Errorf("failed to connect to database: %w", err)
}
err = db.Ping(ctx)
if err != nil {
return nil, fmt.Errorf("database ping failed: %w", err)
}
return &Storage{db: db}, nil
}
// Close database storage
func (s *Storage) Close() {
s.db.Close()
}
// Run handler inside transaction
func (s *Storage) transaction(ctx context.Context, handler func(context.Context, pgx.Tx) (statusCode int, err error)) (statusCode int, err error) {
tx, err := connPool.Begin(ctx)
if err != nil {
statusCode = http.StatusInternalServerError
return
}
statusCode, err = handler(ctx, tx)
if err != nil {
tx.Rollback(ctx)
return
}
err = tx.Commit(ctx)
if err != nil {
statusCode = http.StatusInternalServerError
}
return
}
// Handle database error
func (s *Storage) handleDBError(errIn error) (statusCode int, err error) {
if errIn == nil {
statusCode = http.StatusOK
return
}
if errors.Is(errIn, pgx.ErrNoRows) {
err = fmt.Errorf("not found")
statusCode = http.StatusNotFound
return
}
var pgErr *pgconn.PgError
if errors.As(errIn, &pgErr) {
switch pgErr.Code {
case "22P02", "22007": // Invalid data format
err = fmt.Errorf("%s", pgErr.Message)
statusCode = http.StatusBadRequest
return
case "23505": // Unique constraint violation
err = fmt.Errorf("already exists")
statusCode = http.StatusConflict
return
}
}
return http.StatusInternalServerError, errIn
}
@@ -1,53 +0,0 @@
package postgres
import (
"fmt"
"net/http"
"strconv"
"strings"
)
// Convert "filter" URL param to SQL "WHERE" condition
func filterToSQL(filter string) (sql string, statusCode int, err error) {
// filterTokens := strings.Split(string(filter), ";")
sql = "(true)"
return
}
// Convert "sort" URL param to SQL "ORDER BY"
func sortToSQL(sort string) (sql string, statusCode int, err error) {
if sort == "" {
return
}
sortOptions := strings.Split(sort, ",")
sql = " ORDER BY "
for i, sortOption := range sortOptions {
sortOrder := sortOption[:1]
sortColumn := sortOption[1:]
// parse sorting order marker
switch sortOrder {
case "+":
sortOrder = "ASC"
case "-":
sortOrder = "DESC"
default:
err = fmt.Errorf("invalid sorting order mark: %q", sortOrder)
statusCode = http.StatusBadRequest
return
}
// validate sorting column
var n int
n, err = strconv.Atoi(sortColumn)
if err != nil || n < 0 {
err = fmt.Errorf("invalid sorting column: %q", sortColumn)
statusCode = http.StatusBadRequest
return
}
// add sorting option to query
if i > 0 {
sql += ","
}
sql += fmt.Sprintf("%s %s NULLS LAST", sortColumn, sortOrder)
}
return
}
@@ -1,21 +0,0 @@
package storage
import (
"encoding/json"
"time"
"tanabata/internal/domain"
)
type Storage interface {
FileRepository
Close()
}
type FileRepository interface {
GetSlice(user_id int, filter, sort string, limit, offset int) (files domain.Slice[domain.FileItem], statusCode int, err error)
Get(user_id int, file_id string) (file domain.FileFull, statusCode int, err error)
Add(user_id int, name, mime string, datetime time.Time, notes string, metadata json.RawMessage) (file domain.FileCore, statusCode int, err error)
Update(user_id int, file_id string, updates map[string]interface{}) (statusCode int, err error)
Delete(user_id int, file_id string) (statusCode int, err error)
}
-120
View File
@@ -1,120 +0,0 @@
package models
import (
"encoding/json"
"time"
"github.com/jackc/pgx/v5/pgtype"
)
type User struct {
Name string `json:"name"`
IsAdmin bool `json:"isAdmin"`
CanCreate bool `json:"canCreate"`
}
type MIME struct {
Name string `json:"name"`
Extension string `json:"extension"`
}
type (
CategoryCore struct {
ID string `json:"id"`
Name string `json:"name"`
Color pgtype.Text `json:"color"`
}
CategoryItem struct {
CategoryCore
}
CategoryFull struct {
CategoryCore
CreatedAt time.Time `json:"createdAt"`
Creator User `json:"creator"`
Notes pgtype.Text `json:"notes"`
}
)
type (
FileCore struct {
ID string `json:"id"`
Name pgtype.Text `json:"name"`
MIME MIME `json:"mime"`
}
FileItem struct {
FileCore
CreatedAt time.Time `json:"createdAt"`
Creator User `json:"creator"`
}
FileFull struct {
FileCore
CreatedAt time.Time `json:"createdAt"`
Creator User `json:"creator"`
Notes pgtype.Text `json:"notes"`
Metadata json.RawMessage `json:"metadata"`
Tags []TagCore `json:"tags"`
Viewed int `json:"viewed"`
}
)
type (
TagCore struct {
ID string `json:"id"`
Name string `json:"name"`
Color pgtype.Text `json:"color"`
}
TagItem struct {
TagCore
Category CategoryCore `json:"category"`
}
TagFull struct {
TagCore
Category CategoryCore `json:"category"`
CreatedAt time.Time `json:"createdAt"`
Creator User `json:"creator"`
Notes pgtype.Text `json:"notes"`
}
)
type Autotag struct {
TriggerTag TagCore `json:"triggerTag"`
AddTag TagCore `json:"addTag"`
IsActive bool `json:"isActive"`
}
type (
PoolCore struct {
ID string `json:"id"`
Name string `json:"name"`
}
PoolItem struct {
PoolCore
}
PoolFull struct {
PoolCore
CreatedAt time.Time `json:"createdAt"`
Creator User `json:"creator"`
Notes pgtype.Text `json:"notes"`
Viewed int `json:"viewed"`
}
)
type Session struct {
ID int `json:"id"`
UserAgent string `json:"userAgent"`
StartedAt time.Time `json:"startedAt"`
ExpiresAt time.Time `json:"expiresAt"`
LastActivity time.Time `json:"lastActivity"`
}
type Pagination struct {
Total int `json:"total"`
Offset int `json:"offset"`
Limit int `json:"limit"`
Count int `json:"count"`
}
type Slice[T any] struct {
Pagination Pagination `json:"pagination"`
Data []T `json:"data"`
}
File diff suppressed because it is too large Load Diff
-36
View File
@@ -1,36 +0,0 @@
body {
justify-content: center;
align-items: center;
}
.decoration {
position: absolute;
top: 0;
}
.decoration.left {
left: 0;
width: 20vw;
}
.decoration.right {
right: 0;
width: 20vw;
}
#auth {
max-width: 100%;
}
#auth h1 {
margin-bottom: 28px;
}
#auth .form-control {
margin: 14px 0;
border-radius: 14px;
}
#login {
margin-top: 20px;
}
-54
View File
@@ -1,54 +0,0 @@
html, body {
margin: 0;
padding: 0;
}
body {
background-color: #312F45;
color: #f0f0f0;
position: absolute;
top: 0;
left: 0;
bottom: 0;
right: 0;
min-height: 100vh;
display: flex;
flex-direction: column;
justify-content: space-between;
align-items: stretch;
font-family: Epilogue;
overflow-x: hidden;
}
.btn {
height: 50px;
width: 100%;
border-radius: 14px;
font-size: 1.5rem;
font-weight: 500;
}
.btn-primary {
background-color: #9592B5;
border-color: #454261;
}
.btn-primary:hover {
background-color: #7D7AA4;
border-color: #454261;
}
.btn-danger {
background-color: #DB6060;
border-color: #851E1E;
}
.btn-danger:hover {
background-color: #D64848;
border-color: #851E1E;
}
.btn-row {
display: flex;
justify-content: space-between;
}
-388
View File
@@ -1,388 +0,0 @@
header, footer {
margin: 0;
padding: 10px;
box-sizing: border-box;
}
header {
padding: 20px;
box-shadow: 0 5px 5px #0004;
}
.icon-header {
height: .8em;
}
#select {
cursor: pointer;
}
.sorting {
position: relative;
display: flex;
justify-content: space-between;
}
#sorting {
cursor: pointer;
}
.highlighted {
color: #9999AD;
}
#icon-expand {
height: 10px;
}
#sorting-options {
position: absolute;
right: 0;
top: 114%;
padding: 4px 10px;
box-sizing: border-box;
background-color: #111118;
border-radius: 10px;
text-align: left;
box-shadow: 0 0 10px black;
z-index: 9999;
}
.sorting-option {
padding: 4px 0;
display: flex;
justify-content: space-between;
}
.sorting-option input[type="radio"] {
float: unset;
margin-left: 1.8em;
}
.filtering-wrapper {
margin-top: 10px;
}
.filtering-block {
position: absolute;
top: 128px;
left: 14px;
right: 14px;
padding: 14px;
background-color: #111118;
border-radius: 10px;
box-shadow: 0 0 10px 4px #0004;
z-index: 9998;
}
main {
margin: 0;
padding: 10px;
height: 100%;
display: flex;
justify-content: space-between;
align-content: flex-start;
align-items: flex-start;
flex-wrap: wrap;
box-sizing: border-box;
overflow-x: hidden;
overflow-y: scroll;
}
main:after {
content: "";
flex: auto;
}
.item-preview {
position: relative;
}
.item-selected:after {
content: "";
display: block;
position: absolute;
top: 10px;
right: 10px;
width: 100%;
height: 50%;
background-image: url("/static/images/icon-select.svg");
background-size: contain;
background-position: right;
background-repeat: no-repeat;
}
.file-preview {
margin: 1px 0;
padding: 0;
width: 160px;
height: 160px;
max-width: calc(33vw - 7px);
max-height: calc(33vw - 7px);
overflow: hidden;
cursor: pointer;
}
.file-thumb {
width: 100%;
height: 100%;
object-fit: contain;
object-position: center;
}
.file-preview .overlay {
position: absolute;
top: 0;
right: 0;
bottom: 0;
left: 0;
background-color: #0002;
}
.file-preview:hover .overlay {
background-color: #0004;
}
.tag-preview, .filtering-token {
margin: 5px 5px;
padding: 5px 10px;
border-radius: 5px;
background-color: #444455;
cursor: pointer;
}
.category-preview {
margin: 5px 5px;
padding: 5px 10px;
border-radius: 5px;
background-color: #444455;
cursor: pointer;
}
.file {
margin: 0;
padding: 0;
min-width: 100vw;
min-height: 100vh;
max-width: 100vw;
max-height: 100vh;
display: flex;
justify-content: center;
align-items: center;
background-color: black;
}
.file .preview-img {
max-width: 100vw;
max-height: 100vh;
}
.selection-manager {
position: fixed;
left: 10px;
right: 10px;
bottom: 65px;
box-sizing: border-box;
max-height: 40vh;
display: flex;
flex-direction: column;
padding: 15px 10px;
background-color: #181721;
border-radius: 10px;
box-shadow: 0 0 5px #0008;
}
.selection-manager hr {
margin: 5px 0;
}
.selection-header {
display: flex;
justify-content: space-between;
}
.selection-header > * {
cursor: pointer;
}
#selection-edit-tags {
color: #4DC7ED;
}
#selection-add-to-pool {
color: #F5E872;
}
#selection-delete {
color: #DB6060;
}
.selection-tags {
max-height: 100%;
overflow-x: hidden;
overflow-y: scroll;
}
input[type="color"] {
width: 100%;
}
.tags-container, .filtering-operators, .filtering-tokens {
padding: 5px;
background-color: #212529;
border: 1px solid #495057;
border-radius: .375rem;
display: flex;
justify-content: space-between;
align-content: flex-start;
align-items: flex-start;
flex-wrap: wrap;
box-sizing: border-box;
}
.filtering-operators {
margin-bottom: 15px;
}
.tags-container, .filtering-tokens {
margin: 15px 0;
height: 200px;
overflow-x: hidden;
overflow-y: scroll;
}
.tags-container:after, .filtering-tokens:after {
content: "";
flex: auto;
}
.tags-container-selected {
height: 100px;
}
#files-filter {
margin-bottom: 0;
height: 56px;
}
.viewer-wrapper {
position: absolute;
left: 0;
top: 0;
right: 0;
bottom: 0;
background-color: #000a;
display: flex;
justify-content: center;
/* overflow-y: scroll;*/
}
.viewer-nav {
position: absolute;
top: 0;
bottom: 0;
display: flex;
justify-content: center;
align-items: center;
cursor: pointer;
}
.viewer-nav:hover {
background-color: #b4adff40;
}
.viewer-nav-prev {
left: 0;
right: 80vw;
}
.viewer-nav-next {
left: 80vw;
right: 0;
}
.viewer-nav-close {
left: 0;
right: 0;
bottom: unset;
height: 15vh;
}
.viewer-nav-icon {
width: 20px;
height: 32px;
}
.viewer-nav-close > .viewer-nav-icon {
width: 16px;
height: 16px;
}
#viewer {
width: 100%;
height: 100%;
max-width: 100%;
}
.sessions-wrapper {
padding: 14px;
background-color: #111118;
border-radius: 10px;
}
.btn-terminate {
height: 20px;
cursor: pointer;
}
footer {
display: flex;
justify-content: space-between;
align-items: center;
background-color: #0007;
}
.nav {
display: flex;
justify-content: center;
align-items: center;
height: 40px;
width: 18vw;
background: transparent;
border: 0;
border-radius: 10px;
outline: 0;
}
.nav.curr, .nav:hover {
background-color: #343249;
}
.navicon {
display: block;
height: 30px;
}
#loader {
position: fixed;
left: 0;
top: 0;
right: 0;
bottom: 0;
display: flex;
justify-content: center;
align-items: center;
background-color: #000a;
z-index: 9999;
}
.loader-wrapper {
padding: 15px;
border-radius: 12px;
background-color: white;
}
.loader-img {
max-width: 20vw;
max-height: 20vh;
}
Binary file not shown.

Before

Width:  |  Height:  |  Size: 10 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.6 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 3.4 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 5.0 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 6.6 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 7.8 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 8.3 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 10 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 11 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 3.9 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 4.2 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 5.0 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 5.2 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.6 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.4 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 7.0 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 127 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 158 KiB

@@ -1,4 +0,0 @@
<svg width="22" height="22" viewBox="0 0 22 22" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M4.51245 10.9993C4.51245 10.7415 4.61483 10.4944 4.79705 10.3122C4.97928 10.1299 5.22644 10.0276 5.48415 10.0276H10.0097V5.50203C10.0097 5.24432 10.112 4.99717 10.2943 4.81494C10.4765 4.63271 10.7237 4.53033 10.9814 4.53033C11.2391 4.53033 11.4862 4.63271 11.6685 4.81494C11.8507 4.99717 11.9531 5.24432 11.9531 5.50203V10.0276H16.4786C16.7363 10.0276 16.9835 10.1299 17.1657 10.3122C17.3479 10.4944 17.4503 10.7415 17.4503 10.9993C17.4503 11.257 17.3479 11.5041 17.1657 11.6863C16.9835 11.8686 16.7363 11.971 16.4786 11.971H11.9531V16.4965C11.9531 16.7542 11.8507 17.0013 11.6685 17.1836C11.4862 17.3658 11.2391 17.4682 10.9814 17.4682C10.7237 17.4682 10.4765 17.3658 10.2943 17.1836C10.112 17.0013 10.0097 16.7542 10.0097 16.4965V11.971H5.48415C5.22644 11.971 4.97928 11.8686 4.79705 11.6863C4.61483 11.5041 4.51245 11.257 4.51245 10.9993Z" fill="#9999AD"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M4.91409 0.335277C8.9466 -0.111759 13.0162 -0.111759 17.0487 0.335277C19.4157 0.599579 21.3267 2.46394 21.604 4.84396C22.0834 8.93416 22.0834 13.0658 21.604 17.156C21.3254 19.536 19.4144 21.3991 17.0487 21.6647C13.0162 22.1118 8.9466 22.1118 4.91409 21.6647C2.54704 21.3991 0.636031 19.536 0.358773 17.156C-0.119591 13.0659 -0.119591 8.93404 0.358773 4.84396C0.636031 2.46394 2.54833 0.599579 4.91409 0.335277ZM16.8336 2.26572C12.944 1.83459 9.01873 1.83459 5.12916 2.26572C4.40913 2.3456 3.73705 2.66589 3.2215 3.17486C2.70595 3.68383 2.37704 4.35174 2.28792 5.07069C1.82714 9.01056 1.82714 12.9907 2.28792 16.9306C2.37732 17.6493 2.70634 18.3169 3.22186 18.8256C3.73739 19.3343 4.40932 19.6544 5.12916 19.7343C8.98616 20.1644 12.9766 20.1644 16.8336 19.7343C17.5532 19.6542 18.2248 19.3339 18.7401 18.8253C19.2554 18.3166 19.5842 17.6491 19.6735 16.9306C20.1343 12.9907 20.1343 9.01056 19.6735 5.07069C19.5839 4.3524 19.255 3.68524 18.7397 3.17681C18.2245 2.66839 17.553 2.34835 16.8336 2.26831" fill="#9999AD"/>
</svg>

Before

Width:  |  Height:  |  Size: 2.0 KiB

@@ -1,5 +0,0 @@
<svg width="24" height="24" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M18.875 9.25C21.1532 9.25 23 7.40317 23 5.125C23 2.84683 21.1532 1 18.875 1C16.5968 1 14.75 2.84683 14.75 5.125C14.75 7.40317 16.5968 9.25 18.875 9.25Z" stroke="#9999AD" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"/>
<path d="M5.125 23C7.40317 23 9.25 21.1532 9.25 18.875C9.25 16.5968 7.40317 14.75 5.125 14.75C2.84683 14.75 1 16.5968 1 18.875C1 21.1532 2.84683 23 5.125 23Z" stroke="#9999AD" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"/>
<path d="M14.75 14.75H23V21.625C23 21.9897 22.8551 22.3394 22.5973 22.5973C22.3394 22.8551 21.9897 23 21.625 23H16.125C15.7603 23 15.4106 22.8551 15.1527 22.5973C14.8949 22.3394 14.75 21.9897 14.75 21.625V14.75ZM1 1H9.25V7.875C9.25 8.23967 9.10513 8.58941 8.84727 8.84727C8.58941 9.10513 8.23967 9.25 7.875 9.25H2.375C2.01033 9.25 1.66059 9.10513 1.40273 8.84727C1.14487 8.58941 1 8.23967 1 7.875V1Z" stroke="#9999AD" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"/>
</svg>

Before

Width:  |  Height:  |  Size: 1.1 KiB

@@ -1,3 +0,0 @@
<svg width="16" height="9" viewBox="0 0 16 9" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M8.10279 7.27726L14.8104 0.579294C14.8755 0.513843 14.9531 0.462123 15.0386 0.427199C15.1241 0.392275 15.2157 0.374857 15.308 0.375976C15.4003 0.377095 15.4915 0.396729 15.5761 0.433714C15.6607 0.470699 15.737 0.524284 15.8005 0.591294C15.9306 0.728358 16.0022 0.910756 15.9999 1.09973C15.9977 1.2887 15.9219 1.46935 15.7885 1.60329L8.58484 8.79625C8.5202 8.86132 8.44324 8.91285 8.35845 8.94783C8.27366 8.98282 8.18275 9.00055 8.09103 8.99999C7.99931 8.99943 7.90862 8.98059 7.82427 8.94458C7.73991 8.90857 7.66359 8.8561 7.59975 8.79025L0.204043 1.21929C0.0731536 1.08376 0 0.902704 0 0.714294C0 0.525883 0.0731536 0.344832 0.204043 0.209296C0.268362 0.143072 0.345316 0.0904251 0.43035 0.0544745C0.515384 0.0185239 0.606768 0 0.69909 0C0.791413 0 0.882797 0.0185239 0.967831 0.0544745C1.05286 0.0904251 1.12982 0.143072 1.19414 0.209296L8.10279 7.27726Z" fill="#9999AD"/>
</svg>

Before

Width:  |  Height:  |  Size: 985 B

@@ -1,4 +0,0 @@
<svg width="22" height="22" viewBox="0 0 22 22" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M13.0465 20.4651H8.95346V22H13.0465V20.4651ZM1.53488 13.0465V8.95352H1.29521e-06V13.0465H1.53488ZM20.4651 12.5994V13.0465H21.9999V12.5994H20.4651ZM13.9582 3.43921L18.0092 7.08506L19.0356 5.94311L14.9855 2.29726L13.9582 3.43921ZM21.9999 12.5994C21.9999 10.8711 22.0153 9.77621 21.5804 8.79798L20.1775 9.42319C20.4497 10.0351 20.4651 10.736 20.4651 12.5994H21.9999ZM18.0092 7.08506C19.3937 8.33138 19.9053 8.81231 20.1775 9.42319L21.5804 8.79798C21.1445 7.81873 20.3208 7.09938 19.0356 5.94311L18.0092 7.08506ZM8.98416 1.53494C10.6029 1.53494 11.2138 1.54722 11.7572 1.75596L12.3077 0.323405C11.4359 -0.012222 10.4863 5.70826e-05 8.98416 5.70826e-05V1.53494ZM14.9855 2.29828C13.8743 1.29856 13.1795 0.656985 12.3077 0.323405L11.7582 1.75596C12.3026 1.9647 12.761 2.36172 13.9582 3.43921L14.9855 2.29828ZM8.95346 20.4651C7.00212 20.4651 5.61664 20.4631 4.56371 20.3219C3.53534 20.1837 2.94185 19.9238 2.50902 19.491L1.42437 20.5756C2.18976 21.3431 3.16083 21.6818 4.36008 21.8434C5.53682 22.002 7.04612 22 8.95346 22V20.4651ZM1.29521e-06 13.0465C1.29521e-06 14.9539 -0.00204521 16.4621 0.156559 17.6399C0.318233 18.8392 0.657953 19.8102 1.42335 20.5766L2.50799 19.492C2.07618 19.0581 1.81627 18.4646 1.67814 17.4353C1.53693 16.3844 1.53488 14.9979 1.53488 13.0465H1.29521e-06ZM13.0465 22C14.9538 22 16.4621 22.002 17.6399 21.8434C18.8391 21.6818 19.8102 21.342 20.5766 20.5766L19.4919 19.492C19.0581 19.9238 18.4646 20.1837 17.4352 20.3219C16.3843 20.4631 14.9978 20.4651 13.0465 20.4651V22ZM20.4651 13.0465C20.4651 14.9979 20.463 16.3844 20.3218 17.4363C20.1837 18.4647 19.9238 19.0581 19.4909 19.491L20.5756 20.5756C21.343 19.8102 21.6817 18.8392 21.8434 17.6399C22.002 16.4632 21.9999 14.9539 21.9999 13.0465H20.4651ZM1.53488 8.95352C1.53488 7.00217 1.53693 5.61669 1.67814 4.56376C1.81627 3.53539 2.07618 2.94191 2.50902 2.50907L1.42437 1.42442C0.656929 2.18982 0.318233 3.16088 0.156559 4.36014C-0.00204521 5.53688 1.29521e-06 7.04617 1.29521e-06 8.95352H1.53488ZM8.98416 5.70826e-05C7.06556 5.70826e-05 5.55012 -0.00198942 4.36827 0.156615C3.1639 0.318289 2.18976 0.658009 1.42335 1.4234L2.50799 2.50805C2.94185 2.07624 3.53636 1.81633 4.57189 1.67819C5.62891 1.53698 7.02258 1.53494 8.98416 1.53494V5.70826e-05Z" fill="#9999AD"/>
<path d="M12.0232 1.27905V3.83718C12.0232 6.24899 12.0232 7.45541 12.7722 8.20443C13.5212 8.95345 14.7276 8.95345 17.1395 8.95345H21.2325" stroke="#9999AD" stroke-width="1.5"/>
</svg>

Before

Width:  |  Height:  |  Size: 2.5 KiB

@@ -1,3 +0,0 @@
<svg width="22" height="22" viewBox="0 0 22 22" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M6.07102 2.7508H4.64702C4.80588 1.9743 5.22797 1.27646 5.84196 0.775241C6.45595 0.274027 7.22417 0.000183205 8.01676 0H16.7276C18.1259 0 19.467 0.55548 20.4558 1.54424C21.4445 2.533 22 3.87405 22 5.27237V13.9832C22 14.7743 21.7273 15.5411 21.2279 16.1546C20.7285 16.7681 20.033 17.1907 19.2584 17.3511V15.9253C19.6583 15.7816 20.0042 15.518 20.2487 15.1704C20.4932 14.8228 20.6245 14.4082 20.6246 13.9832V5.27237C20.6246 4.23883 20.214 3.24762 19.4832 2.5168C18.7524 1.78597 17.7612 1.3754 16.7276 1.3754H8.01676C7.59001 1.37527 7.17373 1.50748 6.82525 1.75381C6.47678 2.00014 6.21327 2.34846 6.07102 2.7508ZM3.4385 3.66774C2.52656 3.66774 1.65196 4.03001 1.00711 4.67485C0.36227 5.3197 0 6.19429 0 7.10624V18.5679C0 19.4799 0.36227 20.3545 1.00711 20.9993C1.65196 21.6442 2.52656 22.0064 3.4385 22.0064H14.9002C15.8121 22.0064 16.6867 21.6442 17.3316 20.9993C17.9764 20.3545 18.3387 19.4799 18.3387 18.5679V7.10624C18.3387 6.19429 17.9764 5.3197 17.3316 4.67485C16.6867 4.03001 15.8121 3.66774 14.9002 3.66774H3.4385ZM1.3754 7.10624C1.3754 6.55907 1.59276 6.03431 1.97967 5.64741C2.36658 5.2605 2.89133 5.04314 3.4385 5.04314H14.9002C15.4473 5.04314 15.9721 5.2605 16.359 5.64741C16.7459 6.03431 16.9633 6.55907 16.9633 7.10624V18.5679C16.9633 19.1151 16.7459 19.6398 16.359 20.0268C15.9721 20.4137 15.4473 20.631 14.9002 20.631H3.4385C2.89133 20.631 2.36658 20.4137 1.97967 20.0268C1.59276 19.6398 1.3754 19.1151 1.3754 18.5679V7.10624Z" fill="#9999AD"/>
</svg>

Before

Width:  |  Height:  |  Size: 1.5 KiB

@@ -1,3 +0,0 @@
<svg width="31" height="23" viewBox="0 0 31 23" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M1.414 7.97C1.78906 7.59506 2.29767 7.38443 2.828 7.38443C3.35833 7.38443 3.86695 7.59506 4.242 7.97L11.314 15.042L25.454 0.900004C25.6397 0.714184 25.8602 0.566757 26.1028 0.466141C26.3455 0.365526 26.6056 0.313692 26.8683 0.313599C27.131 0.313506 27.3911 0.365156 27.6339 0.4656C27.8766 0.566044 28.0972 0.713315 28.283 0.899004C28.4688 1.08469 28.6163 1.30516 28.7169 1.54783C28.8175 1.79049 28.8693 2.0506 28.8694 2.3133C28.8695 2.57599 28.8179 2.83614 28.7174 3.07887C28.617 3.32161 28.4697 3.54218 28.284 3.728L11.314 20.698L1.414 10.798C1.03906 10.4229 0.82843 9.91433 0.82843 9.384C0.82843 8.85368 1.03906 8.34506 1.414 7.97Z" fill="white" stroke="black" stroke-width="1"/>
</svg>

Before

Width:  |  Height:  |  Size: 794 B

@@ -1,4 +0,0 @@
<svg width="23" height="24" viewBox="0 0 23 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M11.371 15.3C13.1935 15.3 14.671 13.8226 14.671 12C14.671 10.1775 13.1935 8.70001 11.371 8.70001C9.54844 8.70001 8.07098 10.1775 8.07098 12C8.07098 13.8226 9.54844 15.3 11.371 15.3Z" stroke="#9999AD" stroke-width="1.5"/>
<path d="M13.3125 1.1672C12.9088 1 12.3962 1 11.371 1C10.3458 1 9.8332 1 9.4295 1.1672C9.1624 1.27776 8.91971 1.43989 8.7153 1.6443C8.51089 1.84871 8.34877 2.0914 8.2382 2.3585C8.137 2.6038 8.0963 2.8909 8.0809 3.3078C8.07405 3.60919 7.9907 3.90389 7.8387 4.16423C7.68669 4.42456 7.47101 4.642 7.2119 4.7961C6.94889 4.94355 6.65271 5.02171 6.35119 5.02325C6.04967 5.02479 5.75271 4.94965 5.4882 4.8049C5.1186 4.6091 4.8513 4.5013 4.5862 4.4661C4.00795 4.39006 3.42317 4.54674 2.9604 4.9017C2.615 5.169 2.3576 5.6123 1.845 6.5C1.3324 7.3877 1.075 7.831 1.0189 8.2655C0.981102 8.552 1.00012 8.84314 1.07486 9.12228C1.1496 9.40143 1.2786 9.66312 1.4545 9.8924C1.6173 10.1036 1.845 10.2807 2.1981 10.5029C2.7184 10.8296 3.0528 11.3862 3.0528 12C3.0528 12.6138 2.7184 13.1704 2.1981 13.496C1.845 13.7193 1.6162 13.8964 1.4545 14.1076C1.2786 14.3369 1.1496 14.5986 1.07486 14.8777C1.00012 15.1569 0.981102 15.448 1.0189 15.7345C1.0761 16.1679 1.3324 16.6123 1.8439 17.5C2.3576 18.3877 2.6139 18.831 2.9604 19.0983C3.18968 19.2742 3.45137 19.4032 3.73052 19.4779C4.00967 19.5527 4.30081 19.5717 4.5873 19.5339C4.8513 19.4987 5.1186 19.3909 5.4882 19.1951C5.75271 19.0503 6.04967 18.9752 6.35119 18.9767C6.65271 18.9783 6.94889 19.0565 7.2119 19.2039C7.7432 19.5119 8.0589 20.0784 8.0809 20.6922C8.0963 21.1102 8.1359 21.3962 8.2382 21.6415C8.34877 21.9086 8.51089 22.1513 8.7153 22.3557C8.91971 22.5601 9.1624 22.7222 9.4295 22.8328C9.8332 23 10.3458 23 11.371 23C12.3962 23 12.9088 23 13.3125 22.8328C13.5796 22.7222 13.8223 22.5601 14.0267 22.3557C14.2311 22.1513 14.3932 21.9086 14.5038 21.6415C14.605 21.3962 14.6457 21.1102 14.6611 20.6922C14.6831 20.0784 14.9988 19.5108 15.5301 19.2039C15.7931 19.0565 16.0893 18.9783 16.3908 18.9767C16.6923 18.9752 16.9893 19.0503 17.2538 19.1951C17.6234 19.3909 17.8907 19.4987 18.1547 19.5339C18.4412 19.5717 18.7323 19.5527 19.0115 19.4779C19.2906 19.4032 19.5523 19.2742 19.7816 19.0983C20.1281 18.8321 20.3844 18.3877 20.897 17.5C21.4096 16.6123 21.667 16.169 21.7231 15.7345C21.7609 15.448 21.7419 15.1569 21.6672 14.8777C21.5924 14.5986 21.4634 14.3369 21.2875 14.1076C21.1247 13.8964 20.897 13.7193 20.5439 13.4971C20.2862 13.3405 20.0726 13.1209 19.9231 12.859C19.7736 12.5971 19.6931 12.3015 19.6892 12C19.6892 11.3862 20.0236 10.8296 20.5439 10.504C20.897 10.2807 21.1258 10.1036 21.2875 9.8924C21.4634 9.66312 21.5924 9.40143 21.6672 9.12228C21.7419 8.84314 21.7609 8.552 21.7231 8.2655C21.6659 7.8321 21.4096 7.3877 20.8981 6.5C20.3844 5.6123 20.1281 5.169 19.7816 4.9017C19.5523 4.7258 19.2906 4.59679 19.0115 4.52205C18.7323 4.44731 18.4412 4.4283 18.1547 4.4661C17.8907 4.5013 17.6234 4.6091 17.2527 4.8049C16.9883 4.94946 16.6916 5.02449 16.3903 5.02295C16.089 5.02141 15.793 4.94335 15.5301 4.7961C15.271 4.642 15.0553 4.42456 14.9033 4.16423C14.7513 3.90389 14.668 3.60919 14.6611 3.3078C14.6457 2.8898 14.6061 2.6038 14.5038 2.3585C14.3932 2.0914 14.2311 1.84871 14.0267 1.6443C13.8223 1.43989 13.5796 1.27776 13.3125 1.1672Z" stroke="#9999AD" stroke-width="1.5"/>
</svg>

Before

Width:  |  Height:  |  Size: 3.3 KiB

@@ -1,5 +0,0 @@
<svg width="24" height="24" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M4.00067 16.5511C2.30116 14.8505 1.45086 14.0013 1.13515 12.8979C0.818352 11.7946 1.08895 10.6231 1.63016 8.28122L1.94146 6.93041C2.39576 4.9592 2.62346 3.97359 3.29777 3.29819C3.97317 2.62388 4.95878 2.39618 6.92999 1.94188L8.2808 1.62947C10.6238 1.08937 11.7942 0.818769 12.8975 1.13447C14.0008 1.45127 14.85 2.30158 16.5496 4.00109L18.5626 6.0141C21.5227 8.97312 23 10.4515 23 12.2885C23 14.1267 21.5216 15.6051 18.5637 18.563C15.6046 21.522 14.1262 23.0004 12.2881 23.0004C10.4511 23.0004 8.97161 21.522 6.01369 18.5641L4.00067 16.5511Z" stroke="#9999AD" stroke-width="1.5"/>
<path d="M9.82325 10.1229C10.6824 9.26375 10.6824 7.87077 9.82325 7.01161C8.96409 6.15246 7.57112 6.15246 6.71196 7.01162C5.8528 7.87077 5.8528 9.26375 6.71196 10.1229C7.57112 10.9821 8.96409 10.9821 9.82325 10.1229Z" stroke="#9999AD" stroke-width="1.5"/>
<path d="M11.4962 19.1504L19.1731 11.4724" stroke="#9999AD" stroke-width="1.5" stroke-linecap="round"/>
</svg>

Before

Width:  |  Height:  |  Size: 1.0 KiB

@@ -1,3 +0,0 @@
<svg width="25" height="25" viewBox="0 0 25 25" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M12.5 25C5.59625 25 0 19.4037 0 12.5C0 5.59625 5.59625 0 12.5 0C19.4037 0 25 5.59625 25 12.5C25 19.4037 19.4037 25 12.5 25ZM12.5 22.5C15.1522 22.5 17.6957 21.4464 19.5711 19.5711C21.4464 17.6957 22.5 15.1522 22.5 12.5C22.5 9.84783 21.4464 7.3043 19.5711 5.42893C17.6957 3.55357 15.1522 2.5 12.5 2.5C9.84783 2.5 7.3043 3.55357 5.42893 5.42893C3.55357 7.3043 2.5 9.84783 2.5 12.5C2.5 15.1522 3.55357 17.6957 5.42893 19.5711C7.3043 21.4464 9.84783 22.5 12.5 22.5ZM6.25 11.25H18.75V13.75H6.25V11.25Z" fill="#DB6060"/>
</svg>

Before

Width:  |  Height:  |  Size: 626 B

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.0 KiB

Some files were not shown because too many files have changed in this diff Show More