The pool view's selection bar could only remove files from the current
pool. Add an "Add to pool" action beside it that opens the existing file
picker with the selected files (in selection order), so a multi-select can
be copied into another pool in one step. On success the picker closes and
the selection clears.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The file viewer could only open the pool picker via its top-right button —
there was no `p` shortcut there (only the grid had one), so pressing `p`
on the view page did nothing. Add `p` to open the picker from the viewer,
and give the picker itself full keyboard control: `/` focuses the search
box, arrows move a highlight through the pool list, Enter adds to the
highlighted pool, and Escape clears the search first, then closes.
Both the viewer and the grid now yield the keyboard entirely to the open
picker (the picker owns Escape via its own window handler) so the
clear-then-close behaviour isn't pre-empted by the host's own Escape.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Video thumbnails and previews were extracted ~1s in, which lands on
shared intros, title cards or black lead-in frames. Take the frame from
the middle (duration/2) instead — the same frame used for the perceptual
hash — so the thumbnail/preview reflects what dedup compared. Fold the
midpoint logic into a shared extractVideoFrameMiddle helper reused by
both the thumbnail/preview path and VideoFrameMiddle.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Shift range-select normalized the range with Math.min/Math.max and
always iterated ascending, so the selection's insertion order (which the
Set preserves and which carries through to e.g. pool add order) ignored
the gesture direction. Iterate anchor → target instead via a shared
selectRange helper, so selecting first→last and last→first yield
correspondingly ordered selections.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reorder was the only pool-file operation that didn't record an audit
entry, unlike AddFiles (file_pool_add) and RemoveFiles
(file_pool_remove). Log file_pool_reorder on success and seed the new
action type.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both phases now share a small, dependency-free progress indicator: an in-place
bar on a TTY (e.g. `docker compose run`), and a line every 10% when stdout is
piped (cron/CI) so logs don't fill with carriage returns. Also fixes the pairs
phase, which mislabelled its progress as "hashed" — it now reads "matching".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dedup pairs rebuild reads this tunable (default 10/64); it was added to the
backend config but never documented for operators. No other new env vars were
introduced by duplicate detection — the dedup compose service reuses the
existing PUID/PGID/FILES_DIR/THUMBS_DIR.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a `dedup` task service under the "tools" profile so it's kept out of
`docker compose up` and run on demand:
docker compose run --rm dedup # hashes, then rebuild pairs
docker compose run --rm dedup -pairs # only rebuild pairs
docker compose run --rm dedup -hashes # only backfill hashes
It reuses the app image, .env, volumes and networks, overriding only the
entrypoint to /app/dedup. Unlike `docker exec` on the live server, this runs in
its own container and is self-documented for cron/CI use.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The backend build stage compiled only ./cmd/server, so the dedup maintenance
tool was never available on deploy. Build it alongside the server and copy
/out/dedup to /app/dedup in the runtime image (which already has ffmpeg/ffprobe
for video frames and the /data volume). Run it with `docker exec`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the duplicate-detection UI:
- api/duplicates.ts: getDuplicates / dismissDuplicate / resolveDuplicate, plus
the cluster and merge-field types.
- /files/duplicates: an offset-paginated list of clusters. Each cluster shows its
files (auth-loaded thumbnails via a reusable Thumb component); the user clicks a
file to mark it the survivor, then per other file: Merge, Delete, or "Not a dup"
(dismiss). The list reloads after each action so it stays consistent with the
rescan-gated server state.
- DuplicateMergeDialog: a bottom sheet to merge two files field-by-field — each
scalar from the kept or other file, metadata keep/other/merge, tags & pools
keep-or-union, with a swap-survivor toggle and an optional trash-the-other box.
- Entry point: a Duplicates action in the files Header next to Trash.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds GET /files/duplicates, POST /files/duplicates/dismiss and POST
/files/duplicates/resolve to the OpenAPI spec, plus the DuplicateCluster,
DuplicateClusterPage and DuplicateResolve (with MergeScalarChoice /
MergeRelationChoice) schemas describing the field-by-field merge contract.
Also fills a pre-existing gap in the File schema: it now documents the `tags`
array (always returned by the API) and marks the always-present fields required,
so generated clients type these as non-optional.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cmd/dedup is the offline maintenance tool for duplicate detection. It reuses the
server's config and runs two phases (both by default; -hashes / -pairs to pick):
- hashes: compute the perceptual hash of every live image/video missing one —
images from their bytes, videos from a middle frame via DiskStorage.
VideoFrameMiddle. Per-file failures are reported and counted, not fatal.
- pairs: rebuild data.duplicate_pairs from all current hashes (DuplicateService.
Rescan).
Idempotent and safe to re-run: hashing only touches NULL phashes, the pairs
rebuild is a full replace. This is how video phashes and any backlog get
computed, and how newly uploaded duplicates become visible.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the duplicate-detection backend on top of perceptual hashing:
- Two tables (edited into the original migrations): data.duplicate_pairs holds
precomputed near-duplicate candidates (rebuilt wholesale by the rescan), and
data.duplicate_dismissals is a global "not a duplicate" overlay that survives
rescans. New audit actions file_merge / duplicate_dismiss.
- DuplicateService:
- Rescan builds every pair within DUPLICATE_HASH_THRESHOLD via a BK-tree over
the perceptual hashes and replaces the pairs table. This is the only thing
that populates pairs, so GET never compares all-vs-all (scales to 110k+).
- Clusters reads the precomputed pairs (ACL-filtered, non-trashed, non-
dismissed), groups them into connected components via union-find, and
paginates whole clusters.
- Resolve merges a pair field-by-field: each scalar from keep or discard,
metadata keep/discard/shallow-merge, tags/pools keep or union; then trashes
the discarded file. Enforces edit ACL on both.
- Dismiss records a canonical pair (view ACL on both).
- Endpoints under /files: GET /files/duplicates, POST /files/duplicates/dismiss,
POST /files/duplicates/resolve (registered before /:id to avoid collision).
Plain delete reuses /files/bulk/delete.
- Repo support: ListMissingPHash, ListAllPHashes, CopyPoolMemberships, plus the
DuplicatePairRepo (ReplaceAll via COPY, ListVisible) and DismissalRepo.
Unit tests cover the BK-tree pairing, union-find clustering, metadata merge and
field validation; an integration test covers rescan -> list -> merge -> dismiss
(including that a dismissal survives a re-rescan).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a 64-bit dHash perceptual hash (internal/imagehash, built on the existing
disintegration/imaging — no new dependency) and starts populating the long-unused
data.files.phash column:
- Upload sets phash inline for images (cheap, from the in-memory bytes).
- Replace recomputes it from new content for images and clears it for anything
else, so a stale hash never survives a content swap.
- FileRepo.SetPHash sets/clears the hash (used by Replace and, later, the dedup
backfill).
- DiskStorage.VideoFrameMiddle extracts a frame from the middle of a clip
(ffprobe duration -> ffmpeg -ss duration/2), avoiding the shared-intro collision
a fixed early offset causes. It is a concrete method, not part of the storage
port: only the dedup CLI needs it, keeping ffmpeg off the upload path. Video
phashes are therefore computed by that CLI, not at upload time.
- DUPLICATE_HASH_THRESHOLD config (default 10/64) for the later pair rescan.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The `# syntax=docker/dockerfile:1` line made BuildKit fetch its frontend
image from Docker Hub on every build, even when all base images and layers
were already cached. On a host that briefly can't resolve registry-1.docker.io
this is the first and only mandatory network round-trip, so the build fails at
"resolve image config for docker-image://docker.io/docker/dockerfile:1" before
any stage runs.
This Dockerfile uses no frontend-specific syntax (no heredocs, no RUN --mount,
no COPY --link) — only multi-stage, COPY --from/--chown, RUN, ENV, etc., all
handled by the engine's built-in frontend. Dropping the directive removes the
Docker Hub dependency and lets a fully cached build complete offline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tag pooled connections with application_name="tanabata-backend" via the
parsed pgxpool config, so the backend's sessions are identifiable in
pg_stat_activity and server logs. An application_name supplied in the
DSN (or PGAPPNAME) still takes precedence.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The API client only skipped JSON parsing for 204, so a success with an
empty body (e.g. 201 from POST /pools/:id/files) hit res.json() on an
empty stream, threw, and surfaced as "Failed to add to pool" even though
the add had committed. Read the body as text and parse only when present.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The manifest's largest, maskable icon (ms-icon-310x310) was transparent,
so installing the PWA produced a transparent app icon. Generate opaque
192/512 "any" + maskable icons from favicon-bg.png (solid #524B6B
background, maskable variants padded into the inner 80% safe zone) and
point the manifest at them instead of the transparent ms-icon entry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mirror the Files grid's roving keyboard focus on the tag and category
lists (and the tags shown on a category page): arrows move a focus ring,
Enter opens the focused item, "/" jumps to search, Escape drops the ring.
Extracts the model into a reusable createRovingGrid controller; vertical
movement is geometric since the pills wrap at variable widths. The
tag/category edit pages gain Escape-to-leave parity with the file viewer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The grid windowed to ~4 viewports, dropping off-screen rows as it grew.
That broke large multi-selects: range/drag selection could not span past
trimmed cards, which silently vanished mid-scroll. Accumulate all loaded
rows for the visit instead; the grid is still cleared on sort/filter
change and on leaving the page (reset effect + section cache). Removes
the now-dead trim/anchor-refill fallbacks in loadMore/loadPrev.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The DSL already supported m~/m= tokens but the filter UI had no way to
add them. Add Images/Video quick buttons and a free-text MIME input that
append m~<pattern> tokens (LIKE on the type name), plus friendly token
labels in dsl.ts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FilterBar gains an Any/Needs review/Reviewed segment (r=1/r=0 token);
FileCard shows a "needs review" dot; FileViewer gets a header toggle that
propagates back to the grid; SelectionBar gains a bulk "Mark reviewed"
action. Adds a --color-success theme token.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add File.needs_review, the POST /files/bulk/review path, and the r=1/r=0
filter tokens to the DSL description.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the old "untagged" sentinel tag with a proper per-file workflow
status: needs_review starts true on upload/import and is cleared by an
explicit action (no auto-clear on tagging). Surfaced as a filter token
(r=1 needs review, r=0 done) so it combines with tag/MIME conditions, and
toggled via POST /files/bulk/review (single id or many, edit-ACL enforced,
audit-logged as file_review).
needs_review lives on data.files (column added to the original 003 migration,
partial index in 006, action type seeded in 007).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
os.ReadDir returns entries in name order; sort them by ascending mtime
before importing so each Upload's created_at reflects the files'
chronological order. mtimes are cached once for the sort and reused as the
content_datetime fallback, dropping the redundant per-file Info() call.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CreateRule accepted apply_to_existing but ignored it, so enabling the
checkbox while creating a rule never retroactively tagged files already
carrying the when-tag — only activating an existing rule did. Extract the
retroactive expansion into TagRuleRepo.ApplyToExisting (reused by SetActive)
and call it from CreateRule when the rule is active, inside one transaction
so a file is never left half-tagged. Mirrors SetRuleActive semantics,
including following only active downstream rules.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every duration in the config is a token TTL (access, refresh, content). A zero
or negative value mints already-expired tokens — no login, no media playback —
and previously loaded silently. parseDuration now rejects <= 0 with a clear
error, so misconfiguration fails fast at startup instead of mysteriously at
runtime. The AuthService itself stays permissive (it's constructed directly in
tests with arbitrary TTLs); config load is the gate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add POST /files/{file_id}/content-token to the spec, note that the content
GET's access_token parameter also accepts a content token, and document the
CONTENT_TOKEN_TTL knob (default 6h) and its leak/revocation trade-off.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mint a content token on file load (POST /files/:id/content-token) and put it in
the original-content URL instead of the access token, so opening an original —
especially a long video — in a new tab keeps working past the 15-minute access
token expiry. Falls back to the access token until the content token arrives,
and re-mints when paging to another file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Opening an original by URL (?access_token=) baked in the 15-minute access
token, so a long video opened in a new tab stopped streaming once that token
expired mid-playback: the access token can't be refreshed in an already-opened
tab, and its next Range request 401'd.
Add a content token: a signed, single-file capability (typ=content, fid claim)
with its own longer TTL (CONTENT_TOKEN_TTL, default 6h) and — crucially — no
session id, so it survives refresh rotation and outlives the short access TTL.
POST /files/:id/content-token mints one after the same view-ACL check content
serving does; GET /files/:id/content now runs under content-aware auth that
accepts either a normal access token or a content token scoped to that file.
View permission is still enforced against the token's user, so the token only
changes when a file may be read by URL, never which files. It's a bearer
capability for that one file until expiry, hence the bounded, configurable TTL.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root README covering the stack, quick start, and dev commands, plus a Reverse
proxy (nginx) section: client_max_body_size for large uploads, forwarded headers
feeding the rate limiter, and buffering-off for streaming large media.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bind the published port to 127.0.0.1 so the app is reachable only through the
host reverse proxy, not on the LAN/WAN — a 0.0.0.0 publish would also bypass
ufw/firewalld, since Docker's DNAT rules sit ahead of the host firewall.
Split the stack onto two networks with deterministic bridge names: `web`
(dk-tanabata) for the public-facing side, and `backend` (dk-tanabata-bnd,
internal:true) for the private app↔DB tier. The DB sits only on `backend`, which
has no gateway, so it has no route off-host.
Document TRUSTED_PROXIES and the loopback publish in .env.example.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The auth rate limiter keys on c.ClientIP(), but the router was built with
gin.New() and never called SetTrustedProxies — so Gin trusted all proxies by
default. Behind a host reverse proxy that meant the limiter either bucketed
every request under the proxy's IP, or (with the port reachable directly) could
be bypassed by a forged X-Forwarded-For.
NewRouter now takes a trusted-proxy list and configures SetTrustedProxies,
returning an error on an invalid list so misconfiguration fails fast at startup.
The list comes from a new TRUSTED_PROXIES config (CSV of CIDRs/IPs), defaulting
to loopback plus the Docker bridge ranges a host proxy reaches the container
through.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add vips-tools (vipsthumbnail) to the runtime image alongside ffmpeg and
exiftool, and document THUMB_MAX_PIXELS as the pure-Go fallback guard.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Use vipsthumbnail as the primary still-image path for both thumbnails
and previews (shared serveGenerated), falling back to the pure-Go
imaging pipeline when vips isn't on PATH. vips shrinks on load (e.g.
JPEG DCT scaling), so a 200+ Mpx photo is resized in a fraction of the
memory and CPU of a full in-process decode and no longer exceeds the
decode cap — the source that previously got only a placeholder now gets
a real thumbnail and preview. The output JPEG is written straight to the
cache (atomic temp→rename), fit within the target box and never upscaled.
The in-process pixel cap now guards only the pure-Go fallback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Thumbnails/previews are generated lazily per request with no concurrency
limit, and the imaging resize already fans out across every core — so
scrolling to a handful of large images spawned that many all-core,
hundreds-of-MB decodes at once and pegged the server. Add a generation
semaphore (THUMB_CONCURRENCY, default = half the CPUs) so only a bounded
number run at a time; queued requests wait and re-check the cache.
Also raise the decode cap from 64 Mpx to a configurable ~300 Mpx default
(THUMB_MAX_PIXELS) so genuinely large photos (e.g. 13000×17000 ≈ 221
Mpx) get a real thumbnail instead of falling back to a placeholder.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Refresh tokens rotate on every use and each refresh deletes the old
session server-side, so when one tab refreshed, other open tabs were
left holding a dead access token and a rotated-away refresh token —
their next request 401'd and bounced them to the login screen.
Sync the auth store across tabs via the storage event (propagating
logins, refreshes, and logouts), and make refresh race-resilient: if a
refresh fails but a newer token has meanwhile synced in from another
tab, adopt it and retry instead of ending a still-valid session.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A native <input type="color"> always holds a value, so the form always
sent the input's default colour and a tag could never be colourless. Add
a "Color" checkbox gating the swatch: off by default on the new-tag form
(so tags are colourless unless opted in) and initialised from the tag on
the edit form, which can now clear a colour. Sends color: null when off.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The PATCH "clear colour" path sent an empty string, which violates the
hex CHECK constraint and never falls back to the category colour. Map ''
to NULL via NULLIF in the tag insert/update so a cleared or omitted
colour is stored as NULL.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fire POST /pools/{id}/views fire-and-forget after the pool loads, the
same way the file viewer logs file views.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add POST /pools/{id}/views, mirroring the file-view endpoint: it
enforces view ACL and appends a row to activity.pool_views (viewed_at
defaults to statement_timestamp(), so each view is its own history row).
The table existed but nothing wrote to it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract the bottom-sheet pool picker (load, search, add) into a
reusable PoolPicker component and use it both for the grid's bulk
selection and from a new button in the file viewer's top bar, which adds
the single open file to a chosen pool. While the picker is open the
viewer hands it the keyboard so Escape closes the sheet (even from its
search) instead of the viewer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mirror the server's secondary sort in the client-side sortTags (used for
a file's assigned tags) so the assigned and available lists agree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The category_name tag sort now breaks ties within a category by the
tag's own name (same direction), so tags group by category and read
alphabetically inside each group; uncategorized tags stay last.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Listing files with a tag filter now logs each referenced tag to
activity.tag_uses, flagging it included (positive) or excluded (negated
under an odd number of NOTs); the untagged pseudo-token is skipped. The
filter AST is reused to determine polarity, so grouped negations like
!(A|B) mark both tags excluded.
Recording happens only when a filter is first applied — not on cursor
pagination or an anchored return — so one browse counts once. The write
is best-effort and never fails the listing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In the single-file viewer's tag filter, Escape now clears a non-empty
filter first; on an empty filter it blurs and scrolls the preview back
to the top. In the bulk (multi-file) editor it clears, then releases
focus, so only the next Escape reaches the page handler and closes the
popup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Describe /files/import's application/x-ndjson response and the start/file/
done/error event schema.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consume the import endpoint's NDJSON progress stream via a new postStream
client helper (reuses the bearer token and 401 refresh, but keeps the body
as a stream). The Settings import card now renders a live progress bar
(processed/total) and a scrolling per-file list where each entry shows its
status — imported, skipped or error — with the failure reason inline and the
newest row kept in view. A final summary replaces the old single-shot result.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
H1K0