tech/tanabata
1
1
Fork 0
Code Pull Requests Actions Releases 1 Activity

docs(project): sync docs with code and bump to 3.0.0

Browse Source 
Prepare the 3.0.0 release:
- Bump the version in openapi.yaml and frontend/package.json to 3.0.0.
- Document the existing GET /health endpoint in openapi.yaml (served at the
  root, outside /api/v1) and refine the auth note.
- Add docs/REQUIREMENTS.md (product requirements, in English) and
  docs/ARCHITECTURE.md (system overview); remove the old Russian
  docs/Описание.md.
- Rewrite GO_PROJECT_STRUCTURE.md and FRONTEND_STRUCTURE.md to match the
  current code (dedup CLI, imagehash, real components/stores, Tailwind v4).
- Fix stale counts and references in CLAUDE.md and link the new docs from
  README.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Masahiko AMANO
2026-07-03 08:35:16 +03:00
parent 4883820f25
commit 28f5b5d150
9 changed files with 740 additions and 626 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/GO_PROJECT_STRUCTURE.md
+168 -156
View File
@@ -1,18 +1,28 @@
# Tanabata File Manager — Go Project Structure
> Backend counterpart of [ARCHITECTURE.md](ARCHITECTURE.md). This document
> details the Go layout, the layer rules, and the key backend decisions.
## Stack
- **Router**: Gin
- **Database**: pgx v5 (pgxpool)
- **Migrations**: goose v3 + go:embed (auto-migrate on startup)
- **Auth**: JWT (golang-jwt/jwt/v5)
- **Config**: environment variables via .env (joho/godotenv)
- **Logging**: slog (stdlib, Go 1.21+)
- **Validation**: go-playground/validator/v10
- **EXIF**: rwcarlsen/goexif or dsoprea/go-exif
- **Image processing**: disintegration/imaging (thumbnails, previews)
- **Migrations**: goose v3 + `go:embed` (auto-applied on startup)
- **Auth**: JWT (golang-jwt/jwt/v5), Bearer access tokens + rotating refresh tokens
- **Config**: environment variables via `.env` (joho/godotenv)
- **Logging**: slog (stdlib)
- **Metadata**: exiftool (external, preferred) with a pure-Go EXIF fallback
(rwcarlsen/goexif)
- **Thumbnails / previews**: vipsthumbnail (external, shrink-on-load) and ffmpeg
(video frames), with a pure-Go fallback (disintegration/imaging)
- **Near-duplicate detection**: 64-bit dHash perceptual hashing + a BK-tree /
Hamming-distance pairing (`internal/imagehash`, `internal/service/duplicate_*`)
- **Architecture**: Clean Architecture (domain → service → repository/handler)
The binary is fully static (`CGO_ENABLED=0`). External tools are invoked as
subprocesses when present and are optional — the pure-Go paths keep the server
working without them.
## Monorepo Layout
```
@@ -31,81 +41,98 @@ tanabata/
```
backend/
├── cmd/
│ └── server/
│ └── main.go # Entrypoint: config → DB → migrate → wire → run
│ ├── server/
│ │ └── main.go # Entrypoint: config → DB → migrate → bootstrap admin → wire → serve
│ └── dedup/
│ └── main.go # Offline maintenance CLI: perceptual-hash backfill + duplicate-pairs rescan
│
├── internal/
│ │
│ ├── domain/ # Pure business entities & value objects
│ │ ├── file.go # File, FileFilter, FilePage
│ ├── domain/ # Pure business entities & value objects (stdlib only)
│ │ ├── file.go # File, FileFilter, FileListParams, FilePage
│ │ ├── tag.go # Tag, TagRule
│ │ ├── category.go # Category
│ │ ├── pool.go # Pool, PoolFile
│ │ ├── user.go # User, Session
│ │ ├── acl.go # Permission, ObjectType
│ │ ├── audit.go # AuditEntry, ActionType
│ │ └── errors.go # Domain error types (ErrNotFound, ErrForbidden, etc.)
│ │ ├── duplicate.go # DuplicatePair, PHashEntry
│ │ ├── context.go # WithUser / UserFromContext (identity + session in ctx)
│ │ └── errors.go # Domain error types (ErrNotFound, ErrForbidden, …)
│ │
│ ├── port/ # Interfaces (ports) — contracts between layers
│ │ ├── repository.go # FileRepo, TagRepo, CategoryRepo, PoolRepo,
│ │ │ # UserRepo, SessionRepo, ACLRepo, AuditRepo,
│ │ │ # MimeRepo, TagRuleRepo
│ │ └── storage.go # FileStorage interface (disk operations)
│ │ ├── repository.go # Transactor, FileRepo, TagRepo, TagRuleRepo, CategoryRepo,
│ │ │ # PoolRepo, UserRepo, SessionRepo, ACLRepo, AuditRepo,
│ │ │ # MimeRepo, DuplicatePairRepo, DismissalRepo
│ │ └── storage.go # FileStorage (originals + thumbnail/preview cache)
│ │
│ ├── service/ # Business logic (use cases)
│ │ ├── file_service.go # Upload, update, delete, trash/restore, replace,
│ │ │ # import, filter/list, duplicate detection
│ │ ├── tag_service.go # CRUD + auto-tag application logic
│ │ ├── category_service.go # CRUD (thin, delegates to repo + ACL + audit)
│ │ ├── file_service.go # Upload, update, delete, trash/restore, replace, import, filter/list
│ │ ├── tag_service.go # CRUD + auto-tag (rule) application
│ │ ├── category_service.go # CRUD (thin: repo + ACL + audit)
│ │ ├── pool_service.go # CRUD + file ordering, add/remove files
│ │ ├── auth_service.go # Login, logout, JWT issue/refresh, session management
│ │ ├── auth_service.go # Login, logout, JWT issue/refresh, content tokens, sessions
│ │ ├── acl_service.go # Permission checks, grant/revoke
│ │ ├── audit_service.go # Log actions, query audit log
│ │ └── user_service.go # Profile update, admin CRUD, block/unblock
│ │ ├── user_service.go # Profile update, admin CRUD, block/unblock, EnsureAdmin
│ │ ├── duplicate_service.go # Cluster / resolve (merge) / dismiss + rescan orchestration
│ │ ├── duplicate_index.go # BK-tree, Hamming pairing, connected-component clustering
│ │ └── metadata.go # EXIF / media metadata extraction (exiftool + pure-Go fallback)
│ │
│ ├── handler/ # HTTP layer (Gin handlers)
│ │ ├── router.go # Route registration, middleware wiring
│ │ ├── middleware.go # Auth middleware (JWT extraction → context)
│ │ ├── request.go # Common request parsing helpers
│ │ ├── response.go # Error/success response builders
│ │ ├── router.go # Route registration, middleware, security headers, SPA fallback
│ │ ├── middleware.go # Auth middleware (JWT / content token → context)
│ │ ├── ratelimit.go # Per-IP token-bucket limiter for /auth
│ │ ├── response.go # Error/success builders, domain-error → HTTP mapping
│ │ ├── static.go # Built SPA serving + index.html fallback
│ │ ├── file_handler.go # /files endpoints
│ │ ├── tag_handler.go # /tags endpoints
│ │ ├── duplicate_handler.go # /files/duplicates endpoints
│ │ ├── tag_handler.go # /tags endpoints (+ file–tag relations)
│ │ ├── category_handler.go # /categories endpoints
│ │ ├── pool_handler.go # /pools endpoints
│ │ ├── auth_handler.go # /auth endpoints
│ │ ├── acl_handler.go # /acl endpoints
│ │ ├── user_handler.go # /users endpoints
│ │ └── audit_handler.go # /audit endpoints
│ │ └── audit_handler.go # /audit endpoint
│ │
│ ├── db/ # Database adapters
│ │ ├── db.go # Common helpers: pagination, repo factory, transactor base
│ │ ├── db.go # Shared helpers: Querier, tx-from-context, ScanRow, limit/offset clamps
│ │ └── postgres/ # PostgreSQL implementation
│ │ ├── postgres.go # pgxpool init, tx-from-context helpers
│ │ ├── file_repo.go # FileRepo implementation
│ │ ├── tag_repo.go # TagRepo + TagRuleRepo implementation
│ │ ├── category_repo.go # CategoryRepo implementation
│ │ ├── pool_repo.go # PoolRepo implementation
│ │ ├── user_repo.go # UserRepo implementation
│ │ ├── session_repo.go # SessionRepo implementation
│ │ ├── acl_repo.go # ACLRepo implementation
│ │ ├── audit_repo.go # AuditRepo implementation
│ │ ├── mime_repo.go # MimeRepo implementation
│ │ └── filter_parser.go # DSL → SQL WHERE clause builder
│ │ ├── postgres.go # pgxpool init, Transactor, conn-or-tx helper
│ │ ├── file_repo.go # FileRepo (incl. perceptual-hash projections)
│ │ ├── tag_repo.go # TagRepo + TagRuleRepo
│ │ ├── category_repo.go # CategoryRepo
│ │ ├── pool_repo.go # PoolRepo
│ │ ├── user_repo.go # UserRepo
│ │ ├── session_repo.go # SessionRepo
│ │ ├── acl_repo.go # ACLRepo
│ │ ├── audit_repo.go # AuditRepo
│ │ ├── mime_repo.go # MimeRepo
│ │ ├── duplicate_repo.go # DuplicatePairRepo + DismissalRepo
│ │ └── filter_parser.go # Filter DSL → SQL WHERE clause builder
│ │
│ ├── storage/ # File storage adapter
│ │ └── disk.go # FileStorage implementation (read/write/delete on disk)
│ │ └── disk.go # FileStorage on disk: originals + thumbnail/preview cache
│ │ # (vipsthumbnail / ffmpeg / pure-Go imaging)
│ │
│ ├── imagehash/ # Perceptual hashing (64-bit dHash) for near-duplicate detection
│ │ └── imagehash.go
│ │
│ ├── integration/ # End-to-end HTTP tests against a disposable Postgres
│ │ └── server_test.go
│ │
│ └── config/ # Configuration
│ └── config.go # Struct + loader from env vars
│ └── config.go # Config struct + loader from env vars
│
├── migrations/ # SQL migration files (goose format)
├── migrations/ # SQL migration files (goose format), embedded via go:embed
│ ├── 001_init_schemas.sql
│ ├── 002_core_tables.sql
│ ├── 003_data_tables.sql
│ ├── 004_acl_tables.sql
│ ├── 005_activity_tables.sql
│ ├── 006_indexes.sql
│ └── 007_seed_data.sql
│ ├── 007_seed_data.sql
│ └── embed.go # //go:embed *.sql → migrations.FS
│
├── go.mod
└── go.sum
@@ -126,6 +153,7 @@ handler → service → port (interfaces) ← db/postgres / storage
- **db/postgres/**: imports domain/, port/, and db/ (common helpers). Implements port interfaces.
- **db/**: imports domain/ and port/. Shared utilities for all DB adapters.
- **storage/**: imports domain/ and port/. Implements FileStorage.
- **imagehash/**: leaf package (stdlib + image libs); used by service/ and storage/.
No layer may import a layer above it. No circular dependencies.
@@ -133,138 +161,126 @@ No layer may import a layer above it. No circular dependencies.
### Dependency Injection (Wiring)
Manual wiring in `cmd/server/main.go`. No DI frameworks.
Manual wiring in `cmd/server/main.go`. No DI frameworks. Constructors take their
collaborators explicitly; the shape below matches the real signatures.
```go
// Pseudocode
pool := postgres.NewPool(cfg.DatabaseURL)
goose.Up(pool, migrations)
// Repos (all from internal/db/postgres/)
fileRepo := postgres.NewFileRepo(pool)
tagRepo := postgres.NewTagRepo(pool)
// ...
// Pseudocode — see cmd/server/main.go for the exact calls.
pool := postgres.NewPool(ctx, cfg.DatabaseURL)
goose.Up(stdlib.OpenDBFromPool(pool), ".") // migrations.FS embedded
// Storage
diskStore := storage.NewDiskStorage(cfg.FilesPath)
diskStorage := storage.NewDiskStorage(
cfg.FilesPath, cfg.ThumbsCachePath,
cfg.ThumbWidth, cfg.ThumbHeight, cfg.PreviewWidth, cfg.PreviewHeight,
cfg.ThumbMaxPixels, cfg.ThumbConcurrency,
)
// Repos (all from internal/db/postgres/)
fileRepo := postgres.NewFileRepo(pool)
// … tag, tagRule, category, pool, user, session, acl, audit, mime,
// duplicatePair, dismissal repos + transactor
// Services
aclSvc := service.NewACLService(aclRepo, objectTypeRepo)
auditSvc := service.NewAuditService(auditRepo, actionTypeRepo)
fileSvc := service.NewFileService(fileRepo, mimeRepo, tagRepo, diskStore, aclSvc, auditSvc)
tagSvc := service.NewTagService(tagRepo, tagRuleRepo, aclSvc, auditSvc)
// ...
authSvc := service.NewAuthService(userRepo, sessionRepo,
cfg.JWTSecret, cfg.JWTAccessTTL, cfg.JWTRefreshTTL, cfg.ContentTokenTTL)
aclSvc := service.NewACLService(aclRepo, fileRepo, tagRepo, categoryRepo, poolRepo, transactor)
auditSvc := service.NewAuditService(auditRepo)
tagSvc := service.NewTagService(tagRepo, tagRuleRepo, aclSvc, auditSvc, transactor)
dupSvc := service.NewDuplicateService(fileRepo, duplicatePairRepo, dismissalRepo,
aclSvc, auditSvc, transactor, cfg.DuplicateHashThreshold)
fileSvc := service.NewFileService(fileRepo, mimeRepo, diskStorage,
aclSvc, auditSvc, tagSvc, transactor, cfg.ImportPath)
// … category, pool, user services
// Handlers
fileHandler := handler.NewFileHandler(fileSvc, tagSvc)
// ...
// Bootstrap the initial admin from env (idempotent).
userSvc.EnsureAdmin(ctx, cfg.AdminUsername, cfg.AdminPassword)
router := handler.NewRouter(cfg, fileHandler, tagHandler, ...)
router.Run(cfg.ListenAddr)
// Handlers → router (also wires trusted proxies + optional static SPA dir)
router, _ := handler.NewRouter(authMiddleware, authHandler, fileHandler,
duplicateHandler, tagHandler, categoryHandler, poolHandler,
userHandler, aclHandler, auditHandler, cfg.StaticDir, cfg.TrustedProxies)
srv.ListenAndServe()
```
### Context Propagation
Every service method receives `context.Context` as the first argument.
The handler extracts user info from JWT (via middleware) and puts it
into context. Services read the current user from context for ACL checks
and audit logging.
The auth middleware parses the JWT and puts the caller's identity (user id,
admin flag, session id) into the context. Services read it for ACL checks and
audit logging.
```go
// middleware.go
func (m *AuthMiddleware) Handle(c *gin.Context) {
claims := parseJWT(c.GetHeader("Authorization"))
ctx := domain.WithUser(c.Request.Context(), claims.UserID, claims.IsAdmin)
c.Request = c.Request.WithContext(ctx)
c.Next()
}
// handler/middleware.go
claims := parseJWT(c.GetHeader("Authorization"))
ctx := domain.WithUser(c.Request.Context(), claims.UserID, claims.IsAdmin, claims.SessionID)
c.Request = c.Request.WithContext(ctx)
// domain/context.go
type ctxKey int
const userKey ctxKey = iota
func WithUser(ctx context.Context, userID int16, isAdmin bool) context.Context { ... }
func UserFromContext(ctx context.Context) (userID int16, isAdmin bool) { ... }
func WithUser(ctx context.Context, userID int16, isAdmin bool, sessionID int) context.Context
func UserFromContext(ctx context.Context) (userID int16, isAdmin bool, sessionID int)
```
### Transaction Management
Repository interfaces include a `Transactor`:
The `Transactor` port lets services compose multiple repo calls atomically.
The postgres implementation stores the active `pgx.Tx` in the context; repo
methods pick it up via a conn-or-tx helper, so the same method works inside or
outside a transaction.
```go
// port/repository.go
type Transactor interface {
WithTx(ctx context.Context, fn func(ctx context.Context) error) error
}
```
The postgres implementation wraps `pgxpool.Pool.BeginTx`. Inside `fn`,
all repo calls use the transaction from context. This allows services
to compose multiple repo calls in a single transaction:
```go
// service/file_service.go
func (s *FileService) Upload(ctx context.Context, input UploadInput) (*domain.File, error) {
// service/file_service.go (sketch)
func (s *FileService) Upload(ctx context.Context, p UploadParams) (*domain.File, error) {
return s.tx.WithTx(ctx, func(ctx context.Context) error {
file, err := s.fileRepo.Create(ctx, ...) // uses tx
if err != nil { return err }
for _, tagID := range input.TagIDs {
s.tagRepo.AddFileTag(ctx, file.ID, tagID) // same tx
}
s.auditRepo.Log(ctx, ...) // same tx
return nil
created, err := s.files.Create(ctx, f) // uses tx from ctx
// apply initial tags, etc., in the same tx
return err
})
}
```
### ACL Check Pattern
ACL logic is centralized in `ACLService`. Other services call it before
any data mutation or retrieval:
ACL logic is centralized in `ACLService`. Other services call it before any
mutation or retrieval. The model is private-by-default: admins see everything;
otherwise a `public` flag, creator ownership, or an explicit `acl.permissions`
grant is required.
```go
// service/acl_service.go
func (s *ACLService) CanView(ctx context.Context, objectType string, objectID uuid.UUID) error {
userID, isAdmin := domain.UserFromContext(ctx)
if isAdmin { return nil }
// Check is_public on the object
// If not public, check creator_id == userID
// If not creator, check acl.permissions
// Return domain.ErrForbidden if none match
}
// service/acl_service.go (shape)
func (s *ACLService) CanView(ctx context.Context, userID int16, isAdmin bool,
creatorID int16, isPublic bool, objectType int16, objectID uuid.UUID) (bool, error)
func (s *ACLService) CanEdit(ctx context.Context, userID int16, isAdmin bool,
creatorID int16, objectType int16, objectID uuid.UUID) (bool, error)
```
### Error Mapping
Domain errors → HTTP status codes (handled in handler/response.go):
| Domain Error | HTTP Status | Error Code |
|-----------------------|-------------|-------------------|
| ErrNotFound | 404 | not_found |
| ErrForbidden | 403 | forbidden |
| ErrUnauthorized | 401 | unauthorized |
| ErrConflict | 409 | conflict |
| ErrValidation | 400 | validation_error |
| ErrUnsupportedMIME | 415 | unsupported_mime |
| (unexpected) | 500 | internal_error |
| Domain Error | HTTP Status | Error Code |
| ------------------ | ----------- | ---------------- |
| ErrNotFound | 404 | not_found |
| ErrForbidden | 403 | forbidden |
| ErrUnauthorized | 401 | unauthorized |
| ErrConflict | 409 | conflict |
| ErrValidation | 400 | validation_error |
| ErrUnsupportedMIME | 415 | unsupported_mime |
| (unexpected) | 500 | internal_error |
### Filter DSL
The DSL parser lives in `db/postgres/filter_parser.go` because it produces
SQL WHERE clauses — it is a PostgreSQL-specific adapter concern.
The service layer passes the raw DSL string to the repository; the
repository parses it and builds the query.
The DSL parser lives in `db/postgres/filter_parser.go` because it produces SQL
WHERE clauses — a PostgreSQL-specific adapter concern. The service layer passes
the raw DSL string down; the repository parses it and builds the query. For a
different DBMS, a corresponding parser would live in `db/<dbms>/filter_parser.go`.
For a different DBMS, a corresponding parser would live in
`db/<dbms>/filter_parser.go`.
The interface:
```go
// port/repository.go
type FileRepo interface {
List(ctx context.Context, params FileListParams) (*domain.FilePage, error)
// ...
}
// domain/file.go
type FileListParams struct {
Filter string // raw DSL string
@@ -279,42 +295,38 @@ type FileListParams struct {
}
```
The DSL grammar itself is documented in `openapi.yaml` (the `filter` query
parameter), so the contract stays in one place.
### JWT Structure
```go
type Claims struct {
jwt.RegisteredClaims
UserID int16 `json:"uid"`
IsAdmin bool `json:"adm"`
SessionID int `json:"sid"`
UserID int16 `json:"uid"`
IsAdmin bool `json:"adm"`
SessionID int `json:"sid"`
}
```
Access token: short-lived (15 min). Refresh token: long-lived (30 days),
stored as hash in `activity.sessions.token_hash`.
Access token: short-lived (15 min default). Refresh token: long-lived (30 days
default), rotated on use, stored as a hash in `activity.sessions`. A separate
**content token** (default 6 h) is a single-file capability minted for media
URLs, so a long video keeps streaming past access-token expiry — see
`CONTENT_TOKEN_TTL` in `.env.example`.
### Perceptual Duplicate Detection
Images are dHash-ed inline on upload (`internal/imagehash`); video hashes are
backfilled by the `dedup` CLI (ffmpeg stays off the upload path). A rescan
rebuilds `data.duplicate_pairs` by inserting every pair within
`DUPLICATE_HASH_THRESHOLD` Hamming distance (BK-tree lookups, not O(N²)); the
duplicates API then groups pairs into connected-component clusters. See
`service/duplicate_service.go` and `service/duplicate_index.go`.
### Configuration (.env)
```env
# Server
LISTEN_ADDR=:42776
JWT_SECRET=<random-32-bytes>
JWT_ACCESS_TTL=15m
JWT_REFRESH_TTL=720h
# Database
DATABASE_URL=postgres://user:pass@host:5432/tanabata?sslmode=disable
# Storage
FILES_PATH=/data/files
THUMBS_CACHE_PATH=/data/thumbs
# Thumbnails
THUMB_WIDTH=160
THUMB_HEIGHT=160
PREVIEW_WIDTH=1920
PREVIEW_HEIGHT=1080
# Import
IMPORT_PATH=/data/import
```
Every variable the server reads is documented in `.env.example` (1:1 with
`config.Config`). Required at startup: `JWT_SECRET`, `ADMIN_PASSWORD`,
`DATABASE_URL`, `FILES_PATH`, `THUMBS_CACHE_PATH`, `IMPORT_PATH`. Everything
else has a sensible default (see `config.go`).