Compare commits
2 Commits
fce71bb946
...
76942721ad
| Author | SHA1 | Date | |
|---|---|---|---|
 H1K0
|
76942721ad | ||
|
H1K0
|
437b66e73a |
@@ -0,0 +1,44 @@
|
||||
name: deploy
|
||||
|
||||
# Build the image and (re)start the compose stack on the production host
|
||||
# whenever master moves. Also runnable manually from the Gitea Actions tab.
|
||||
on:
|
||||
push:
|
||||
branches: [master]
|
||||
workflow_dispatch: {}
|
||||
|
||||
# One deploy at a time; queue rather than cancel an in-flight run.
|
||||
concurrency:
|
||||
group: deploy-prod
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
# Self-hosted act_runner registered on the prod host with the "host" label
|
||||
# (shell executor), so the job uses the host's git + Docker daemon and the
|
||||
# existing clone in /opt/tanabata. See docs/DEPLOY.md for runner setup.
|
||||
#
|
||||
# Only shell steps here (no `uses:` actions), so the host needs git + docker
|
||||
# and nothing else — no node, no rsync.
|
||||
runs-on: host
|
||||
|
||||
env:
|
||||
DEPLOY_DIR: /opt/tanabata
|
||||
|
||||
steps:
|
||||
- name: Pull latest master
|
||||
# DEPLOY_DIR is a git clone set up once at deploy time. reset --hard
|
||||
# makes it match origin exactly; .env is untracked (.gitignore) so it
|
||||
# is never touched.
|
||||
run: |
|
||||
cd "$DEPLOY_DIR"
|
||||
git fetch --prune origin
|
||||
git reset --hard origin/master
|
||||
|
||||
- name: Build image and start the stack
|
||||
working-directory: /opt/tanabata
|
||||
# .env must already exist in DEPLOY_DIR on the host (secrets + DB mode).
|
||||
run: docker compose up -d --build --remove-orphans
|
||||
|
||||
- name: Prune dangling build layers
|
||||
run: docker image prune -f
|
||||
+182
@@ -0,0 +1,182 @@
|
||||
# Deployment (Gitea Actions → host)
|
||||
|
||||
Tanabata is deployed by a [Gitea Actions](https://docs.gitea.com/usage/actions/overview)
|
||||
workflow ([`.gitea/workflows/deploy.yml`](../.gitea/workflows/deploy.yml)) that
|
||||
runs on the **production host itself**. On every push to `master` it updates the
|
||||
git clone in `/opt/tanabata` and runs `docker compose up -d --build` there, so the
|
||||
image is built from the freshly-pushed code and the stack is restarted.
|
||||
|
||||
```
|
||||
push master ──> Gitea (container) ──> act_runner (host, "host" label)
|
||||
│ git fetch + reset --hard (in /opt/tanabata)
|
||||
└ docker compose up -d --build
|
||||
```
|
||||
|
||||
The Gitea server runs in a container, but the **runner runs directly on the host**
|
||||
(shell executor) so it can use the host's git, the host Docker daemon, and the
|
||||
clone in `/opt/tanabata`. Nothing needs a registry — the host builds the image
|
||||
locally. The workflow uses only shell steps, so the host needs just **git** and
|
||||
**docker** (no node, no rsync).
|
||||
|
||||
## What is a runner?
|
||||
|
||||
Gitea (like GitHub) only *coordinates* CI: it stores the workflow, queues jobs,
|
||||
and shows logs. It does **not** execute anything itself. A **runner** is a
|
||||
separate agent program that polls Gitea for queued jobs, runs the steps on a
|
||||
machine you control, and reports results back.
|
||||
|
||||
Gitea's official runner is **act_runner** (a single Go binary; it uses the
|
||||
`act` engine to interpret workflow YAML). One act_runner process can serve many
|
||||
repos. Each runner advertises one or more **labels**, and a job's `runs-on:`
|
||||
picks a runner by label. A label also decides *how* a job runs — the **executor**:
|
||||
|
||||
- **docker executor** — each job runs in a fresh container from an image (e.g.
|
||||
`node:20-bookworm`). Isolated and reproducible; the usual default. Label form
|
||||
at registration: `ubuntu:docker://node:20-bookworm`.
|
||||
- **host / shell executor** — the job runs directly on the host as the runner's
|
||||
user, using host-installed tools. Label form: `host:host`. This is what we use,
|
||||
because the deploy needs the host's Docker daemon and `/opt/tanabata`.
|
||||
|
||||
So `runs-on: host` in the workflow ⇒ "run this job on a runner that registered a
|
||||
`host` label" ⇒ our shell executor on the prod box.
|
||||
|
||||
## One-time setup
|
||||
|
||||
### 1. Enable Actions in Gitea
|
||||
|
||||
Gitea 1.21+ has Actions on by default. Otherwise add to `app.ini` and restart:
|
||||
|
||||
```ini
|
||||
[actions]
|
||||
ENABLED = true
|
||||
```
|
||||
|
||||
### 2. A runner user on the host
|
||||
|
||||
Pick (or create) the Linux user the runner runs as. It must be able to use Docker
|
||||
and own the deploy dir — so the workflow needs no `sudo`:
|
||||
|
||||
```bash
|
||||
sudo useradd -r -m -d /home/gitea-runner gitea-runner # or reuse an existing user
|
||||
sudo usermod -aG docker gitea-runner # host Docker access
|
||||
```
|
||||
|
||||
The host needs `git` and a Docker engine with the Compose plugin:
|
||||
|
||||
```bash
|
||||
sudo apt install -y git docker.io docker-compose-plugin # Debian/Ubuntu
|
||||
```
|
||||
|
||||
### 3. Clone the repo to /opt/tanabata once
|
||||
|
||||
The workflow only does `git fetch` + `reset --hard`, so the clone (and its auth)
|
||||
is established here, once. Use a **read-only deploy key** so the host never holds
|
||||
write credentials:
|
||||
|
||||
```bash
|
||||
# As the runner user, create a key and add the PUBLIC half to the repo in Gitea:
|
||||
# Repo → Settings → Deploy Keys → Add (read-only)
|
||||
sudo -u gitea-runner ssh-keygen -t ed25519 -f /home/gitea-runner/.ssh/tanabata_deploy -N ''
|
||||
|
||||
# Clone with that key (SSH URL of your Gitea repo):
|
||||
sudo -u gitea-runner GIT_SSH_COMMAND='ssh -i /home/gitea-runner/.ssh/tanabata_deploy' \
|
||||
git clone git@gitea.example.com:you/tanabata.git /opt/tanabata
|
||||
sudo chown -R gitea-runner:gitea-runner /opt/tanabata
|
||||
```
|
||||
|
||||
> HTTPS works too — clone with a URL that carries a read-only token. SSH deploy
|
||||
> keys are the cleaner, per-repo, read-only option.
|
||||
|
||||
After cloning, recurring `git fetch` reuses the remote + key stored in
|
||||
`/opt/tanabata/.git/config`, so the runner itself needs no standing credentials.
|
||||
|
||||
### 4. Register and run act_runner on the host
|
||||
|
||||
Get a registration token in Gitea. **Where you create it sets the runner's
|
||||
scope** (and `--name` is only a display label, unrelated to scope):
|
||||
|
||||
- **Repository** (Tanabata repo → Settings → Actions → Runners) → serves only
|
||||
this repo. **Use this.**
|
||||
- Organization → all repos in the org; Site (admin) → all repos on the instance.
|
||||
|
||||
> Security: this runner is a host/shell executor with access to the Docker
|
||||
> socket — effectively root on the host. Register it at the **repository** level
|
||||
> so only Tanabata's workflows can run on your prod server; a site-wide runner
|
||||
> would let any repo's workflow execute arbitrary commands here.
|
||||
|
||||
Then, as the runner user:
|
||||
|
||||
```bash
|
||||
# Download act_runner: https://gitea.com/gitea/act_runner/releases
|
||||
act_runner register --no-interactive \
|
||||
--instance https://gitea.example.com \
|
||||
--token <REGISTRATION_TOKEN> \
|
||||
--name prod-host \
|
||||
--labels host:host # <-- maps `runs-on: host` to the shell executor
|
||||
|
||||
# Run it (use a systemd unit in production so it survives reboots):
|
||||
act_runner daemon
|
||||
```
|
||||
|
||||
`--labels host:host` is what makes jobs run **on the host** instead of in a
|
||||
container. The instance URL must be reachable from the host (Gitea's published
|
||||
port / domain — not the in-container address). Registration writes a `.runner`
|
||||
file (the runner's credentials) in the working directory.
|
||||
|
||||
Minimal systemd unit (`/etc/systemd/system/act_runner.service`):
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Gitea act_runner
|
||||
After=docker.service
|
||||
Requires=docker.service
|
||||
|
||||
[Service]
|
||||
User=gitea-runner
|
||||
WorkingDirectory=/home/gitea-runner
|
||||
ExecStart=/usr/local/bin/act_runner daemon
|
||||
Restart=always
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
```bash
|
||||
sudo systemctl enable --now act_runner
|
||||
```
|
||||
|
||||
### 5. Create /opt/tanabata/.env (secrets)
|
||||
|
||||
The workflow **never** writes `.env` — it lives on the host and holds the real
|
||||
secrets and the chosen DB mode. `.env` is git-ignored, so `git reset --hard`
|
||||
leaves it untouched. Create it once:
|
||||
|
||||
```bash
|
||||
cd /opt/tanabata
|
||||
sudo -u gitea-runner cp .env.example .env
|
||||
sudo -u gitea-runner $EDITOR .env # set JWT_SECRET, ADMIN_PASSWORD, DATABASE_URL, etc.
|
||||
```
|
||||
|
||||
See [`.env.example`](../.env.example) for every variable. For the bundled
|
||||
Postgres keep `COMPOSE_PROFILES=with-db`; to use a Postgres already on the host,
|
||||
set it empty and point `DATABASE_URL` at `host.docker.internal`.
|
||||
|
||||
> Data lives in named Docker volumes by default (or the `*_DIR` host paths you
|
||||
> set in `.env`, e.g. `/var/lib/tanabata/...`) — **not** in `/opt/tanabata`. So
|
||||
> `git reset --hard` on the code dir never touches your data.
|
||||
|
||||
## Deploying
|
||||
|
||||
Push to `master` (or hit **Run workflow** on the Actions tab). Watch progress
|
||||
under the repo's **Actions** tab. The first build pulls the Node/Go base images
|
||||
and takes a few minutes; later builds reuse the host's layer cache.
|
||||
|
||||
## Notes / alternatives
|
||||
|
||||
- **Docker-executor runner instead of host.** If you'd rather the runner itself
|
||||
run in a container, register with a Docker label and bind-mount
|
||||
`/var/run/docker.sock` and `/opt/tanabata` into the job (act_runner
|
||||
`config.yaml` → `container.valid_volumes`), then change `runs-on` accordingly.
|
||||
The host executor above is simpler for host deploys.
|
||||
- **Zero-downtime** isn't attempted: `compose up` recreates changed containers.
|
||||
For a single-node setup the brief restart is usually fine.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,103 @@
|
||||
# Legacy data migration
|
||||
|
||||
Moves data from the **old** Tanabata database (the Python/Flask version, schema
|
||||
in [`docs/reference/schema.sql`](../../docs/reference/schema.sql)) into the
|
||||
**new** `core` / `data` / `acl` / `activity` schema.
|
||||
|
||||
- [`transform.sql`](transform.sql) — the actual data transformation. Reads a
|
||||
`legacy` schema (the old tables) and writes the new schema, in one
|
||||
transaction. Idempotent.
|
||||
- [`migrate.sh`](migrate.sh) — links the new DB to the old one via
|
||||
`postgres_fdw`, imports the old `public` schema as `legacy`, runs
|
||||
`transform.sql`, then removes the link. The old DB is only **read**.
|
||||
|
||||
Tested end-to-end against PostgreSQL 16 (schema applied, synthetic legacy data,
|
||||
all transformations + idempotency verified).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. The **new** schema exists and is seeded — start the app once (it runs the
|
||||
goose migrations incl. `007_seed_data`), or run goose manually.
|
||||
2. `NEW_DSN` connects as a role allowed to `CREATE EXTENSION postgres_fdw`
|
||||
(a superuser — the compose Postgres' `POSTGRES_USER` is one).
|
||||
3. The new Postgres server can reach the old DB host over the network.
|
||||
4. `psql` on PATH.
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
cd scripts/migrate-legacy
|
||||
|
||||
NEW_DSN='postgres://tanabata:PASS@localhost:42777/tanabata' \
|
||||
OLD_HOST=192.168.1.10 OLD_PORT=5432 OLD_DB=tfm \
|
||||
OLD_USER=hiko OLD_PASSWORD=SECRET \
|
||||
./migrate.sh
|
||||
```
|
||||
|
||||
It prints the source (legacy) row counts, then the resulting new-schema counts.
|
||||
Re-running is safe — `ON CONFLICT DO NOTHING` everywhere means a second run only
|
||||
fills in what is missing.
|
||||
|
||||
### Without postgres_fdw
|
||||
|
||||
`transform.sql` only needs the old tables to be visible as a `legacy` schema. If
|
||||
you'd rather not use fdw, load the old dump into a schema named `legacy` in the
|
||||
new database by whatever means, then run just the transform:
|
||||
|
||||
```bash
|
||||
psql "$NEW_DSN" -v ON_ERROR_STOP=1 -f transform.sql
|
||||
```
|
||||
|
||||
## What gets migrated, and how
|
||||
|
||||
| Old (`public`) | New | Notes |
|
||||
|-----------------------|-------------------------|-------|
|
||||
| `users` | `core.users` | id **uuid → smallint** (remapped by unique `name`); `can_edit` → `can_create`; `is_blocked` = false |
|
||||
| `mime` | `core.mime_types` | id **uuid → smallint** (remapped by `name`); types not already seeded are added |
|
||||
| `categories` | `data.categories` | id kept; `is_private` → **`is_public`** (inverted) |
|
||||
| `tags` | `data.tags` | id + `category_id` kept; inverted privacy |
|
||||
| `autotags` | `data.tag_rules` | `parent_id` → `when_tag_id`, `child_id` → `then_tag_id` |
|
||||
| `files` | `data.files` | id kept; `datetime` → `content_datetime`; `orig_name` → `original_name`; **EXIF** lifted from `metadata->'exif'` into the `exif` column, the rest stays as user `metadata` |
|
||||
| `file_tag` | `data.file_tag` | orphan rows skipped |
|
||||
| `pools` | `data.pools` | id kept; `parent_id` + `created` preserved under `metadata` (see below) |
|
||||
| `file_pool` | `data.file_pool` | `position` synthesised (gapped 1000s, ordered by file id) |
|
||||
| `acl` | `acl.permissions` | object type **derived** by locating the object; `read`/`write` → `can_view`/`can_edit` |
|
||||
| `file_views` | `activity.file_views` | `datetime` → `viewed_at` |
|
||||
|
||||
Throughout: empty `notes` (`''`) → `NULL`; colours that aren't 6-hex are set to
|
||||
`NULL` (the old `CHECK` was `NOT VALID`, so bad values could exist).
|
||||
|
||||
### Decisions / lossy points
|
||||
|
||||
- **Passwords** are copied verbatim. If the old hashes are bcrypt (as the new
|
||||
app expects) logins keep working; otherwise affected users need a reset.
|
||||
- **`created` timestamps** on categories/tags/files are dropped — their UUIDv7
|
||||
ids already encode creation time. Pools use random v4 ids, so their `created`
|
||||
(and the dropped **pool hierarchy** `parent_id`) are preserved under
|
||||
`data.pools.metadata` as `legacy_created` / `legacy_parent_id`.
|
||||
- **`file_pool` ordering**: the old schema stored none, so position is generated
|
||||
from file-id order (≈ chronological) with gaps of 1000.
|
||||
- **Not migrated**: `sessions` / `user_agents` — the new app uses JWTs, so users
|
||||
simply log in again. There were no audit-log / pool-view / tag-use tables in
|
||||
the old schema, so those start empty. `phash` and `is_deleted` are new
|
||||
(`NULL` / `false`).
|
||||
|
||||
## Physical files (separate, manual)
|
||||
|
||||
The script migrates the **database only**. File blobs must be copied too. The
|
||||
new layout stores originals at `FILES_PATH/{uuid}` with **no extension**;
|
||||
thumbnails/previews are regenerated on demand, so don't copy those. Because ids
|
||||
are preserved, the old `{uuid}.{ext}` files map 1:1 — just strip the extension:
|
||||
|
||||
```bash
|
||||
OLD_FILES=/srv/old-tanabata/files # old originals ({uuid}.{ext})
|
||||
NEW_FILES=/var/lib/tanabata/files # new FILES_PATH
|
||||
|
||||
for src in "$OLD_FILES"/*; do
|
||||
id="$(basename "$src")"; id="${id%.*}" # uuids contain no dots
|
||||
cp -n "$src" "$NEW_FILES/$id"
|
||||
done
|
||||
|
||||
# Make them readable by the container user (uid/gid 42776):
|
||||
chown -R 42776:42776 "$NEW_FILES"
|
||||
```
|
||||
Executable
+92
@@ -0,0 +1,92 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# Tanabata legacy -> new schema migration (orchestrator)
|
||||
#
|
||||
# Connects the NEW database to the OLD one via postgres_fdw, imports the old
|
||||
# `public` schema as `legacy`, runs transform.sql (the actual data move, in one
|
||||
# transaction), then tears the foreign link down again. The OLD database is
|
||||
# only read.
|
||||
#
|
||||
# Prerequisites:
|
||||
# - The NEW schema already exists and is seeded (start the app once, or run
|
||||
# goose, so all migrations incl. 007_seed_data have applied).
|
||||
# - NEW_DSN connects as a role allowed to CREATE EXTENSION postgres_fdw
|
||||
# (a superuser; the compose Postgres' POSTGRES_USER is one).
|
||||
# - The NEW Postgres server can reach OLD_HOST:OLD_PORT over the network.
|
||||
# - `psql` is on PATH.
|
||||
#
|
||||
# Usage:
|
||||
# NEW_DSN='postgres://tanabata:pass@localhost:42777/tanabata' \
|
||||
# OLD_HOST=192.168.1.10 OLD_DB=tfm OLD_USER=hiko OLD_PASSWORD=secret \
|
||||
# ./migrate.sh
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
|
||||
# --- Config from the environment --------------------------------------------
|
||||
NEW_DSN="${NEW_DSN:?set NEW_DSN to the new database connection string}"
|
||||
OLD_HOST="${OLD_HOST:?set OLD_HOST}"
|
||||
OLD_PORT="${OLD_PORT:-5432}"
|
||||
OLD_DB="${OLD_DB:?set OLD_DB (old database name)}"
|
||||
OLD_USER="${OLD_USER:?set OLD_USER}"
|
||||
OLD_PASSWORD="${OLD_PASSWORD:?set OLD_PASSWORD}"
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
TRANSFORM_SQL="$SCRIPT_DIR/transform.sql"
|
||||
|
||||
psql_new() { psql "$NEW_DSN" -v ON_ERROR_STOP=1 "$@"; }
|
||||
|
||||
# --- Always remove the foreign link on exit, success or failure -------------
|
||||
teardown() {
|
||||
psql "$NEW_DSN" -q >/dev/null 2>&1 <<'SQL' || true
|
||||
DROP SCHEMA IF EXISTS legacy CASCADE;
|
||||
DROP SERVER IF EXISTS legacy_src CASCADE;
|
||||
SQL
|
||||
}
|
||||
trap teardown EXIT
|
||||
|
||||
echo ">> Linking NEW database to OLD ($OLD_USER@$OLD_HOST:$OLD_PORT/$OLD_DB) via postgres_fdw ..."
|
||||
psql_new \
|
||||
-v old_host="$OLD_HOST" \
|
||||
-v old_port="$OLD_PORT" \
|
||||
-v old_db="$OLD_DB" \
|
||||
-v old_user="$OLD_USER" \
|
||||
-v old_pw="$OLD_PASSWORD" <<'SQL'
|
||||
CREATE EXTENSION IF NOT EXISTS postgres_fdw;
|
||||
|
||||
-- Start clean in case a previous run was interrupted.
|
||||
DROP SCHEMA IF EXISTS legacy CASCADE;
|
||||
DROP SERVER IF EXISTS legacy_src CASCADE;
|
||||
|
||||
CREATE SERVER legacy_src FOREIGN DATA WRAPPER postgres_fdw
|
||||
OPTIONS (host :'old_host', port :'old_port', dbname :'old_db');
|
||||
|
||||
-- :'old_user' / :'old_pw' are quoted+escaped by psql, so passwords with
|
||||
-- special characters are safe.
|
||||
CREATE USER MAPPING FOR CURRENT_USER SERVER legacy_src
|
||||
OPTIONS (user :'old_user', password :'old_pw');
|
||||
|
||||
CREATE SCHEMA legacy;
|
||||
IMPORT FOREIGN SCHEMA public LIMIT TO (
|
||||
users, mime, categories, tags, autotags, files, file_tag, pools, file_pool, acl, file_views
|
||||
) FROM SERVER legacy_src INTO legacy;
|
||||
SQL
|
||||
|
||||
echo ">> Source (legacy) row counts:"
|
||||
psql_new -P pager=off -c "
|
||||
SELECT 'users' AS table, count(*) FROM legacy.users
|
||||
UNION ALL SELECT 'mime', count(*) FROM legacy.mime
|
||||
UNION ALL SELECT 'categories', count(*) FROM legacy.categories
|
||||
UNION ALL SELECT 'tags', count(*) FROM legacy.tags
|
||||
UNION ALL SELECT 'autotags', count(*) FROM legacy.autotags
|
||||
UNION ALL SELECT 'files', count(*) FROM legacy.files
|
||||
UNION ALL SELECT 'file_tag', count(*) FROM legacy.file_tag
|
||||
UNION ALL SELECT 'pools', count(*) FROM legacy.pools
|
||||
UNION ALL SELECT 'file_pool', count(*) FROM legacy.file_pool
|
||||
UNION ALL SELECT 'acl', count(*) FROM legacy.acl
|
||||
UNION ALL SELECT 'file_views', count(*) FROM legacy.file_views
|
||||
ORDER BY 1;"
|
||||
|
||||
echo ">> Running transform (single transaction) ..."
|
||||
psql_new -P pager=off -f "$TRANSFORM_SQL"
|
||||
|
||||
echo ">> Done. The foreign link will be removed now."
|
||||
@@ -0,0 +1,220 @@
|
||||
-- =============================================================================
|
||||
-- Tanabata legacy -> new schema data migration (transform step)
|
||||
--
|
||||
-- Reads the OLD database (exposed as the `legacy` schema — see migrate.sh, which
|
||||
-- imports it via postgres_fdw) and inserts the transformed rows into the new
|
||||
-- core / data / acl / activity schemas.
|
||||
--
|
||||
-- Assumes the new schema already exists (goose migrations applied) and is seeded
|
||||
-- (core.mime_types, core.object_types from 007_seed_data.sql).
|
||||
--
|
||||
-- Idempotent: ON CONFLICT DO NOTHING everywhere + preserved UUID PKs, so a
|
||||
-- re-run inserts only what is missing. Runs as one transaction — all or nothing.
|
||||
--
|
||||
-- Run with: psql "<new-dsn>" -v ON_ERROR_STOP=1 -f transform.sql
|
||||
-- (migrate.sh does this for you after setting up the `legacy` schema.)
|
||||
-- =============================================================================
|
||||
|
||||
\set ON_ERROR_STOP on
|
||||
|
||||
-- Fail early and clearly if the legacy data hasn't been made available.
|
||||
DO $$
|
||||
BEGIN
|
||||
IF to_regclass('legacy.users') IS NULL THEN
|
||||
RAISE EXCEPTION
|
||||
'legacy.* tables not found. Populate the "legacy" schema first '
|
||||
'(run migrate.sh, or load the old dump into a schema named legacy).';
|
||||
END IF;
|
||||
END $$;
|
||||
|
||||
BEGIN;
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- 1. Users. Old PK is uuid; the new table uses a smallint identity. Insert by
|
||||
-- the unique `name`, then build a uuid -> smallint map used by every FK below.
|
||||
-- Old `can_edit` becomes the new `can_create`; nobody is blocked on import.
|
||||
-- ---------------------------------------------------------------------------
|
||||
INSERT INTO core.users (name, password, is_admin, can_create, is_blocked)
|
||||
SELECT name, password, is_admin, can_edit, false
|
||||
FROM legacy.users
|
||||
ON CONFLICT (name) DO NOTHING;
|
||||
|
||||
CREATE TEMP TABLE user_id_map ON COMMIT DROP AS
|
||||
SELECT lu.id AS old_id, nu.id AS new_id
|
||||
FROM legacy.users lu
|
||||
JOIN core.users nu ON nu.name = lu.name;
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- 2. MIME types. Same uuid -> smallint remap, keyed by the MIME name. The new
|
||||
-- DB is pre-seeded with the common types; add any legacy ones not seeded.
|
||||
-- ---------------------------------------------------------------------------
|
||||
INSERT INTO core.mime_types (name, extension)
|
||||
SELECT name, extension
|
||||
FROM legacy.mime
|
||||
ON CONFLICT (name) DO NOTHING;
|
||||
|
||||
CREATE TEMP TABLE mime_id_map ON COMMIT DROP AS
|
||||
SELECT lm.id AS old_id, nm.id AS new_id
|
||||
FROM legacy.mime lm
|
||||
JOIN core.mime_types nm ON nm.name = lm.name;
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- 3. Categories. UUID PK preserved. is_private -> is_public (inverted),
|
||||
-- '' notes -> NULL, non-hex colors -> NULL (to satisfy the hex CHECK that the
|
||||
-- old NOT VALID constraint may not have enforced on existing rows).
|
||||
-- ---------------------------------------------------------------------------
|
||||
INSERT INTO data.categories (id, name, notes, color, metadata, creator_id, is_public)
|
||||
SELECT c.id,
|
||||
c.name,
|
||||
NULLIF(c.notes, ''),
|
||||
CASE WHEN c.color ~* '^[A-Fa-f0-9]{6}$' THEN c.color END,
|
||||
NULL,
|
||||
um.new_id,
|
||||
NOT c.is_private
|
||||
FROM legacy.categories c
|
||||
JOIN user_id_map um ON um.old_id = c.creator_id
|
||||
ON CONFLICT (id) DO NOTHING;
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- 4. Tags. UUID PK + category_id preserved.
|
||||
-- ---------------------------------------------------------------------------
|
||||
INSERT INTO data.tags (id, name, notes, color, category_id, metadata, creator_id, is_public)
|
||||
SELECT t.id,
|
||||
t.name,
|
||||
NULLIF(t.notes, ''),
|
||||
CASE WHEN t.color ~* '^[A-Fa-f0-9]{6}$' THEN t.color END,
|
||||
t.category_id,
|
||||
NULL,
|
||||
um.new_id,
|
||||
NOT t.is_private
|
||||
FROM legacy.tags t
|
||||
JOIN user_id_map um ON um.old_id = t.creator_id
|
||||
ON CONFLICT (id) DO NOTHING;
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- 5. Tag rules (old `autotags`): parent -> when_tag, child -> then_tag.
|
||||
-- Skip rules whose tags didn't migrate.
|
||||
-- ---------------------------------------------------------------------------
|
||||
INSERT INTO data.tag_rules (when_tag_id, then_tag_id, is_active)
|
||||
SELECT a.parent_id, a.child_id, a.is_active
|
||||
FROM legacy.autotags a
|
||||
WHERE EXISTS (SELECT 1 FROM data.tags t WHERE t.id = a.parent_id)
|
||||
AND EXISTS (SELECT 1 FROM data.tags t WHERE t.id = a.child_id)
|
||||
ON CONFLICT (when_tag_id, then_tag_id) DO NOTHING;
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- 6. Files. UUID PK preserved. old `datetime` -> content_datetime,
|
||||
-- `orig_name` -> original_name. EXIF is lifted out of the old metadata blob
|
||||
-- into its own column; whatever else was in metadata stays as user metadata
|
||||
-- (NULL if nothing remains). No phash / soft-delete existed before.
|
||||
-- ---------------------------------------------------------------------------
|
||||
INSERT INTO data.files (id, original_name, mime_id, content_datetime, notes,
|
||||
metadata, exif, phash, creator_id, is_public, is_deleted)
|
||||
SELECT f.id,
|
||||
f.orig_name,
|
||||
mm.new_id,
|
||||
f.datetime,
|
||||
NULLIF(f.notes, ''),
|
||||
NULLIF(f.metadata - 'exif', '{}'::jsonb),
|
||||
f.metadata -> 'exif',
|
||||
NULL,
|
||||
um.new_id,
|
||||
NOT f.is_private,
|
||||
false
|
||||
FROM legacy.files f
|
||||
JOIN user_id_map um ON um.old_id = f.creator_id
|
||||
JOIN mime_id_map mm ON mm.old_id = f.mime_id
|
||||
ON CONFLICT (id) DO NOTHING;
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- 7. File <-> tag. Skip orphan junction rows.
|
||||
-- ---------------------------------------------------------------------------
|
||||
INSERT INTO data.file_tag (file_id, tag_id)
|
||||
SELECT ft.file_id, ft.tag_id
|
||||
FROM legacy.file_tag ft
|
||||
WHERE EXISTS (SELECT 1 FROM data.files f WHERE f.id = ft.file_id)
|
||||
AND EXISTS (SELECT 1 FROM data.tags t WHERE t.id = ft.tag_id)
|
||||
ON CONFLICT DO NOTHING;
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- 8. Pools. UUID PK preserved. The new schema has neither pool hierarchy nor a
|
||||
-- `created` column, so the legacy parent_id and created timestamp are kept
|
||||
-- under metadata (pool ids are random v4, so created isn't otherwise
|
||||
-- recoverable). is_private -> is_public.
|
||||
-- ---------------------------------------------------------------------------
|
||||
INSERT INTO data.pools (id, name, notes, metadata, creator_id, is_public)
|
||||
SELECT p.id,
|
||||
p.name,
|
||||
NULLIF(p.notes, ''),
|
||||
jsonb_strip_nulls(jsonb_build_object(
|
||||
'legacy_parent_id', p.parent_id,
|
||||
'legacy_created', p.created)),
|
||||
um.new_id,
|
||||
NOT p.is_private
|
||||
FROM legacy.pools p
|
||||
JOIN user_id_map um ON um.old_id = p.creator_id
|
||||
ON CONFLICT (id) DO NOTHING;
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- 9. File <-> pool. The old table has no ordering column; synthesise a stable
|
||||
-- gapped position per pool, ordered by file id (UUID v7 ≈ chronological), so
|
||||
-- the app's gap-based reordering keeps working.
|
||||
-- ---------------------------------------------------------------------------
|
||||
INSERT INTO data.file_pool (file_id, pool_id, position)
|
||||
SELECT fp.file_id,
|
||||
fp.pool_id,
|
||||
(row_number() OVER (PARTITION BY fp.pool_id ORDER BY fp.file_id))::int * 1000
|
||||
FROM legacy.file_pool fp
|
||||
WHERE EXISTS (SELECT 1 FROM data.files f WHERE f.id = fp.file_id)
|
||||
AND EXISTS (SELECT 1 FROM data.pools p WHERE p.id = fp.pool_id)
|
||||
ON CONFLICT DO NOTHING;
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- 10. ACL. The old table stored no object type; derive it by locating the
|
||||
-- object among files/tags/categories/pools. read/write -> can_view/can_edit.
|
||||
-- Rows whose object no longer exists are skipped.
|
||||
-- ---------------------------------------------------------------------------
|
||||
INSERT INTO acl.permissions (user_id, object_type_id, object_id, can_view, can_edit)
|
||||
SELECT um.new_id, ot.id, a.object_id, a.read, a.write
|
||||
FROM legacy.acl a
|
||||
JOIN user_id_map um ON um.old_id = a.user_id
|
||||
JOIN LATERAL (
|
||||
SELECT CASE
|
||||
WHEN EXISTS (SELECT 1 FROM data.files f WHERE f.id = a.object_id) THEN 'file'
|
||||
WHEN EXISTS (SELECT 1 FROM data.tags t WHERE t.id = a.object_id) THEN 'tag'
|
||||
WHEN EXISTS (SELECT 1 FROM data.categories c WHERE c.id = a.object_id) THEN 'category'
|
||||
WHEN EXISTS (SELECT 1 FROM data.pools p WHERE p.id = a.object_id) THEN 'pool'
|
||||
END AS type_name
|
||||
) k ON true
|
||||
JOIN core.object_types ot ON ot.name = k.type_name
|
||||
ON CONFLICT (user_id, object_type_id, object_id) DO NOTHING;
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- 11. File view history. old `datetime` -> viewed_at.
|
||||
-- ---------------------------------------------------------------------------
|
||||
INSERT INTO activity.file_views (file_id, user_id, viewed_at)
|
||||
SELECT fv.file_id, um.new_id, fv.datetime
|
||||
FROM legacy.file_views fv
|
||||
JOIN user_id_map um ON um.old_id = fv.user_id
|
||||
WHERE EXISTS (SELECT 1 FROM data.files f WHERE f.id = fv.file_id)
|
||||
ON CONFLICT DO NOTHING;
|
||||
|
||||
COMMIT;
|
||||
|
||||
-- ---------------------------------------------------------------------------
|
||||
-- Summary of what now lives in the new schema.
|
||||
-- ---------------------------------------------------------------------------
|
||||
\echo ''
|
||||
\echo 'Migration committed. New row counts:'
|
||||
SELECT 'core.users' AS table, count(*) FROM core.users
|
||||
UNION ALL SELECT 'core.mime_types', count(*) FROM core.mime_types
|
||||
UNION ALL SELECT 'data.categories', count(*) FROM data.categories
|
||||
UNION ALL SELECT 'data.tags', count(*) FROM data.tags
|
||||
UNION ALL SELECT 'data.tag_rules', count(*) FROM data.tag_rules
|
||||
UNION ALL SELECT 'data.files', count(*) FROM data.files
|
||||
UNION ALL SELECT 'data.file_tag', count(*) FROM data.file_tag
|
||||
UNION ALL SELECT 'data.pools', count(*) FROM data.pools
|
||||
UNION ALL SELECT 'data.file_pool', count(*) FROM data.file_pool
|
||||
UNION ALL SELECT 'acl.permissions', count(*) FROM acl.permissions
|
||||
UNION ALL SELECT 'activity.file_views', count(*) FROM activity.file_views
|
||||
ORDER BY 1;
|
||||
Reference in New Issue
Block a user