# Backups, phase 1 (contract between workers)

The background is in `docs/architecture.md` §8.2–8.4. Phase 1 makes complete, verifiable backups **locally**: a directory that an external tool (restic, rclone, or borg with append-only keys) ships off-site. Shipping it, standby replication and push mirrors come later.

The global rules apply:
- No unwrap in non-test code.
- Tests use `tempfile::TempDir`.
- Never put cargo target dirs or big files in /tmp.

## Backup set layout
```
<out>/forge-backup-<UTC yyyymmddThhmmssZ>/
  manifest.json          # format version, forge version, created_at, instance base_url, files [{path, bytes, sha256}], repos [{id, owner, name, refs_sha256, bundle | null (empty repo)}]
  db/forge.db            # consistent snapshot
  repos/<repo id>.bundle # `git bundle create --all`, then `git bundle verify` before the manifest is written
  keys/forge.key         # only with --include-key; otherwise the manifest records "key_included": false
```
- `refs_sha256` is the sha256 of sorted `git for-each-ref --format='%(objectname) %(refname)'` (the reconcile checksum from architecture §8.2). It's computed from the bundle-time ref list, so a restore can compare against it.
- The directory is written as `.partial-…` and renamed atomically when complete. A crash never leaves something that looks like a finished backup.
- Files are 0600 and directories 0700.

## DB snapshot
- Use `VACUUM INTO '<path>'` if Turso 0.8.x supports it; the worker must verify this with a test.
- **Fallback:** hold the store's connection mutex so there are no concurrent writers, run `PRAGMA wal_checkpoint(TRUNCATE)` (or the Turso equivalent), copy the DB file (and the WAL, if any is left), then release.
- Verify the result by opening it read-only with a fresh connection and running `PRAGMA integrity_check` (or `quick_check`), with the migrations version check.

## Commands (`forge-web admin …`)
- `backup --out DIR [--include-key] [--repos-only | --db-only]`: prints the path and a summary, and exits non-zero on any failure.
- `backup-verify PATH`: recomputes every sha256, runs `git bundle verify` on each bundle (in a scratch repo under PATH's parent, **not /tmp**), opens the DB snapshot read-only and runs `integrity_check`, and checks the manifest's repo list against the DB snapshot's `repos` table.
- `restore --from PATH --data-dir NEW_DIR`: refuses a non-empty NEW_DIR. Copies the DB, then for each repo runs `git clone --mirror <bundle>` into `repo_path(NEW_DIR, id)` (or `init --bare` + `fetch` from the bundle with `+refs/*:refs/*`) and removes the remote. Then writes the managed gitconfig and hooks the same way boot does, compares `refs_sha256` per repo, and writes `forge.key` if the set has one (otherwise warns loudly that the key must be restored by hand).
- `fsck [--full]`: `git fsck` (`--full` if asked) on every repo plus the DB `integrity_check`. Reports per repo, and exits non-zero on any problem.

## Scheduler (config `[backup]`)
```toml
[backup]
dir = ""                  # empty = disabled
interval_hours = 24       # full backup set
keep = 7                  # keep the newest N complete sets in dir (delete older complete sets only; never touch other files)
healthcheck_url = ""      # optional; GET <url>/start, then <url> on success or <url>/fail with the error (healthchecks.io style)
fsck_interval_days = 7
```
- A tokio task started at boot runs the same code as the CLI. Healthcheck pings use a plain reqwest client with a 10s timeout; the URL is admin-configured and trusted.
- `GET /admin/backups` (instance admins): the last 20 runs (time, duration, size, ok or error), backup and fsck statuses, and a "Run backup now" POST with CSRF. Runs are recorded in a table `backup_runs(id, kind 'backup'|'fsck', started_at, finished_at, ok, path, bytes, error)`, migration `0014_backup_runs.sql`.

## Docs
`docs/backup.md` covers:
- what is in a set and what isn't (the key);
- how to ship it off-site with restic append-only or rclone, plus a sample systemd timer;
- how to restore, step by step;
- a quarterly restore drill;
- the fencing reminder from architecture §8.5.