# Git forge research notes Compiled 2026-10-15 from 10 parallel research passes. Sources are linked inline. Items marked *(unverified)* came from secondary sources or aggregators. Many 2026 CVE IDs below came from aggregator sites, so check them against GHSA/NVD before citing them. --- ## 1. Smart HTTP - **Endpoints:** - `GET /info/refs?service=git-{upload,receive}-pack` - `POST /git-upload-pack` - `POST /git-receive-pack` - **Content types:** - advertisement: `application/x-git--advertisement` - request: `application/x-git--request` - result: `application/x-git--result` - Always send `Cache-Control: no-cache`. - - **Advertisement body:** a pkt-line `# service=git-upload-pack`, then a flush `0000`, then the output of `git upload-pack --stateless-rpc --advertise-refs`. - **Protocol v2:** pass the `Git-Protocol` request header through to the `GIT_PROTOCOL` env var. With v2, clients go straight to POST. - **Gzip:** request bodies may arrive with `Content-Encoding: gzip`. Inflate them before piping to git. - **Streaming:** stream request and response bodies to and from the git child process. Never buffer a whole push. - Clients use chunked POSTs above `http.postBuffer` (1 MiB). - GitLab moved git traffic out of Rails into Workhorse because the app server's 60s timeout killed long requests. **Git routes must have no request timeout and no body-size middleware.** - **Keepalive:** `uploadpack.keepAlive` defaults to 5s, so proxies must not apply idle timeouts to git routes. - **Auth:** answer 401 with `WWW-Authenticate: Basic`. The username is ignored or must match, and the password is a personal access token. Anonymous read is allowed only for public repos. - **Dumb protocol:** skip it, and set `http.getanyfile=false`. - **Partial clone:** enable `uploadpack.allowFilter` and `uploadpack.allowRefInWant`. ## 2. SSH - The client runs `git-upload-pack '/path'`, `git-receive-pack '/path'` or `git-upload-archive '/path'`. - **Allow-list these exact command names.** - Parse the single-quoted argument strictly: reject quotes and newlines inside it. - Never use a shell. - - **Environment:** accept only `GIT_PROTOCOL` from env requests. Gogs CVE-2024-39930 was an RCE through env handling in its built-in SSH server. - **Key lookup** must be an exact, case-sensitive match on the key blob. Gitea had an auth bypass from a case-insensitive SQLite `LIKE`. - **Key types:** disable `ssh-rsa` with SHA-1 signatures (OpenSSH 8.8 default). russh still lists it as supported, so turn it off explicitly. - **Timeouts:** set handshake and auth timeouts. Gogs had a DoS from stalled SSH handshakes. - **russh:** version 0.64.x. `server::Handler` provides `auth_publickey`, `env_request`, `exec_request` and `data`; `Session` provides `data`, `eof`, `exit_status_request` and `close`. - **Alternative:** OpenSSH with `AuthorizedKeysCommand` plus `AcceptEnv GIT_PROTOCOL`. ## 3. Push handling and hooks - **pre-receive:** - Runs once per push and gets ` ` lines on stdin. - A non-zero exit rejects the **whole** push. **All policy goes here.** - New objects sit in a **quarantine** directory (`GIT_QUARANTINE_PATH`, `GIT_OBJECT_DIRECTORY`, `GIT_ALTERNATE_OBJECT_DIRECTORIES`). They aren't visible to other processes until the hook accepts, so any git command the hook runs must inherit those env vars. - **update:** runs per ref and can reject a single ref. - **post-receive:** runs after the push and cannot reject. Use it to enqueue events: webhooks, PR updates, cache invalidation, mirror pushes. - **reference-transaction** (git 2.28+) can veto any ref update, including ones that don't come from a push. - **Hook design (Gitea pattern):** a tiny hook binary calls back into the app's internal API. - Use a unix socket plus a shared secret. - Pass user ID, repo ID and protocol through env vars set when spawning receive-pack. - - **Force-push detection:** `git rev-list --max-count=1 ^`; any output means a force push. - **Deletion:** `new == 0000…`. - **Server config:** - `receive.fsckObjects=true` - `receive.maxInputSize` - `receive.advertisePushOptions=true` - `core.hooksPath` pointing to our own read-only directory. Users never write hooks or config. - **Gitea gap to avoid:** its branch protection with required checks only applies to PR merges, so direct pushes bypass it. - **AGit flow:** `git push origin HEAD:refs/for/main -o title=…` creates a PR without a fork. It goes through the `proc-receive` hook with `receive.procReceiveRefs`. ## 4. Security checklist - **Argument injection:** - Always put `--end-of-options` before user-supplied revisions and `--` before paths (git 2.24+). - Reject refs and names starting with `-`. - Build every git command through one builder that enforces this. - Examples: Gitea CVE-2022-42968; go-git CVE-2025-21613. - **Symlinks and paths:** check **every** path component for symlinks when rendering, archiving or writing, not just the last one. Gogs CVE-2025-8110 is in the CISA KEV catalog of actively exploited bugs. - **Untrusted repos (imports and mirrors):** - Never recurse into submodules. - Never trust repo-local config. - Keep `protocol.file.allow=user`. - Run as a dedicated service user. - Keep the git binary patched: CVE-2024-32002 and CVE-2025-48384 are hook-execution bugs via submodules. - **SSRF** (migrations, mirrors, webhooks): resolve DNS, pin the resulting IP, block private and link-local ranges, and **re-check after every redirect**. Block `file://`. Gitea and Gogs had several 2026 CVEs here *(unverified IDs)*. This matters most for us, because mirrors are a core feature. - **XSS:** - Serve raw and user-uploaded content from a **separate origin** (e.g. `raw.`) with `X-Content-Type-Options: nosniff` and a strict CSP. - Use `Content-Disposition: attachment` for HTML and SVG. - Sanitize rendered markdown (ammonia) and allow only http(s) and mailto links. - **DoS:** - Cap request bodies, diff sizes, blame and log time, and archive sizes. - Use only linear-time regex (Rust's `regex` crate is fine). Gitea had a ReDoS in CODEOWNERS matching. - **Private repos:** return **404 for both missing and forbidden**, everywhere, including any go-get style endpoints. - **Tokens:** store hashes and compare in constant time. Never put tokens in URLs or logs. - **Names:** - NFKC-normalize and case-fold, with a unique index on the folded form. - Keep a reserved-names list. - Don't allow names to end in `.git` or `.atom`. ## 5. Storage and maintenance - **Layout by ID**, e.g. `repos/ab/cd/.git` (GitLab hashed storage). Renames are then DB-only, with a redirect table for old web and git URLs. GitHub redirects git remotes after a rename. - **Maintenance:** - Schedule `git maintenance` (incremental-repack plus loose-objects), or `git repack --geometric=2 --write-midx`. - Write a commit-graph with changed-path Bloom filters; these make `git log -- path` fast. - Write bitmaps for clone speed. - **Hidden refs:** set `uploadpack.hideRefs` / `transfer.hideRefs` for internal refs. - **Atomic ref updates:** use `git update-ref --stdin`, which updates all refs in one transaction. - **Tree view "last commit per file":** this is expensive and needs a cache keyed by (commit, path). GitLab measured a ~22× gain from caching it. - **Archives:** cache them, but don't promise stable checksums. Bytes change across git versions. - **Forks:** dedupe with alternates or object pools, but pools must never be pruned. Leaking objects between private forks is a risk. **Defer dedup and do full copies at first.** - **Backups:** snapshot the refs first, then the objects, or briefly pause writes. Back up the **instance secret key** separately; Gitea loses 2FA and mirror credentials without it. ## 6. Pull requests - **PR refs:** store the head as `refs/pull/N/head` in the base repo and update it on every push to the head branch. - **Mergeability:** - Check with `git merge-tree --write-tree base head`, which needs no worktree and reports conflicts. - Recompute whenever the base or head moves. - A test-merge ref `refs/pull/N/merge` is optional. - **Diffs:** show **three-dot** diffs (from the merge base). - **Merge strategies:** - merge commit, built with `commit-tree` from the merge-tree result - squash - rebase - fast-forward only - **Review comments:** store the commit SHA, path, line, side and the diff hunk context. When the anchored code changes, mark the comment **outdated**; don't delete it. - **Large diffs:** collapse or lazy-load big files, detect binary files, and use rename detection (`-M`). - **Range-diff:** `git range-diff` gives "what changed since the last review". - **Gerrit/AGit:** an alternative workflow that pushes to `refs/for/`. ## 7. Git library split - gix 0.89 *(unverified version)*: - **Use it in-process for:** - object, tree and blob reads - rev-parse and rev-walk - tree and blob diff - rename detection - tar and zip archives - pack encode and decode - basic ref transactions - **Missing in gix:** - the upload-pack/receive-pack server side - signature verification - commit-graph writing and bitmaps - **Partial in gix:** - blame (no rename tracking; "not competitive with git") - merge workflow - **Shell out to `git` for:** serving fetch and push, merges and conflicts, blame, `verify-commit`/`verify-tag`, and maintenance. - **Hot-path fallback:** a pool of long-running `git cat-file --batch-command` processes, one per repo. Gitea and Gitaly both use this. - **Skip git2**, since it adds a C dependency. - **Minimum git version:** 2.38+ for `merge-tree --write-tree`. In practice, track the latest release for security fixes. ## 8. Features and crates - **Markdown:** comrak (GFM, alerts, footnotes, math) followed by ammonia for sanitization. Rewrite relative links and images to repo paths. - **Syntax highlighting:** syntect (mature), or arborium/tree-sitter. Tree-sitter grammars must match the core version. - **Language stats:** tokei (not equivalent to Linguist). - **Code search:** a trigram index (zoekt-style) or tantivy. Defer it. - **Commit signatures:** `git verify-commit` with `gpg.format=ssh` and a generated `allowedSignersFile` built from users' SSH signing keys. - **Webhooks:** - Sign with an `X-Hub-Signature-256`-style HMAC-SHA256 over the raw body. - Send a delivery ID header and use a 10s timeout. - Keep a retry queue and a delivery log. - **LFS:** the batch API (`POST /objects/batch`) plus the locks API. Storage can go to `in` or `object_store`. - **CODEOWNERS:** the last matching pattern wins. - **Mailmap:** apply `.mailmap` in display. - **Avatars:** Gravatar uses SHA-256 of the email. - **Feeds:** atom_syndication. ## 9. Ecosystem and portability - **Woodpecker CI** supports Gitea and Forgejo forges, and so do Renovate and `tea`. **A Gitea-compatible API subset would make them all work for us almost for free.** - **Forgejo Actions:** its runner is a fork of `act` with a Connect/JSON runner protocol *(unverified)*. It's explicitly not identical to GitHub Actions. - **Minimum CI surface:** webhooks plus a commit-status API. - **F3 (Friendly Forge Format):** JSON plus attachments for issues, PRs, labels and milestones, with a Forgejo driver. A good export target. - **Radicle and git-bug** store issues and patches **inside git**, so they travel with every clone and mirror. - **Software Heritage** has a Gitea/Forgejo lister ("Add Forge Now"). Being Gitea-API-compatible would also let it archive us automatically. - **Forgejo federation:** ForgeFed; v16 (2026-07) shipped federated stars as a first step. ## 10. Operations and product - **Permission model:** Read, Triage, Write, Maintain and Admin, plus collaborators. Org teams come later. - **One authorization function** must serve web, API, smart HTTP, SSH and hooks. - **Ghost user:** on account deletion, reassign issues and comments to a ghost user; never hard-delete them. - **Personal access tokens:** expiry (default 30 days), scopes, stored hashed. - **Deploy keys:** read-only by default, with write as an opt-in. - **Archived repos:** fully read-only. - **Mirrors:** - Pull mirrors run on an interval, with a minimum interval. - Push mirrors keep encrypted credentials. - Run them after each push with a backoff. - **Rate limiting:** per user and per IP, with writes weighted heavier. - **Open registration** (if ever): require email verification and rate-limit new accounts. Codeberg sees bot accounts opening hundreds of issues per hour. - **Takedowns:** only relevant if the instance becomes public. --- # Part 2: Data safety and reliability ## 11. Lessons from real incidents - **SourceHut (Jan 2024):** a DDoS plus upstream null-routing took down the primary datacenter. They migrated to Amsterdam over about a week. The standby DB was under 30s behind, but **storage backups were up to 12h old and object storage had no off-site copy**. - **OVH Strasbourg fire (2021):** backups kept **at the same site** burned with the primary. Courts awarded customers damages. - **Provider account risk:** Hetzner can terminate with 30 days' notice, and customers have reported notices that never arrived. OVH has reports of suspensions without warning *(unverified reviews)*. The protection is diversity: different provider, **payment method, contact email** and jurisdiction. - **Git corruption** happens on power loss or storage hiccups, typically as zero-length loose objects. GitLab's runbook says to run `git fsck` after any unclean shutdown. ## 12. How big hosts replicate, and what fits us - **GitHub Spokes:** 3 replicas, with each write committed by a majority under a lock. Repos re-replicate after a failure. This needs consensus and is overkill for us. - **GitLab Gitaly Cluster (Praefect):** has documented known issues, such as snapshot restores that can't be reconciled and hook races. It's a cautionary example of complexity. - **Raft (openraft):** pre-1.0. Overkill here. - **Split-brain** is the main danger of more than one writer. **Fence the old primary before promoting the standby.** DNS and health checks alone don't enforce a single writer. - **Never rsync or `cp` a live repo or a live DB.** ## 13. Git replication approach (single writer) 1. After post-receive, a replication job takes a **per-repo lock** and runs `git push --mirror` to each replica over SSH. It records failures and retries with backoff. 2. **Hourly reconcile:** compare a ref checksum (sha256 of sorted `git for-each-ref --format='%(objectname) %(refname)'`) between primary and replica. On a mismatch, re-mirror and run `git fsck`. 3. **Daily `git bundle create --all`** for each repo to off-provider storage, then `git bundle verify`. 4. Replicas get a weekly `git fsck --full` from cron. `git maintenance` has no fsck task. ## 14. Metadata (Turso) safety - Turso supports only `synchronous` OFF or FULL. **Use WAL with FULL.** - **Litestream is not verified on Turso files.** Turso says mixed SQLite/Turso access across processes is unsupported. Run a spike before relying on it. - **Turso Sync/CDC** is documented only against Turso Cloud, and self-hosted sync is unconfirmed. Don't depend on it. - **Verified path:** `VACUUM INTO` (supported by Turso) produces consistent snapshots on a schedule, which are then shipped to the replica and to backups. - **Integrity checks:** `quick_check` daily, `integrity_check` plus `foreign_key_check` weekly. Alert on any result other than `ok`. - **Instance secret key** (it encrypts mirror credentials, 2FA secrets and similar): back it up separately and escrow it offline. ## 15. Backups (3-2-1-1-0) - **3-2-1-1-0:** 3 copies, 2 media, 1 offsite, 1 **immutable**, 0 verify errors. - **Immutable storage:** Backblaze B2 Object Lock in **compliance mode** (nobody can shorten it), Cloudflare R2 bucket locks, or Wasabi compliance mode. - **Survive a compromised server:** the server holds **no delete credentials**. Either a separate machine pulls the backups, or the server pushes with append-only keys (rest-server `--append-only`, or BorgBase append-only). - **Tools:** - restic: mature, with `check --read-data-subset` for verification. - rustic: Rust and restic-compatible, but newer. - borg: 1.4 is stable. - **Encryption keys:** add a second restic key and store the keys offline. A lost key means lost data. - **Monitoring:** a dead-man switch via healthchecks.io pings with `/start` and `/fail`. - **Restore drill:** monthly, automated, restoring to a fresh VPS. Time it; that time is the real recovery time (RTO). ## 16. Storage-level integrity - **Filesystem:** - btrfs (or ZFS with ≥4GB RAM) gives checksums, scrub, snapshots and `send`/`receive`. - On tiny VPSes, ext4/XFS with tested backups is fine. - **Git fsync:** set `core.fsync` explicitly to cover loose objects and refs, plus a `core.fsyncMethod`. The default `committed,-loose-object` "risks losing recent work". - **App files:** write a temp file, fsync it, rename it, then fsync the directory.