# CI forge runs CI workflows defined in `.forge/ci.toml` at the root of a repository. Jobs run on separate **runners**, which are `forge runner` on machines you control. The forge never connects to a runner. Runners pull jobs over HTTPS. What v1 has: jobs with `needs`, secrets, artifacts, live logs, commit statuses, and manual, push, tag and pull request triggers. What it does not have: a cache, or YAML. The workflow file is TOML. Jobs run in a rootless sandbox, with no container tool needed. The runner pulls and unpacks the image itself, and runs each job in its own container built with libcontainer, the crate behind youki. Each job gets its own container, its own workspace, and a read-only token. Podman is still supported as an option (`executor = "podman"`). ## Format reference The file is limited to 64 KiB. Unknown keys are an error at every level (`deny_unknown_fields`). A file with a parse error still creates a run, in state `error`, with the message shown on the run page and a `ci/config` status set to `error`. ### `[on]` | Key | Type | Default | Meaning | |----------------|-----------------|---------|---------| | `push` | array of string | `[]` | Branch globs. A push to a matching `refs/heads/*` starts a run. | | `tags` | array of string | `[]` | Tag globs. A push of a matching `refs/tags/*` starts a run. | | `pull_request` | bool | `false` | Start a run when a pull request is opened, updated or reopened. The config is read at the **head** commit. | | `manual` | bool | `true` | Writers can start a run on an existing branch from the CI tab. | ### `[jobs.]` There must be at least one job, and at most 20. | Key | Type | Default | Rules | |-------------------|----------------------|---------|-------| | `image` | string | none | Required. Non-empty, no whitespace. Image reference, pulled by the runner. | | `run` | string | none | Required. Non-empty. Runs as `sh -eu -c ""` in `/workspace`. | | `needs` | array of string | `[]` | Job names that must succeed first. Must exist. No cycles. | | `secrets` | array of string | `[]` | Repo secret names, each `^[A-Z_][A-Z0-9_]{0,63}$`. Exported as env vars. | | `env` | table of string | `{}` | Plain env vars. Names match the same pattern. A secret with the same name wins. | | `artifacts` | array of string | `[]` | Relative globs in the workspace. No `..`, no leading `/`. | | `runs_on` | array of string | `[]` | Runner labels. Every label must be present on the runner. | | `timeout_minutes` | integer | `60` | Range 1 to 360. | | `network` | bool | `true` | `false` gives the job an empty network namespace (loopback only). `true` shares the runner host's network namespace. Read [the caveats](#builtin-executor-caveats). | Job names match `^[a-z0-9][a-z0-9_-]{0,39}$`. The job's workspace is a fresh checkout of the exact commit, at `/workspace`. The job starts in that directory. ### Example ```toml [on] push = ["main", "release/*"] tags = ["v*"] pull_request = true manual = true [jobs.test] image = "docker.io/library/rust:1" run = "cargo test --workspace" timeout_minutes = 30 [jobs.package] image = "docker.io/library/rust:1" needs = ["test"] run = """ cargo build --release mkdir -p dist cp target/release/forge dist/ """ artifacts = ["dist/*"] [jobs.deploy] image = "docker.io/library/alpine:3" needs = ["package"] runs_on = ["linux", "x86_64"] secrets = ["DEPLOY_KEY"] env = { TARGET = "staging" } network = true run = "sh ./deploy.sh" ``` Artifacts are stored under the flattened name: `dist/forge` is stored as `dist_forge`. ### Triggers and glob rules - Branch and tag globs use `*` and `**` with `/` as a separator. `*` matches within one path segment and does **not** cross `/`. `**` matches across segments. - `release/*` matches `release/1.2`, but not `release/1.2/hotfix`. - `release/**` matches both. - `main` matches only `main`. - Artifact globs follow the same rule. `out/*.txt` matches `out/a.txt`, not `out/sub/a.txt`. - Jobs whose `needs` fail or are skipped are marked `skipped`. A skipped job gets the status `ci/` = `error` with the description `skipped`. ### Commit statuses Every job reports one status on the commit it ran on. Its context is `ci/`. A run whose config failed to parse reports `ci/config`. | Job state | Status state | Description | |------------|--------------|-------------| | queued, blocked, running | pending | | | success | success | | | failure | failure | | | error, cancelled | error | | | skipped | error | `skipped` | Pull request runs from forks, or from authors without write access, wait for approval. Their statuses are pending with the description `Waiting for approval`. ## Security model ### Fork pull requests need approval and get no secrets A pull request run needs approval when its head repository is a fork, or when the author has no write access to the base repository. Such a run stays in `waiting_approval`, and no job is queued until a writer clicks **Approve and run** on the run page. Even after approval, a run from a fork gets **no secrets**. The runner receives an empty secret map for that job, so `$MY_SECRET` is unset. A job from a fork still gets its plain `env` values, and it runs on a runner like any other job, so review the workflow file in the pull request before you approve. ### Masking limits The forge and the runner both mask secret values in logs, replacing each occurrence with `***`. Read these limits before you rely on masking: - Values shorter than 4 characters are **not** masked. A secret of `abc` will appear in the log. - Masking covers the raw value and its standard base64 encoding. It does **not** cover any other transformation. A hex, URL-encoded, reversed or truncated copy of a secret is printed as is. - A value is masked only where the whole value appears in the output. Printing the secret one character at a time is not caught. - Masking is applied while the stream is read, so a value split across two output chunks is still masked. Treat job logs as sensitive. Never `echo` a secret, even though it is masked. ### Job tokens Each claimed job gets its own token. The token: - is **read-only**. It can fetch the job's repository, and its wiki (`.wiki`), through smart HTTP with the user name `ci-job` and the password `:`. Every write is refused; - only works while the job is `running`. Once the job finishes, the token stops working for logs, artifacts, heartbeat, finish and git reads; - is masked in the log, along with the secrets. ### Secrets at rest Secret values are sealed with the instance key (`forge.key`), like mirror credentials. The secrets page shows only names and the time each was last updated. Values are write-only, and writing a value is audited by name. ## Runner install A runner is the `forge` binary, run as `forge runner`. The forge and the runner ship as one binary, so an operator deploys one file. The default `builtin` executor needs nothing else. The `forge-runner` binary is a development and test convenience only. It has the same subcommands, but operators should use `forge runner`, which the examples below show. ### 1. Check the host The builtin executor needs: - **Linux 5.11 or newer.** Overlayfs inside a user namespace needs it. On an older kernel, jobs still run, but each one gets a full copy of the image instead of an overlay. - **Unprivileged user namespaces.** Check that `/proc/sys/user/max_user_namespaces` is above 0. On distributions that gate them, `kernel.unprivileged_userns_clone` must be 1. - **cgroup v2 delegation for limits.** The `cpu`, `memory` and `pids` controllers must be delegated to your user. systemd does this for user sessions on current distributions. Without it, jobs run without `[limits]`, and each job log gets a `resource limits unavailable` line. Run the check: ```sh forge runner check ``` It runs one trivial job through the real sandbox. It prints `ok` or `FAIL` for each requirement, with the reason, and exits non-zero when a check fails. Pass `--work-dir` to test the filesystem that your jobs will use. ### Subordinate ids The builtin executor can give a job a range of uids and gids, so that package managers which change file ownership (`apt`, `apk add`, `useradd`) work. Inside the job, uid `0` is your user, and ids `1` to `n` are your subordinate range, where `n` is the smaller of the uid and gid range sizes, at most 65536. It needs: - `newuidmap` and `newgidmap`, from the shadow-utils package. They are setuid helpers in `/usr/bin`. - An entry for your user in `/etc/subuid` and `/etc/subgid`. The podman command below adds one. Set the mode with `[sandbox] userns` in `runner.toml`: ```toml [sandbox] userns = "auto" # "auto" (the default), "subuid", or "single" ``` - `auto` uses the subordinate range when the host has it. Otherwise it uses your own uid and gid only (single mode), and logs `sandbox ids: single uid ...` at startup. - `subuid` requires the range. A host without it fails at startup. - `single` always uses your own uid and gid only. `forge runner check` prints the mode it resolved. In subuid mode it also checks that a job can `chown` a file in `/tmp` to `1000:1000`. ### Optional: podman Set `executor = "podman"` in `runner.toml` to run jobs with rootless podman instead. Install podman with your distribution's package manager, for example `sudo dnf install podman`. Rootless podman needs subordinate user and group ranges for your user: ```sh grep "^$USER:" /etc/subuid /etc/subgid ``` If either line is missing, add a range (it must not overlap any existing one): ```sh sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 "$USER" podman system migrate ``` Check that podman runs without root: ```sh podman run --rm docker.io/library/alpine:3 echo ok ``` ### 2. Keep the runner alive after logout Systemd user services stop when the user logs out, unless lingering is enabled: ```sh sudo loginctl enable-linger "$USER" ``` ### 3. Register the runner Create a registration token on the forge admin page at `/admin/runners`. It is shown once, works once, and expires after `registration_token_hours` (default 24). Then run on the runner machine: ```sh forge runner register \ --url https://forge.example \ --token \ --name build-1 \ --labels linux,x86_64 ``` This writes `~/.config/forge-runner/runner.toml` with mode 0600, because it holds the runner secret. Job workspaces go under `~/.cache/forge-runner/work` unless you pass `--work-dir`. `--url` must be the forge's public base URL, and the runner must reach it over HTTPS. The config file looks like this: ```toml url = "https://forge.example" runner_id = "..." runner_secret = "..." # keep private name = "build-1" labels = ["linux", "x86_64"] work_dir = "/home/ci/.cache/forge-runner/work" concurrency = 1 executor = "builtin" # or "podman" [limits] memory = "4g" # the job cgroup's memory.max, or podman --memory cpus = "2" # the job cgroup's cpu.max, or podman --cpus pids = 4096 # the job cgroup's pids.max, or podman --pids-limit image_max_gb = 20 # cap on the unpacked size of one image (builtin) [images] keep_days = 7 # images unused for this many days are deleted (builtin) [[registry]] # optional login for private images, matched by registry host host = "ghcr.io" username = "bot" password = "..." # keep private, the file is mode 0600 ``` `concurrency` is the number of jobs run in parallel. The default is 1. Each job runs with no capabilities and `no_new_privileges`. The builtin executor also runs each job under a seccomp profile that denies calls such as `mount`, `ptrace`, `bpf` and `unshare`. Image garbage collection runs at startup and then daily. Unpacked images are kept under `work_dir/oci`. ### Builtin executor caveats Read these before you run untrusted or sensitive jobs on a runner: - **`network = true` shares the host's network namespace.** The builtin executor does not set up a separate network, so a job with `network = true` can reach every service that the runner host can reach, including services bound to the host's `localhost`. Use `network = false` for any job that does not need the network. It gets only a loopback interface. - **Subordinate ids are mapped in subuid mode.** Container ids `1` to `n` map to your subordinate range, so a command can change file ownership to another user or group, as `apt` and `useradd` do. This needs the setup in [Subordinate ids](#subordinate-ids). Without it, the runner falls back to single mode, where only your own uid and gid exist in a job and such commands fail. - **Image files on the host are owned by subordinate ids.** Unpacked images under `work_dir/oci` have the ownership a job sees. A file that is owner `1000` inside a job is owned by `start + 999` on the host, for example `525287` with the range `524288`. `ls -ln` shows the numbers. The runner removes these files through a helper, so it can still clean them up. - **With `network = true`, `/sys` is the host's `/sys`,** bind-mounted read-only. The job can read host hardware and kernel details from it. - **The job cgroup is placed in the parent slice.** It is named `forge-job--` and sits next to the runner's own cgroup, not under it. The runner's cgroup is never used for a job, because stopping a job kills everything in its cgroup. ### 4. Run it under systemd Create `~/.config/systemd/user/forge-runner.service`, with the `forge` binary in `~/.local/bin`: ```ini [Unit] Description=forge CI runner After=network-online.target Wants=network-online.target [Service] ExecStart=%h/.local/bin/forge runner run --config %h/.config/forge-runner/runner.toml Restart=on-failure RestartSec=10 TimeoutStopSec=120 [Install] WantedBy=default.target ``` Then enable and start it: ```sh systemctl --user daemon-reload systemctl --user enable --now forge-runner.service journalctl --user -u forge-runner.service -f ``` On shutdown the runner stops claiming, and it cancels and reports any job still running. A job whose runner disappears without reporting is failed as `runner lost` after about two minutes, with no heartbeat. ### Labels A job with `runs_on = ["linux", "x86_64"]` is only claimed by a runner that has both labels. Jobs with no `runs_on` can go to any runner. ## Forge limits (`[ci]`) The `[ci]` section of `forge.toml` controls CI on the forge. These are the defaults: ```toml [ci] enabled = true max_log_mb = 16 # a log past this is cut with "[log truncated]" max_artifact_mb = 512 # per artifact max_artifacts_per_job = 50 log_retention_days = 90 artifact_retention_days = 30 registration_token_hours = 24 ``` Retention runs once a day. Logs older than `log_retention_days` are deleted. The job rows stay, so statuses and run pages still work. Artifacts older than `artifact_retention_days` are deleted from the blob store. ## Runner protocol The runner talks to the forge only over HTTPS. JSON bodies are defined in `forge-ci::proto`. | Method and path | Auth | Purpose | |-----------------|------|---------| | `POST /api/v1/ci/runners/register` | registration token | Exchange a single-use token for a runner id and secret. | | `POST /api/v1/ci/runners/claim` | `Bearer` runner secret | Long-poll (up to about 25 s) for a job. Returns the job assignment, or 204. Disabled runners get 403. | | `POST /api/v1/ci/jobs/{id}/logs` | `Bearer` job token | Append a log chunk. The header `X-Forge-Log-Offset` must equal the current size. A mismatch returns 409 with `X-Forge-Log-Size`, and the runner resumes from there. | | `POST /api/v1/ci/jobs/{id}/heartbeat` | `Bearer` job token | Sent every 10 s. The reply says whether the job was cancelled. | | `PUT /api/v1/ci/jobs/{id}/artifacts/{name}` | `Bearer` job token | Upload one artifact as a raw body. | | `POST /api/v1/ci/jobs/{id}/finish` | `Bearer` job token | Report `success`, `failure`, `cancelled` or `error`, with the exit code and an optional message. | The claim response (`JobAssignment`) carries the repository, the commit, the job's run script, image, env, secrets, artifact globs, timeout and network setting. Secrets are empty for fork pull request runs. For the workspace, the runner runs `git fetch --depth=1` on the clone URL with the job token, then `git checkout --detach `. The token goes in an HTTP header, never on the command line. Browsers and other clients read job logs with `GET /{owner}/{repo}/ci/jobs/{id}/log.txt`. It streams the file as it grows and ends when the job finishes.