Restoring from R2 (rig-gitops#897, P6)¶

Run provenance¶

The gap. The bucket lock stops anyone deleting or overwriting objects for 14 days, but not adding new ones, and the age recipient is public. So whoever holds the R2 writer token alone (leaked from a log, a CI run, or the operator's machine during P8) can upload a newer run: dumps encrypted to the recipient, plus a matching SHA256SUMS. "Restore the newest complete run" would load it, and age can't tell, because it doesn't authenticate the sender.

The mechanism. Each backup job signs its run's SHA256SUMS (ECDSA P-256 over SHA-256) with a key that only the two backup jobs mount. Before uploading anything it checks that signature: it must be non-empty and verify under a current pinned key, because dgst -sign -out exits 0 even when its write fails. Only then does it PUT SHA256SUMS.sig and, last, SHA256SUMS; otherwise the run fails red. A probe signed the same way at start-up stops the job before any dump if openssl, the pins or the key are wrong. SHA256SUMS names every object of the run by its full key, so one signature covers every object's hash and size, the location and the run. A restore starts with deploy/backup-r2/scripts/fetch-run.sh on the MacBook (P6 2/4, below), which takes the newest run that verifies.

The job and fetch-run.sh read the pins with one routine, load_pins in lib.sh: each PEM block of deploy/backup-r2/scripts/signing.pub must parse. That path is fixed beside lib.sh, in the checkout and in the pod's ConfigMap; no environment variable can replace it.

The tool is openssl, and no package is added. Its delta review is Stig-Johnny/claude-3 docs/ai-tooling-registry/backup/2026-10-03-security-review-openssl-backup-signing.md ("the openssl review"): approve with conditions. Its defects D1 to D9 are fixed here.

Where	Which openssl	How it is called
rig-backup image	Debian `openssl 3.5.7-1~deb13u3`, from the base image: `postgresql-common` pulls it in through `ssl-cert` (the P3 build's `dpkg-query` list, publish run 37120938091). It already computes the upload's `Content-MD5` (security review §7)	Only as `/usr/bin/openssl`, through `lib.sh`'s `ossl`, with `OPENSSL_CONF=/dev/null`, so no config file, engine or provider module loads. Only `dgst -sha256 -sign`, `dgst -sha256 -verify`, `pkey -pubin -noout` and `dgst -md5 -binary`. The job fails closed without it
Operator's MacBook	`/usr/bin/openssl`: LibreSSL 3.3.6 on macOS 26.5	The same `ossl`. `fetch-run.sh` stops unless `version` prints exactly `LibreSSL 3.3.6`, naming the review. It uses only `version`, `dgst -sha256 -verify -signature` and `pkey -pubin -noout`/`-outform DER`; O2b uses only `ecparam -name -param_enc -genkey -noout -out`, `pkey -in -pubout -noout` and `dgst -sha256 -sign/-verify`. LibreSSL 3.3.6 has all of them

ECDSA P-256 over SHA-256 works the same in both, and in GitHub Actions, so P9's watcher could check signatures too, with the public key only. Until O2b, signing.pub is a placeholder and the jobs refuse to start. The review's condition 6.1.1, pinning openssl and libssl3t64 in the image's apt-get install line, is a P3 image change, not made here.

Key	Where it lives	Made by
Private (P-256)	Secret `rig-backup-signing`, key `signing-key.pem`: SOPS in rig-gitops (P8), mounted read-only at `/secrets/signing` into `rig-backup-db` and `rig-backup-agent-home` only. Never the drill, Bitwarden or the MacBook's disk	O2b, on an offline medium
Public	`deploy/backup-r2/scripts/signing.pub` in rig-conductor, in the scripts ConfigMap. The jobs refuse a private key that no current pin verifies; `fetch-run.sh` trusts nothing else	A rig-conductor PR after O2b; never P7's patch

Operator steps¶

O2b, the signing key (MacBook, attended, after the openssl review and before P8). M is the offline medium, as for the age key. ecparam and pkey exit 0 even when their write fails, so the key is checked and used once before anything trusts it (openssl review D9). Stop at the first command that fails:

umask 077; M=/Volumes/<offline medium>; O=/usr/bin/openssl; export OPENSSL_CONF=/dev/null
[ "$($O version)" = "LibreSSL 3.3.6" ]
$O ecparam -name prime256v1 -param_enc named_curve -genkey -noout -out "$M/rig-backups-dell-signing.pem"
[ -s "$M/rig-backups-dell-signing.pem" ] && $O pkey -in "$M/rig-backups-dell-signing.pem" -noout   # non-empty, parses
$O pkey -in "$M/rig-backups-dell-signing.pem" -pubout -out "$M/rig-backups-dell-signing.pub"
head -c 4096 /dev/urandom > "$M/rt.bin"
$O dgst -sha256 -sign "$M/rig-backups-dell-signing.pem" -out "$M/rt.sig" "$M/rt.bin"
$O dgst -sha256 -verify "$M/rig-backups-dell-signing.pub" -signature "$M/rt.sig" "$M/rt.bin"   # must print Verified OK
rm -f "$M/rt.bin" "$M/rt.sig"; cat "$M/rig-backups-dell-signing.pub"   # → scripts/signing.pub (PR)

-param_enc named_curve makes LibreSSL write the curve by name, as the image's openssl does. The printed public key replaces the placeholder in deploy/backup-r2/scripts/signing.pub; record its SHA-256, which fetch-run.sh prints for each pinned key.

P8, its Secret (rig-gitops, straight from the medium, in a separate sitting from the writer token):

{ printf 'apiVersion: v1\nkind: Secret\nmetadata:\n  name: rig-backup-signing\n  namespace: rig-conductor\ntype: Opaque\nstringData:\n  signing-key.pem: |\n'
  sed 's/^/    /' "$M/rig-backups-dell-signing.pem"; } \
  | sops --encrypt --input-type yaml --output-type yaml \
      --filename-override apps/rig-backup-r2/signing.sops.yaml /dev/stdin > apps/rig-backup-r2/signing.sops.yaml
grep -c 'ENC\[' apps/rig-backup-r2/signing.sops.yaml          # must be 7 (5 + 1 + MAC)
grep -c 'PRIVATE KEY' apps/rig-backup-r2/signing.sops.yaml    # must be 0

Then add - signing.sops.yaml to apps/rig-backup-r2/kustomization.yaml, and delete the key from the medium once P8 has merged. A lost signing key costs only a new pair and a pin PR: it decrypts nothing.

Rotation (add to O7), and the not_after rule (openssl review D5). A pinned key with no not_after is current. A retired key keeps a line # not_after <UTC> just above its block in signing.pub, and then counts only for a run whose SHA256SUMS and SHA256SUMS.sig R2 stored before that time. R2 sets LastModified itself, so a writer can't backdate it, and the signed list binds every other object of the run by hash, so those two timestamps are enough. fetch-run.sh lists the run's _runs/ objects to read them. The job itself refuses to sign with a retired key.

Routine rotation: make a new pair (O2b). In one rig-conductor PR, put the new public key first in signing.pub and move the old one after it, under # not_after <the time of this PR>. Replace the Secret (P8). After 35 days, when the old key's last runs have expired, remove it.
After a suspected signing-key leak: the same, but not_after is the earliest time the key may have leaked. A forged run signed with it after that is REJECTED; older runs still restore. Then rotate the writer token too.

Residual risk¶

What leaks	What an attacker can then do	What stops or shows it
The writer token alone	Add objects and whole runs; once an object's 14-day lock ends, delete or overwrite it	Forged runs are `REJECTED`, never chosen, and named in a closing warning: treat it as an incident and rotate the token (O7). An overwritten object fails its sha256
Both of the backup pod's Secrets, `rig-backup-r2-writer` and `rig-backup-signing`: anyone who can read Secrets in `rig-conductor`, the k3s datastore, the Dell's disk, or the rig's `sops-age` key	Forge runs that verify. Until the key is retired, the newest would be chosen	Not stopped until noticed. Contents stay secret, because the age identity is offline. Once suspected, retire the key with `not_after` at the earliest leak time (rotation, above): forged runs after it are `REJECTED`, older runs still restore. Whoever holds the Dell can also corrupt the live data a run is taken from, which no signature can help with
The operator's machine during O2b or P8	Copy the signing key	Handle it from the offline medium only, apart from the writer token, so one session never exposes both

Splitting the key from the token is what turns a leaked token, the likely case, from silent poisoning into a named rejection. Keeping the key out of the cluster entirely would need offline signing of every run, which a 6-hourly unattended job can't have.

The drill kit¶

deploy/backup-r2/drill/ is a kustomization of its own: namespace rig-backup-drill and the CronJob rig-backup-drill, suspended for good and run on demand. Nothing references it until P7's Dell-only Flux Kustomization rig-backup-drill. P7 must also add ghcr-pull-secret to that namespace, as I4 does for rig-conductor: the restore container is the rig-backup image.

Where it differs from the plan. The plan's drill decrypts in the pod, from an age-identity Secret and an r2-reader Secret. The security review forbids that: age only encrypts in the cluster, the private key never enters it, and decryption is by hand on the MacBook (review §2 conditions 1 and 5, §9, §11.4). So the MacBook proves the run (fetch-run.sh) and decrypts it, then streams the plaintext into the pod. The pod restores and checks it, and never holds a key, a token or a password.

Part	What
`postgres`, a native sidecar	postgres-0's own digest, `pgvector/pgvector:pg16@sha256:7d400e34…`, so the restore meets the same server and pgvector. `initdb --auth=trust`, listening on 127.0.0.1 only; data in a 2 Gi emptyDir. Needs Kubernetes 1.29 or later. The security review doesn't cover this digest, and the rig-backup image may not run a server (review §5, condition 4)
`restore`, the rig-backup image	`restore-drill.sh run`: waits up to 45 min for the operator's `restore-drill.sh receive`, then restores each database with `pg_restore --no-owner --no-privileges` and checks it against the run's manifest
`seed-fixture.sql`	Drill A's `rig_drill_fixture`, with `rig_memory` exactly as rig-memory creates it (dashecorp/rig-memory-mcp `db.js` at `fe405b1`, `SCHEMA` and `HNSW_INDEX`): the `vector` and `uuid-ossp` extensions, every column, `embedding VECTOR(1536)`, the five B-tree and GIN indexes, and `idx_rm_embedding`, `USING hnsw (embedding vector_cosine_ops)`. 2,000 rows with random embeddings; the index is built after them

restore-drill.sh runs the backup's own counting query on each restored database; the guard fails if the two copies differ. A database passes only if its tables' row counts, its non-null embedding counts, its vector version and its pgvector indexes (definition and indisvalid) all equal the manifest. Wherever the manifest lists rig-memory's idx_rm_embedding, the restored index must still be USING hnsw (embedding vector_cosine_ops), and a KNN query with enable_seqscan = off must use it and return the probe row at distance 0. Each agent-home archive must pass gzip -t, and tar's listing must give the manifest's file count and bytes. The pod prints PASS or FAIL per check and TIMING per phase, fails if restoring and checking take over 30 minutes, and ends with DRILL PASS or DRILL FAIL, which is also its exit status. The guard proves that an off-by-one row count, a missing index, a KNN plan that falls back to a sequential scan, and idx_rm_embedding with another opclass each fail the drill.

fetch-run.sh¶

RC=~/src/rig-conductor   # a clean checkout of origin/main (git status clean): signing.pub comes from it, and only it
"$RC/deploy/backup-r2/scripts/fetch-run.sh" db dell 'https://<ACCOUNT_ID>.eu.r2.cloudflarestorage.com' rig-backups-dell ~/restore/db
# then paste the R2 reader key ID and secret from Bitwarden at the hidden prompt

Use home instead of db for agent-home, and add a run ID such as 2026-10-03T1230Z to take exactly that run, which must still verify. It first prints each pinned key's SHA-256: compare them with O2b's. Then it tries runs newest first and takes one only if:

SHA256SUMS.sig verifies under a pinned key;
every line of that SHA256SUMS names an object of that same run, and its manifest is listed, so a genuine signed list copied under a newer run name fails;
if that key is retired, R2 stored the run's SHA256SUMS and .sig before its not_after.

Its verdicts are kept apart (openssl review D3):

Output	Meaning	What happens
`REJECTED <run>: …`	The run failed one of the three checks	Never chosen; the next older run is tried. A closing `WARNING` names them: a forged run means a leaked token or key
`ERROR: …`	A fault: openssl isn't exactly LibreSSL 3.3.6, a pin doesn't parse, a listing or GET failed, a write failed	Stops at once, choosing nothing, so a fault never silently falls back to an older run
`TAMPERED or missing: <key>`	An object of the chosen run doesn't match its signed size and sha256	Fails, keeping nothing, without falling back by itself: to restore an older run, name it
`SELECTED`, then `VERIFIED <run>`	The run and every object passed	The run is in the output directory

Objects are downloaded into a staging directory beside the output directory, which an exit trap removes. Only when every listed object, counted against the lines of SHA256SUMS, has matched is the staging directory renamed to the output directory, so an interrupted or failed fetch leaves nothing that looks like a download (D6, D7). Decrypt in the same sitting, from the verified directory only. A listing that R2 truncates is an ERROR.

Runbook: the drill¶

Drill A runs before the first tenant moves, on the seeded fixture; drill B within 7 days after it, on real rig_memory data; then monthly. Every step runs from the MacBook, attended (zsh; a function, because zsh doesn't word-split variables). This replaces plan step O5, which put the age identity and a reader token in the cluster.

dell() { ssh -i ~/.ssh/dell-stig-1 [email protected] "$@"; }
RC=~/src/rig-conductor; EP='https://<ACCOUNT_ID>.eu.r2.cloudflarestorage.com'
M=/Volumes/<offline medium>   # holds the age key file (O2); the plaintext goes here too, never the MacBook's disk
# 1. Drill A only: seed the fixture, then take a backup that includes it
dell 'sudo k3s kubectl -n rig-conductor exec -i postgres-0 -- psql -U conductor -d postgres -v ON_ERROR_STOP=1 -f -' \
  < "$RC/deploy/backup-r2/drill/seed-fixture.sql"
dell 'sudo k3s kubectl -n rig-conductor create job --from=cronjob/rig-backup-db drill-backup-1 && sudo k3s kubectl -n rig-conductor wait --for=condition=complete --timeout=30m job/drill-backup-1'
# 2. Prove and fetch the newest runs, from a clean checkout of origin/main. Compare the printed key fingerprints with
#    O2b's. Any REJECTED line is an incident: rotate the writer token (O7). An ERROR stops: fix the fault and rerun.
"$RC/deploy/backup-r2/scripts/fetch-run.sh" db dell "$EP" rig-backups-dell ~/drill/db
"$RC/deploy/backup-r2/scripts/fetch-run.sh" home dell "$EP" rig-backups-dell ~/drill/home

Decrypt by hand, in the same sitting (openssl review D6), one object at a time, only as review §11.4 allows: attended, exit status 0, never -j or a plugin. First re-check, in each fetched directory, that nothing changed since VERIFIED: while read -r s b k; do [ "$(shasum -a 256 "$k" | cut -d' ' -f1)" = "$s" ] || echo "CHANGED $k"; done < _runs/dell/<db|home>/<run>/SHA256SUMS must print nothing. Then, for each <key>.age, including both manifest.json.age files: mkdir -p "$M/plain/$(dirname <key>)", then age -d -i "$M/<identity file>" -o "$M/plain/<key without .age>" <key>.age. Join any .age.p000, .age.p001, … parts with cat first. Delete the output of any age that fails.
Run the drill, then clean up:

dell 'sudo k3s kubectl -n rig-backup-drill create job --from=cronjob/rig-backup-drill drill-1 && sudo k3s kubectl -n rig-backup-drill wait --for=condition=ready --timeout=10m pod -l job-name=drill-1'
COPYFILE_DISABLE=1 tar --no-xattrs -C "$M/plain" -cf - . \
  | dell 'sudo k3s kubectl -n rig-backup-drill exec -i job/drill-1 -c restore -- bash /opt/drill/restore-drill.sh receive'
dell 'sudo k3s kubectl -n rig-backup-drill logs -f job/drill-1 -c restore'   # PASS lines and TIMINGs, then DRILL PASS
dell 'sudo k3s kubectl -n rig-backup-drill delete job drill-1'               # the pod's data goes with it
rm -rf "$M/plain" ~/drill
dell 'sudo k3s kubectl -n rig-conductor exec postgres-0 -- psql -U conductor -d postgres -c "DROP DATABASE IF EXISTS rig_drill_fixture WITH (FORCE)"'   # drill A only

Pass: fetch-run.sh VERIFIED both runs with no REJECTED, every age exited 0, and the pod printed DRILL PASS. Write the TIMING lines and the wall-clock time of steps 2 to 4 into the Dell doc: that is the measured RTO of a restore, which must stay within 30 minutes for the drill.

Runbook: a real restore¶

The target is a running location: GCP while it is on, or a rebuilt Dell, with Postgres up and empty. RTO ≤ 2 h from the decision to restore.

Prove and fetch the newest db run, as in drill step 2, or name a run made before a known incident. A REJECTED newer run means a leaked writer token, or signing key: carry on with the verified run, and rotate (O7) afterwards. An ERROR chooses nothing: fix the fault and rerun.
Decrypt by hand on the MacBook, as in drill step 3: the manifest, control.dump, then each tenant's dumps.
Re-erase first (erasure design, restore-re-erase). List every tenant erased after the run's start time, which is its run ID: from rig_control if the old location survives, else from the Art.17 register. Restore nothing under backups/<that tenant>/.
Restore each dump into the target's empty server, rig_control first, then each tenant's evt and mem DB: dell 'sudo k3s kubectl -n rig-conductor exec -i postgres-0 -- pg_restore -U conductor --no-owner --no-privileges --create -d postgres' < "$M/plain/<key without .age>".
Before serving, run DELETE /api/admin/tenants/{id} for each tenant from step 3 that the restored rig_control still lists (erasure runbook). Then start the conductor and let the reconcile sweep (#2562) rebuild open work from GitHub.
agent-home, if needed: fetch and decrypt the newest home run, then unpack each archive into its claim through a pod that mounts it (kubectl exec -i … -- tar -xzf - -C <mount>). Codex tokens in it may be stale: re-run docs/codex-auth-bootstrap.md on missing_auth.
Delete $M/plain, and record the times.