Off-site backups to R2 (rig-gitops#897, P5)

deploy/backup-r2/ is the Dell's off-site backup, ready to deploy but inert:

• deploy/kustomization.yaml doesn't list it, so kubectl kustomize deploy/ (what GCP's Flux applies) renders byte-identical to before. Only P7's Dell-only Flux Kustomization will point here, with targetNamespace: rig-conductor.
• The CronJobs arrive suspend: true. P9 unsuspends them after the proving run and drill A.
• The image is P3's rig-backup image; the scripts are P4's backup scripts, mounted from a generated ConfigMap.
• Proving runs, signatures, the restore drill and real restores: Restoring from R2 (P6).

File	What
kustomization.yaml	Namespace rig-conductor; configMapGenerator rig-backup-scripts from scripts/ (hashed, so a script change rolls the CronJobs)
serviceaccount.yaml	rig-backup, automountServiceAccountToken: false, no Role or binding
config.yaml	ConfigMap rig-backup-config: location values as .invalid placeholders, which the scripts refuse
cronjob-db.yaml	rig-backup-db
cronjob-agent-home.yaml	rig-backup-agent-home

Configuration

Key	Set by	Placeholder
LOCATION, R2_ENDPOINT, R2_BUCKET	P7's Dell patch, from P1's outputs	location.invalid, https://account-id.invalid, bucket.invalid
AGE_RECIPIENT	P7's Dell patch, from O2	age-recipient.invalid
AGE_RECIPIENT_PIN	A rig-conductor PR after O2, not P7, so the recipient can only change with a reviewed PR in both repos	age-recipient-pin.invalid
scripts/signing.pub, a file in the scripts ConfigMap: the run-signing public keys (P6, Restoring from R2)	A rig-conductor PR after O2b, not P7	signing-public-key.invalid
BACKUP_EXCLUDE_DBS	P7, from the proving run	empty: every database must be dumped
HOME_INCLUDE_CLAIMS, HOME_EXCLUDE_CLAIMS	P7: <namespace>/<claim> lists. Include agent-home and the tenant-fleet claims (I5); exclude Postgres's rig-conductor/data-postgres-0 and Valkey	ns.invalid/claim.invalid

Credentials are never in this ConfigMap, in env values or in args. Each job mounts them as files, at the paths the scripts read:

Mount	Secret	Keys
/secrets/r2-writer	rig-backup-r2-writer (P8)	AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
/secrets/signing	rig-backup-signing (P8), in these two jobs only	signing-key.pem, which signs each run (P6)
/secrets/rig-conductor	rig-conductor-secrets (I2), only these keys via items:	postgres-password (db job only), discord-webhook-url

The pods can't start until P8 creates rig-backup-r2-writer. That's fine while the CronJobs are suspended.

rig-backup-db

Schedule	30 */6 * * *, timeZone: Etc/UTC
Run limits	concurrencyPolicy: Forbid, startingDeadlineSeconds: 3600, activeDeadlineSeconds: 3600, backoffLimit: 1, ttlSecondsAfterFinished: 172800 (2 d of logs after an alert)
Command	nice -n 10 bash /opt/rig-backup/backup-db.sh (overrides the image entrypoint)
Pod	uid/gid 999, runAsNonRoot, seccompProfile: RuntimeDefault, no service account token, no service links
Container	readOnlyRootFilesystem, allowPrivilegeEscalation: false, all capabilities dropped, HOME=/nonexistent
Volumes	/work emptyDir 8 Gi (ciphertext only), /tmp memory emptyDir 16 Mi (pgpass copy), scripts read-only
Resources	requests 50m CPU and 128 Mi; limits 1 CPU and 512 Mi

Retention is not in the cluster: R2 lifecycle and bucket lock (P1) keep backups/ and _runs/ 35 d, locked 14 d.

rig-backup-agent-home

scripts/backup-agent-home.sh tars each included volume through a read-only hostPath of k3s's local-path root (/var/lib/rancher/k3s/storage; confirm it in kube-system/local-path-config), gzips and encrypts it, and PUTs agent-home/<location>/<run>/<namespace>_<claim>.tar.gz.age (1 GiB parts as .pNNN). Then the encrypted manifest (per claim: key, regular files and bytes from tar's own listing, tar's exit status), then SHA256SUMS, under _runs/<location>/home/<run>/.

Schedule	45 2 * * *, timeZone: Etc/UTC; lifecycle keeps agent-home/ 7 d, locked 7 d (P1)
Run limits	As the db job, but activeDeadlineSeconds: 7200: gigabytes over a site's uplink
Pod	uid 0, seccompProfile: RuntimeDefault, no service account token. All capabilities dropped except DAC_READ_SEARCH, which reads other users' 0600 files (plan: The jobs). Read-only root, allowPrivilegeEscalation: false
Mounts	The local-path root read-only at /storage, and only discord-webhook-url from rig-conductor-secrets: no database credential

It finds volumes by local-path's directory names, pvc-<uid>_<namespace>_<claim>, not through the Kubernetes API, so there is no ClusterRole. The plan wanted get and list on PVCs and PVs, but the image has no kubectl, and the security review lets its curl reach only R2 and Discord (§4, condition 2). The plan's cases become:

Case	Handling
Included claim with no directory: Pending (WaitForFirstConsumer, as on a standby Dell)	Logged only
Included claim Bound but its directory missing	Can't be told apart from Pending without the API, so also logged only. A wrong or unmounted root, where every directory is missing, fails the run: Postgres's volume is always there
A volume in neither list	🟠, named
tar exit 1 ("file changed as we read it")	🟠; the archive is kept. Exit 2 or more fails that claim

scripts/check-backup-r2.sh runs the real script against a fake local-path root in five scenarios: a clean run with a Pending claim, an unlisted volume, tar exit 1, tar exit 2, and no volumes at all. Its --selftest puts back nine agent-home defects, among them a plaintext upload, a wrong key prefix and an excluded volume archived.

Proving run (before P9)

After P7 and P8, from the MacBook (plan, O5's dell function):

dell 'sudo k3s kubectl -n rig-conductor create job --from=cronjob/rig-backup-db proving-db-1'
dell 'sudo k3s kubectl -n rig-conductor logs -f job/proving-db-1'   # expect "rig-backup-db <run> complete"

Any not-covered:<db> goes into BACKUP_EXCLUDE_DBS (P7) or the dump set before go-live. Then list _runs/dell/db/ with the reader token: the run must have its SHA256SUMS. The same for cronjob/rig-backup-agent-home and _runs/dell/home/; each unlisted:<namespace>/<claim> goes into one of the claim lists.

Known gaps

• If the rig-conductor namespace enforces a Pod Security level, check the pods against it on the Dell. agent-home's hostPath and uid 0 fail both baseline and restricted.
• The jobs need egress to Postgres, R2 and Discord only; a NetworkPolicy for that (security review §8) is not here yet.