Skip to content

Operator runbook: GDPR Art.17 tenant erasure

DELETE /api/admin/tenants/{tenantId} irreversibly erases a tenant: DROPs its physical event + memory DBs (rig_t_<id>_evt / _mem), removes its Marten master record + control-plane allowlist rows, and writes a durable TENANT_ERASED audit record to rig_control. The inverse of POST /api/admin/tenants. Design of record: docs/2026-06-15-tenant-erasure-drop-design.md.

Irreversible — no undo. Physical backups are a separate, operationally-gated step (delete the tenant's backup-bucket prefix; see docs/2026-06-15-multi-tenancy-backup-erasure-design.md). For the off-site R2 copies, see Off-site backups in R2 below.

Auth (two factors + confirmation)

  1. Cloudflare Access at the edge (all /api/admin/*).
  2. Operator allowlist — an allowlisted X-Operator-Discord-Id (OperatorOverrideAuthGuard; empty allowlist ⟹ 403). Stronger than onboarding because destructive.
  3. Confirmation — X-Erasure-Confirm must equal {tenantId}, else 428.

Operator-token gate (rc#2369). This route requires X-Rig-Operator-Token once OPERATOR_ROUTE_GATE=enforce; until then it is report-only (logged, never refused). See docs/2026-09-24-operator-route-gate.md.

curl -sS -X DELETE "$CONDUCTOR/api/admin/tenants/acme" \
  -H "X-Operator-Discord-Id: <allowlisted-snowflake>" \
  -H "X-Erasure-Confirm: acme" \
  -H "X-Erasure-Reason: GDPR Art.17 request ref ABC-123"
# 200 { ok, tenantId, deregistered, evtDbDropped, memDbDropped, allowlistRowsRemoved }
Code Meaning
200 Erased (fields report what was removed/dropped).
400 Invalid tenant id (format / reserved).
401 / 403 Operator identity missing / not on the allowlist (fail-closed).
409 Policy refused — tenant-0 (invotek), blank, or unknown/unprovisioned tenant.
428 X-Erasure-Confirm missing or ≠ {tenantId}.

invotek (tenant-0) is never erasable. An already-erased tenant 409s (no longer provisioned).

Order (the survivability invariant)

gate-check → write TENANT_ERASED to rig_control (FIRST, survives the DROP) → RemoveTenantAsync (evicts the in-memory tenant cache + disposes the pool + deletes the master record) → delete Tenant / RepoTenant / TenantNotificationSink rows → DROP DATABASE … WITH (FORCE) (idempotent IF EXISTS). The audit record (TenantErasureRecord doc in rig_control, id "{tenantId}:{utcTicks}") names the verified operator, reason, dropped DBs, and timestamp — the Art.17 trail (pod logs are a redundant signal only).

Partial-failure recovery

If the DROP fails after the deregister (rare with WITH (FORCE) + IF EXISTS), the tenant is gone from the registry/allowlist but its DBs may be orphaned. The audit record + log name them; drop manually as the conductor superuser (connected to rig_conductor, NOT the tenant DB): DROP DATABASE IF EXISTS "rig_t_acme_evt" WITH (FORCE); / …_mem. Re-invoking the endpoint 409s (already deregistered), so the manual drop is the recovery path.

DROP ROLE rig_t_<id>_app / _ro is not part of erasure today (interim superuser model; no per-tenant roles created) — add it with the same guards when the per-tenant role hardening slice lands.

Off-site backups in R2 (rig-gitops#897)

Every location writes its own bucket, rig-backups-<location> (today only rig-backups-dell), so do this in each location's bucket. The tenant's dumps sit under backups/<tenant>/, kept 35 days and locked 14. If the tenant has claims of its own in agent-home, those archives are agent-home/<location>/<run>/<namespace>_<claim>.tar.gz.age, kept and locked 7 days. _runs/ holds only SHA256SUMS files and encrypted manifests: they name the tenant's object keys and its tables' row counts, never rows, and lifecycle removes them by day 35.

  1. Delete what the lock allows, now. List the tenant's objects with the reader token, then DELETE each one older than 14 days with the writer token (204). R2 refuses younger objects: the bucket lock keeps every object for 14 days after upload.
  2. Record what stays. Note in the erasure record each key that remains, with the date its lock ends. Delete those on that date, or let lifecycle remove them by day 35. Both are inside the DPA tail (erasure design §5); tell the DPO.
  3. Restore-re-erase. A restore from any run older than the erasure brings the tenant back. The restore runbook skips the erased tenant's dumps and re-runs this endpoint before serving.
# Credentials only through curl's config, on a pipe: read -rs AK; read -rs SK (reader to list, writer to delete)
EP='https://<ACCOUNT_ID>.eu.r2.cloudflarestorage.com'; B=rig-backups-dell; T=acme
curl -q --proto =https --noproxy '*' --globoff -sS --fail-with-body -K <(printf 'user = "%s:%s"\n' "$AK" "$SK") \
  --aws-sigv4 'aws:amz:auto:s3' "$EP/$B/?list-type=2&prefix=backups%2F$T%2F" | tr '<' '\n' | grep -E '^(Key|LastModified|IsTruncated)>'
curl -q --proto =https --noproxy '*' --globoff -sS --fail-with-body -K <(printf 'user = "%s:%s"\n' "$AK" "$SK") \
  --aws-sigv4 'aws:amz:auto:s3' -X DELETE "$EP/$B/backups/$T/dell/<run>/evt.dump.age"
unset AK SK

If the listing says IsTruncated>true, list again from the last key (&start-after=…) before deciding anything is gone.

Scope — what the DROP covers, and what it deliberately does not

Tenant erasure is structural. A tenant's read-model documents and its granular token_usage events live only inside that tenant's physical database (rig_t_<id>_evt): the inline projections write through the tenant-scoped session, and the cost/usage queries open one QuerySession(tenant) per tenant database. So DROP DATABASE takes the read models and the underlying events with it — there is nothing left to scrub per-row, and CrossTenantIsolationTests proves a tenant-A-resolved connection returns zero of tenant-B's rows.

Because of that, do not add a TenantId column to the read models and do not introduce a WHERE tenant_id read path in order to "make them erasable." Isolation is the per-request connection, not a row predicate; a predicate would be a second, weaker enforcement path that masks connection-resolution bugs. This supersedes the original (2026-06-08) #1495 acceptance criterion. Rationale of record: docs/2026-06-14-multi-tenancy-read-model-erasure-reconciliation.md (rc#1773, resolving the #1495 design tension); per-tenant DBs are the cutover in #1477 / docs/2026-06-12-multi-tenancy-pr5-invotek-cutover-runbook.md.

Two cases the per-tenant DROP does not cover (both DPO-gated — tracked separately, not part of this runbook):

  • Subject-level erasure — one named natural person exercising Art.17 across read models + events. You cannot DROP a whole tenant's database to erase a single contributor, and tenant-0 (invotek) can never be DROPped (it holds invotek's live data) yet carries third-party contributor PII. Mechanism + legality: #1496.
  • Cost/usage minimisation — the retained token_usage shape (AgentId + repo full name + issue number) is over-retention, not a minimised financial record. Rolling it up to tenant-level monetary totals is gated on the DPO's Art.17(3) minimal-retained-field-set judgment (#1486 §3.4). Cost track of #1495.