Skip to content

Operator-token gate on the destructive write routes (rc#2369)

Slice 1 of #1999 and infra#448.

Why

Nothing in-cluster restricts who reaches the conductor: - there is no NetworkPolicy in rig-conductor, dev-e or review-e; - NodePort 30080 serves the tailnet; - Cloudflare Access fronts only rig-conductor.dashecorp.com.

The rig's threat model treats agent pods as untrusted, because a hostile issue body steers them. Before this, any of them could start the GPU pod (spend), force-done an issue, create a tenant, or delete agents and repo learnings, with no credential. Merging was already closed by #2350, and tenant erasure and POST /api/admin/overrides by their empty allowlist. That left these.

The gated routes

Route Scope
POST /api/pods/{name}/start wake: the wake token or the operator token
POST /api/pods/{name}/stop operator
POST /api/admin/issues/force-done operator
POST /api/admin/tenants, DELETE /api/admin/tenants/{tenantId} operator (the DELETE keeps its own allowlist and confirm-header guards)
POST /api/admin/overrides operator (it keeps its own allowlist guard)
DELETE /api/agents/{agentId}, DELETE /api/agents/offline, DELETE /api/repo-learnings operator

Not gated, deliberately: - GET /api/pods/{name}, which Rig-E, the SPA and the worker's liveness monitor poll; - GET /api/admin/tenants and GET /api/admin/overrides, read-only lists behind the cockpit and the conductor's static dashboard; - GET/POST /api/repo-learnings, which the runtime uses.

OperatorRouteGateE2eTests derives the gated set from the app's own endpoint table. Every DELETE, and every non-GET route under /api/admin/ or /api/pods/, must carry the gate, and nothing else may. A new destructive route without it fails the build.

How it decides (OperatorRouteGatePolicy, Core)

  • Header: X-Rig-Operator-Token. It is compared in constant time with RIG_OPERATOR_TOKEN, and, on the wake route only, RIG_WAKE_TOKEN. Rig-E gets the wake token, so it can wake its GPU without holding a credential that could stop it, force-done work or touch tenants.
  • Configured tokens: trimmed, because a k8s secret often ends in a newline. One shorter than 32 characters is ignored.
  • OPERATOR_ROUTE_GATE: report by default, and for anything that is not exactly enforce.
  • Report never refuses. Its outcome says what enforce would have done: report:no-token, report:bad-token or report:unconfigured.
  • Enforce refuses a missing or wrong token, and refuses everything when no usable token is configured. Enforcement without a secret fails closed.
  • A refusal is 403, never 401. The cockpit SPA and the iOS app treat a 401 as a logout.
  • Every gated call is logged, allowed or not: 
    [OperatorGate] <outcome> <METHOD> <route> from <ip> ua="<ua>" credential=<operator|wake|none>

It is an endpoint filter attached per endpoint (.RequireOperatorToken(scope)), not path-prefix middleware. Two reasons: - some gated routes share a path with open ones; - routing matches /API/Admin/… case-insensitively, which a prefix check would miss. The e2e asserts the mixed-case path is refused.

Query and body parameters bind before endpoint filters run. A malformed request is therefore a 400 that reaches neither the gate nor the handler.

Rollout

  1. This change: nothing is configured in deploy/, so production runs in report mode and refuses nothing. Read the callers from the log: kubectl logs -n rig-conductor deploy/rig-conductor-api | grep '\[OperatorGate\]'
  2. Callers:
  3. cockpit worker: inject the operator token on stop, after its header-copy loop, and add the header name to STRIPPED_INBOUND_HEADERS;
  4. Rig-E: send the wake token on start, and log a 401/403 instead of swallowing it (rar endpoint-resolver.js:107-110);
  5. runbooks: add the header (gpu-brain.md, runpod-pod-control.md, api.md, the onboarding and erasure runbooks).
  6. Operator only (credential policy): mint both tokens. They go into a new Secret, rig-conductor-gate-secrets (rig-gitops#846), which is separate from rig-conductor-secrets so it could be created from the public age recipient alone. The same values go to Rig-E (rig-e-wake-secrets) and to the worker (its RIG_OPERATOR_TOKEN repo secret). deploy/k8s/deployment.yaml reads both tokens as optional secretKeyRefs (#2372). Order: the Secret has to exist before the deployment change rolls out, because env from secretKeyRef is read only at container start. The conductor then logs [OperatorGate] ok … credential=operator on a Stop GPU and credential=wake on a Rig-E wake.
  7. When the report log shows only the expected callers, all presenting a token, set OPERATOR_ROUTE_GATE=enforce. Done 2026-09-25 (#2435).
  8. The 24 h report log (2026-09-24 12:26Z to 2026-09-25 12:28Z) held three calls, each with a token: two Stop GPU calls from the cockpit (credential=operator) and one wake from Rig-E (credential=wake).
  9. A code search found no other caller of any gated route.

To roll back to report mode, remove the OPERATOR_ROUTE_GATE env from deploy/k8s/deployment.yaml.

Out of scope, because each needs a per-agent identity: forged /api/events, and Valkey's unauthenticated assignments:*, which is NodePort 30079.