Deployment¶

All components are managed by FluxCD via the rig-gitops repo.

GitOps Structure¶

rig-gitops/
├── clusters/gcp/
│   └── rig-conductor.yaml          FluxCD Kustomization (entrypoint)
└── apps/rig-conductor/
    ├── namespace.yaml             rig-conductor namespace
    ├── rig-conductor-api-source.yaml GitRepository: rig-conductor repo
    └── rig-conductor-api-kustomization.yaml  Kustomization: API + PostgreSQL

Components¶

Component	Image	Managed By	Source
rig-conductor API	`ghcr.io/dashecorp/rig-conductor:sha-<commit>`, pulled with `ghcr-pull-secret` (bumped by the `chore: pin rig-conductor image` PR)	Kustomization (`deploy/`)	rig-conductor repo
PostgreSQL 16.13 + pgvector	`pgvector/pgvector:pg16@sha256:7d400e34…` (digest-pinned, rc#2332)	Kustomization	rig-conductor repo
Backup	CronJob `rig-conductor-backup`, 02:00 UTC daily, `google/cloud-sdk:slim@sha256:cfca8415…` (digest-pinned, rc#2330)	Kustomization	rig-conductor repo
Restore test	CronJob `rig-conductor-restore-test`, same image, `suspend: true` pending one manual proving run (rc#2310, rc#2315)	Kustomization	rig-conductor repo
Off-site backup (Dell only)	CronJobs in `deploy/backup-r2/`, `ghcr.io/dashecorp/rig-backup` by tag and digest, `suspend: true`; not in `deploy/`'s render. See Off-site backups to R2	Dell Flux Kustomization (P7, rig-gitops#897)	rig-conductor repo
Restore drill (Dell only)	CronJob `rig-backup-drill` in namespace `rig-backup-drill`, `deploy/backup-r2/drill/`, suspended for good, run on demand. See Restoring from R2	Dell Flux Kustomization (P7, rig-gitops#897)	rig-conductor repo
Valkey 9.0.6	`docker.io/valkey/valkey:9.0.6@sha256:63346cb2…` (upstream, digest-pinned; auth off; replaced Bitnami 9.0.3 on 2026-09-23, rig-gitops#819)	StatefulSet `valkey-upstream`, Service `valkey-primary`	rig-gitops

Coordination Cron¶

Every 5 minutes, a CronJob runs the coordination loop:

Check event store health
Get current priority queue
Search GitHub for agent-ready issues across managed repos
Sync new issues to event store as ISSUE_APPROVED events
Post summary to #conductor-e via Discord webhook

Cost: ~$0.005 per run ($1.44/month with Haiku).

Secrets¶

The rig-conductor-secrets secret (created manually, not in GitOps):

Key	Purpose	Used By
`discord-bot-token`	Discord bot (rig-conductor, formerly ATL-E)	Rig Agent Runtime
`anthropic-api-key`	rig-conductor dedicated API key (see below)	Rig Agent Runtime
`github-token`	GitHub PAT for MCP tools	Rig Agent Runtime, Cron
`postgres-password`	PostgreSQL auth	API + PostgreSQL
`discord-webhook-url`	Cron results webhook for #conductor-e	Cron
`database-url`	Empty (in-memory mode for Rig Agent Runtime memory)	Rig Agent Runtime
`github-webhook-secret`	Required. GitHub webhook HMAC (`X-Hub-Signature-256`) — see below	API

github-webhook-secret (required, rc#1925; formerly optional, rc#1689). Wired into deploy/k8s/deployment.yaml as a secretKeyRef without optional: true, so a pod cannot start without it. Program.cs reads GITHUB_WEBHOOK_SECRET from IConfiguration and constant-time-compares the X-Hub-Signature-256 header on every delivery (fail-closed): if the secret is unset at request time the endpoint returns 503 webhook_secret_not_configured and fires a #admin alert (GitHub will auto-retry); if the header is missing or wrong the endpoint returns 400 Signature mismatch and fires a #admin alert (GitHub will not retry). The previous fail-open behaviour — verification skipped when the env var was empty — let any party who knew the URL POST forged GitHub events into the orchestrator.

Operator requirement. The same value must be set on every producer: (a) dashecorp/infra Actions secret WEBHOOK_SECRET (signs every per-repo github_repository_webhook via OpenTofu) and (b) every GitHub App webhook config (dev-e-bot, etc.). Roll-out order (per #1925): mint + Bitwarden → infra WEBHOOK_SECRET + tofu apply → set on each App → add to rig-conductor-secrets + Flux apply → conductor restart. If the conductor is rolled before producers are signing with the new value, every delivery returns 400 + alert (GitHub will not retry — operator must Redeliver from the GitHub UI).

Multi-location fencing (off by default)¶

The conductor runs inert until LOCATION is set: no tenant-map call, no check, no extra field or header. The values are set per location in rig-gitops, not here. See multi-location fencing (rig-conductor#2561).

Env var / config key	Purpose	Secret?
`LOCATION`	This conductor's location id (e.g. `gcp`, `dell`). Set and not blank: fencing on. Must equal the `location` the map uses.	no
`TENANT_MAP_BASE_URL`	rig-cockpit-worker base URL; the conductor calls `GET {TENANT_MAP_BASE_URL}/api/tenants`. Unset while `LOCATION` is set: every tenant is fenced (`map_missing`).	no
`TENANT_MAP_ACCESS_CLIENT_ID`	Cloudflare Access service-token client id, sent as `CF-Access-Client-Id`.	yes: from a Kubernetes secret
`TENANT_MAP_ACCESS_CLIENT_SECRET`	Cloudflare Access service-token client secret, sent as `CF-Access-Client-Secret`.	yes: from a Kubernetes secret
`ReconcileSweep__DryRun` (`ReconcileSweep:DryRun`)	The activation reconcile sweep (rig-conductor#2562), which runs only with `LOCATION` set. Default `true`: it only logs and records what it would do. `false` makes it act.	no

The conductor reads them through IConfiguration, so the env var names above are the keys. Check the result at GET /api/location-fence; the startup log prints Location fence ON: LOCATION=… when it is on.

Per-Agent Anthropic API Keys¶

Each rig agent uses a dedicated Anthropic API key so usage is broken down per agent in the Anthropic Console. No shared keys.

Agent	Secret Name	Key Name in Console
rig-conductor	`rig-conductor-secrets`	`rig-conductor`
Dev-E	`dev-e-secrets`	`dev-e`
Review-E	`review-e-secrets`	`review-e`

Why separate keys¶

The Anthropic Console shows token usage grouped by API key. With a single shared key, you cannot tell which agent spent what. With one key per agent, you get free per-agent cost tracking without any additional instrumentation.

Creating a dedicated key¶

Go to console.anthropic.com/settings/keys
Click Create Key, name it after the agent (e.g. rig-conductor)
Copy the key value — it is only shown once
Update the agent's K8s secret:

# Create or update the secret
kubectl create secret generic rig-conductor-secrets \
  --namespace rig-conductor \
  --from-literal=anthropic-api-key=<new-key> \
  --from-literal=discord-bot-token=<existing-value> \
  --from-literal=github-token=<existing-value> \
  --from-literal=postgres-password=<existing-value> \
  --from-literal=discord-webhook-url=<existing-value> \
  --from-literal=database-url="" \
  --dry-run=client -o yaml | kubectl apply -f -

# Restart the agent to pick up the new key
kubectl rollout restart deployment/rig-conductor-rig-agent-runtime -n rig-conductor

Verifying key separation¶

After updating all secrets, verify in the Anthropic Console that usage is correctly attributed — each agent's calls should appear under its own key name.

FluxCD Commands¶

# Check status
flux get kustomizations
flux get helmreleases -A

# Force reconciliation
flux reconcile source git flux-system
flux reconcile kustomization rig-conductor

# Check what FluxCD manages
kubectl get pods -n rig-conductor

Building Images¶

Images are built automatically via GitHub Actions CI on push to main. The rig-conductor API is pushed to both GHCR and Google Artifact Registry under the same tags.

The conductor pulls from GHCR, with ghcr-pull-secret in the rig-conductor namespace, and the update-gitops pin writes the GHCR line (rig-gitops#896 PR-4). AR is still pushed on every build, so rollback is a revert of that PR.

Image	Registry Path
rig-conductor API (pulled)	`ghcr.io/dashecorp/rig-conductor:sha-<commit>`
rig-conductor API (still pushed)	`europe-north1-docker.pkg.dev/invotek-github-infra/dashecorp/rig-conductor:sha-<commit>`
Rig Agent Runtime	`europe-north1-docker.pkg.dev/invotek-github-infra/dashecorp/rig-agent-runtime:latest`

Flux rolls the pod when the pin PR merges. To force a restart:

kubectl rollout restart deployment/rig-conductor-api -n rig-conductor
kubectl rollout restart deployment/rig-conductor-rig-agent-runtime -n rig-conductor

Verifying¶

# All pods
kubectl get pods -n rig-conductor

# API health
kubectl port-forward -n rig-conductor svc/rig-conductor-api 18080:8080 &
curl http://localhost:18080/health

# API logs
kubectl logs -n rig-conductor -l app=rig-conductor-api --tail=20

# Rig Agent Runtime logs
kubectl logs -n rig-conductor -l app.kubernetes.io/name=rig-agent-runtime --tail=20

# Cron job logs (last run)
kubectl logs -n rig-conductor -l app.kubernetes.io/component=cron --tail=20