Multi-location fencing (rig-conductor#2561)¶

Part 2 of 8 of the multi-location rig (dashecorp/infra#508). Pure rule in LocationFencePolicy, tests first.

Decisions taken elsewhere (not reopened here)¶

The rig runs in more than one location: the GCP cluster now, a Dell k3s cluster next.
Every tenant lives in exactly one location. An operator moves it by hand from Rig Cockpit.
The tenant → location map lives in Cloudflare D1, owned by rig-cockpit-worker, served at GET /api/tenants behind Cloudflare Access (rig-cockpit-worker#252/#253).
A move drains the tenant for at most 15 min, then forces the switch: the epoch goes up by 1 and the old location is fenced for that tenant. The new location rebuilds work state from GitHub; nothing is copied.

The rule¶

With LOCATION set, the conductor may write to GitHub or dispatch work for a tenant only when all hold:

the map is fresh: the last successful fetch is at most 60 s old;
the tenant has exactly one row in the map;
that row's location equals LOCATION;
that row's state is active.

Anything else fails closed: the action is skipped and logged with its reason. Comparisons are trimmed and case-insensitive.

Reason	When
`map_missing`	no map has been fetched, and none is being fetched (`TENANT_MAP_BASE_URL` unset)
`map_unreachable`	no map has ever been fetched, and the last fetch failed
`map_stale`	the last successful fetch is more than 60 s old
`tenant_unknown`	the action has no tenant (no wired path sends a blank one; a guard)
`tenant_not_in_map`	no row for the tenant (an empty map fences everyone)
`tenant_ambiguous`	more than one row for the tenant
`mapped_elsewhere`	the row's `location` is another location
`tenant_not_active`	the row's `state` is `draining`, `reconciling` or anything but `active`
`fence_error`	the check itself failed (e.g. cancelled); never an exception to the caller

Inert by default. With LOCATION unset or blank nothing changes: no map call, no check, no extra field or header, and the GitHub write handler below is not even added. This code ships to the live GCP rig before the map exists.

Configuration¶

Read through IConfiguration, so each key is also the env var name. Values are set per location in rig-gitops.

Key	Meaning
`LOCATION`	this conductor's location id, e.g. `gcp`. Set: fencing on.
`TENANT_MAP_BASE_URL`	rig-cockpit-worker base URL; the conductor calls `GET {TENANT_MAP_BASE_URL}/api/tenants`.
`TENANT_MAP_ACCESS_CLIENT_ID`	Cloudflare Access service-token id, sent as `CF-Access-Client-Id`. Secret.
`TENANT_MAP_ACCESS_CLIENT_SECRET`	Cloudflare Access service-token secret, sent as `CF-Access-Client-Secret`. Secret.

LOCATION set without TENANT_MAP_BASE_URL fences every tenant (map_missing) and logs a warning at startup. ReconcileSweep:DryRun belongs to the activation reconcile sweep below.

The map contract (rig-cockpit-worker#252/#253)¶

The worker fixed GET /api/tenants as camelCase JSON. The conductor reads it into one DTO, TenantLocationMapResponse:

{
  "tenants": [
    { "id": "invotek", "name": "Invotek", "githubOrg": "dashecorp", "installationId": 12345678,
      "location": "gcp", "epoch": 3, "state": "active", "moveId": null, "updatedAt": 1790990000000 }
  ],
  "repoOverrides": [ { "repo": "Stig-Johnny/nutri-e", "tenantId": "invotek" } ],
  "generatedAt": 1790990005000
}

Field	Type	Conductor use
`tenants[].id`	string, required	the conductor's own tenant id: the worker syncs its tenants from this conductor's `GET /api/admin/tenants`. A row without it is dropped.
`tenants[].location`	string	compared with `LOCATION`. Must use exactly the ids rig-gitops sets.
`tenants[].epoch`	integer, required	+1 on every move. Stamped on events and assignments. A row without it is dropped.
`tenants[].state`	`active` \| `draining` \| `reconciling`	only `active` lets the location act.
`tenants[].name`, `githubOrg`, `installationId`	string / integer	read, not used by the fence.
`tenants[].moveId`, `updatedAt`, any other field	string or null / integer (epoch ms) / any	not read: unknown members are ignored, so their type cannot fail the fetch.
`repoOverrides`	`[{repo, tenantId}]`	ignored: the conductor resolves repo → tenant itself (below).
`generatedAt`	integer, epoch ms (the worker sends `Date.now()`); a string is also accepted	informational, read as ISO-8601 UTC. Never fails the fetch: another type reads as null (rig-conductor#2716). Freshness is this conductor's own fetch time, not the worker's clock.

200 with that body, application/json. Any other status, an Access login page, a body without a tenants array or a non-integer epoch is a failed fetch.
Every tenant the rig serves appears exactly once. Missing or duplicated: that tenant is fenced everywhere.
Each conductor fetches at most every 30 s, and every 5 s while fetches fail; its timeout is 5 s.

Cache and fail-closed behavior¶

LocationFence (singleton) holds the last good map.

A map younger than 30 s is served from the cache. The first check after that refetches; concurrent checks share one fetch.
A failed refetch keeps the last good map, which stays usable until it is 60 s old (map_stale after that). The next try is 5 s later, so an outage costs one timeout per 5 s, not one per call.
The fence never throws. A refused write is skipped and returns normally; a refused GitHub HTTP call gets a synthetic 503 (below), which callers already handle as a failed GitHub call.

Which tenant a repo is¶

Every check that starts from a repo uses the GitHub webhook's own rule, TenantFiling (Core): the tenant the resolver answers, else invotek, also when the resolver throws. So a repo that maps to no tenant (e.g. Stig-Johnny/*) is fenced as invotek, exactly as its deliveries are recorded and dispatched, never as "no tenant". The webhook, MergeGate, POST /api/merge and the GitHub write handler all call it. (MergeGate still attributes its own events with the stricter CoalesceInDefaultOrg.)

Where it is wired¶

Path	How
Assignment publish: every XADD/LPUSH to `assignments:` / `signal:` (webhook, `/api/events` `ISSUE_ASSIGNED`, all sweeps)	`ValkeyStreamPublisher.PublishAssignmentAsync`: no write for a fenced tenant (a blank tenant is `invotek`, as for the stream key)
Planner intake (`assignments:rig-planner*`), webhook and `POST /api/planner/trigger`	`ValkeyStreamPublisher.PublishPlannerSignalAsync` returns 0; the webhook records `skipped:location_fenced`
MergeGate merge (webhook, `POST /api/merge`, cockpit release, rechecks)	`MergeGate.Merge` returns `Deferred("location_fenced: <reason>")` before the guard: no GitHub call, no cleared condition
`POST /api/merge`	`200 {"triggered": false, "fenced": true, "reason"}` before the condition is set
Pull dispatch: `GET /api/assignments/next`, `GET /api/pr-reviews/next`	`204` (the claim appends nothing)
GitHub webhook	`200` for a fenced tenant. Events are recorded (with `location`/`epoch`); the `agent-ready` branch stops after `ISSUE_APPROVED` and releases its dedup key. Each refusal is handled where the write is, so there is no catch for fenced deliveries: an unrelated exception answers `500` exactly as for any tenant.
Webhook review, iteration and CI-fix dispatch	the site asks the publisher's own check (`LocationFenceScope.CheckDispatchAsync`) before it publishes. Refused: logged, `200 {"skipped": true, "fenced": true, "reason"}`, and nothing recorded: no `REVIEW_ASSIGNED`, no review dedup stamp, no "→ agent" broadcast.
Sweeps that dispatch or write GitHub: `IssueScanService`, `ReviewScanService` (both passes), `ReconciliationService`, `SloEnforcerService`, `StaleAgentReadyWatcher`, `MainGuardService`, `MainRedRecoverySweeperService`, `OrphanScanService`, `StuckWatcherService`, `SelfImprovementService` (tenant-local and fleet-wide)	the per-tenant pass is skipped for a fenced tenant (`LocationFenceScope.FencedAsync`)
`ImagePinDispatchService` (the pin PR on `dashecorp/rig-gitops`; the image-pin auto-merge itself was deleted in rc#2306)	checked once for `invotek` before the first git object
Every other GitHub write: labels, comments, issue/PR close/reopen, reviews, hooks, git objects (webhook inline writes, `TaskSpecConsumerService`, `UmbrellaIssueGuardService`, `DuplicateCloseService`, `PinIssueSupersedeService`, `RepoDiscoveryService`, `OperatorOverrideService`, the cockpit task issue and audit comment, `HttpGitHubIssueWriter`)	`LocationFenceGitHubHandler`, added to every `IHttpClientFactory` client only when `LOCATION` is set. A `POST`/`PUT`/`PATCH`/`DELETE` to `api.github.com/repos/{owner}/{name}/…` goes out only if the repo's tenant (`TenantFiling`, above) is allowed; otherwise `503`, header `X-Rig-Location-Fenced: <reason>`, nothing sent.

With LOCATION set, IssueScanService also reads each repo's open PRs. It does not take up an agent-ready issue that an open PR closes (Closes/Fixes/Resolves, same repo; rig-conductor#2562). A tenant moved here starts with an empty projection, so "not in the projection" no longer means "nobody is working on it". If the open PRs cannot be read, the scan skips that repo for that pass.

Left alone, on purpose:

Reads, and GitHub App token mints (POST /app/installations/{id}/access_tokens): they change nothing.
Discord and #admin alerts, the operator push relay, memory-MCP writes: not GitHub, and an operator should still hear from a fenced location.
Stream maintenance (ReclaimStalePendingAsync, ReleaseOwnPendingAsync, StreamReclaimService): it re-delivers messages already in this location's Valkey, and new ones are fenced at publish.
Sweeps that touch only this location's database (retention, index, archive, phantom cleanup, issue-row repair): no GitHub write, no dispatch.
RunPod pod control: a rig resource, not a tenant's.
MergeService: not registered.

Location and epoch on outputs¶

Only when LOCATION is set:

Every assignment and planner-intake stream entry gets location (this LOCATION) and epoch fields.
Every appended Marten event gets location and epoch headers, next to tenant_id, at all four append sites (MartenEventStore.AppendAsync, the PR-review claim, the operator task store, the pod wake history).
epoch is the tenant's epoch in the map this conductor holds, whatever its freshness or location, and -1 when it has no single row. Events of a fenced tenant are still recorded, with this location, so a mixed-up write can be traced later.

Operating it¶

GET /api/location-fence: LOCATION, the map's age, the rows, the last fetch error, and the last 100 refusals with reasons. It never fetches.
Logs: [LocationFence] fenced <action> for tenant <t> at <location> (<target>): <reason>, once per minute per tenant, action and reason; tenant map fetch failed: … per failed fetch.

Activation reconcile sweep¶

rig-conductor#2562, part 6 of 8. A move is a fresh start: the new location has none of the tenant's Valkey or Postgres state. MergeGate merges only on events (review-approved and ci-passed in Valkey, 24 h), so a PR approved before the move would never merge, and the old location's assignments are gone. When the tenant becomes active here under a new epoch, ReconcileSweep rebuilds that state from GitHub once. Its rules are in ReconcileSweepPolicy.

It walks the tenant's discovered repos with the tenant's token (TenantGitHubToken):

Step	What it does
MergeGate	For each open PR (first 100 per repo): `ci-passed` when every check on the head passed; `review-approved` when review-e-bot's latest verdict approves exactly the head (the guard's own rule). It sets them and runs the normal evaluation (`MergeGate.SetConditionsAndTryMerge`). It never merges: `Merge()` still checks the fence and the guard. A PR labelled `do-not-merge` is skipped. No `blocked` flag is set: the normal path sets none (`MergeGate.Block` has no caller), and nothing would clear it.
Reviews	The ReviewScan GitHub-poll pass for the tenant: agent PRs and `needs-review` PRs with no binding review on the head.
Issues	IssueScan for the tenant: `agent-ready` issues not in the projection. One an open PR closes is in progress and is skipped (above).

Dispatch goes through those two scans, so their dedup, routing and the publisher's fence apply unchanged. Each scan runs one pass per tenant at a time, so the sweep's pass waits for the scan's own instead of racing its check-then-set dedup.

Dry run reads GitHub, then logs and counts what the sweep would do: [ReconcileSweep] … dry run, would set …, [ReviewScan] Dry run: would route …, [IssueScan] Dry run: would take up …. It sets no condition and dispatches nothing.

Fail closed. The sweep asks the fence before each repo, each condition write and each scan. If the fence refuses the tenant, or the epoch is not the one the sweep started for, the sweep stops (stopped, with the fence's reason or epoch_changed). It also stops at a 403 or 429, or when X-RateLimit-Remaining is below 1000 (rate_limited, the issue-row repair's floor). A failed read of one repo or PR is counted (readFailures), logged and skipped. Every step is idempotent: a restored condition, a scan pass, the guarded merge.

When it runs¶

ReconcileSweepService is off with LOCATION unset: it returns at start and reads nothing. With LOCATION set, it waits 60 s for repo discovery. Then, every 30 s (the map's cache window), it looks at each tenant whose database is registered here. It uses ILocationFence.PeekAsync, which gives CheckAsync's decision but records no refusal, so this polling does not flood /api/location-fence. A tenant is swept when the fence allows it here (a fresh map, one row, location == LOCATION, state == active) and its epoch is higher than the last epoch reconciled for it. Each epoch is swept once, and a move away and back is a new epoch.

The mark is durable, in Postgres. Every sweep appends one ReconcileSweepFinished to reconcile-sweep:{LOCATION} in the tenant's own database. The last reconciled epoch is the newest completed one of the same mode. The trigger reads it when a tenant is due, then keeps it in memory. A restart therefore does not repeat a completed sweep. It is an event, not a document or a Valkey key, because state here is derived from event streams. That one event is also the Rig Cockpit record. Valkey is the store a move leaves empty, and a Valkey reset would repeat every sweep. A sweep that stopped is retried after 5 min, and so is one that threw, whose stream could not be read or whose record could not be written, so it is never retried on every tick. A sweep counts as done only once its record is written: one that ran but was not recorded is swept again (#2601).

Configuration¶

Key	Meaning
`ReconcileSweep:DryRun` (env `ReconcileSweep__DryRun`)	Default `true`: the sweep reads GitHub, logs and records what it would do, and sets or dispatches nothing. Only `false` makes it act. A value that is not a bool stays a dry run, with a warning at start.

A dry run marks its epoch for dry runs only. Setting false and restarting sweeps the current epoch once more, for real.

The record¶

ReconcileSweepFinished (RECONCILE_SWEEP_FINISHED in /api/events/recent, so the Rig Cockpit Activity view shows it) carries tenantId, epoch, location, dryRun, outcome (completed or stopped), reason, startedAt and finishedAt. It also carries the counts: repos, openPrs, ciPassed, reviewApproved, mergeEvaluations, doNotMerge, linkedIssues, reviewsDispatched, issuesDispatched, gitHubReads and readFailures. The log has one [ReconcileSweep] … sweeping line and one summary line per sweep.

Known gaps (sweep)¶

The PR lists read one page: 100 open PRs per repo for the sweep and for IssueScan's in-progress rule, and 10 for ReviewScan's poll (unchanged).
Only agent-ready issues and review-pending PRs are re-dispatched. A PR whose CI failed, or whose changes were requested, before the move gets its CI-fix or iteration dispatch only from the next webhook.
An issue counts as in progress only when an open PR closes it. If the old location stopped an agent before it opened its PR, the issue is taken up again from the start, which is the fresh-start rule.
A tenant needs its database registered at this location; otherwise neither the sweep nor the scans run for it.
One conductor replica is assumed, as the scans' in-memory dedup already assumes. Two replicas could both sweep an epoch; every step is idempotent.
Turning LOCATION on at a location that already holds a tenant's state (GCP at first rollout) also sweeps each owned tenant once at its current epoch. That is a dry run by default. Live, it re-triggers MergeGate for approved, green PRs, and the guard still decides.

Rollout order¶

This change merges and deploys everywhere, inert (LOCATION unset). Nothing changes on the live GCP rig.
rig-cockpit-worker GET /api/tenants is live with the contract above, a row for every tenant, and the Access service token works.
rig-gitops sets LOCATION per location, with TENANT_MAP_BASE_URL and the two Access secrets: GCP first, then check /api/location-fence (fresh map, no refusals for the tenants it owns); then Dell.
The activation reconcile sweep stays a dry run (ReconcileSweep:DryRun unset) until a test move to the Dell shows, in its RECONCILE_SWEEP_FINISHED event and the would set / would route / would take up lines, what it would do. Then the operator sets ReconcileSweep__DryRun=false per location in rig-gitops.

Known gaps¶

A resolver failure files a repo under invotek, as the webhook does. During a tenant-database outage a location that owns invotek can act for another tenant's repo.
A webhook dispatch site asks the fence, then the publisher asks again. If the map refreshes between the two and the answer flips to refused, the site has recorded a dispatch the publisher then refuses.
After a forced switch, the old location stops within 30 s if it can read the map, within 60 s if it cannot.
LOCATION and the map settings are read at startup; changing them needs a restart.