Skip to content

Fleet-wide agent liveness in the stale-claim watcher

Issue: rig-conductor#2382 (Slice A: #2401, the pure FleetLivenessPolicy; Slice B: this change).

Problem

HEARTBEAT carries no repo, so /api/events records it in the default tenant only (the PR-4M comment above resolvedEventsTenant in Program.cs). The StaleClaimWatcherService runs once per tenant and reads that tenant's AgentStatus rows. In any non-default tenant those rows never get a heartbeat: the row is either missing or was projected from IssueAssigned with no LastHeartbeat. So ComputeStaleness reported no agent record or never sent a heartbeat for a live agent. The claim was released, and the reconciler re-dispatched it on the next tick, every 5 minutes.

Decision

Liveness is fleet-wide, not tenant data.

  • Core port IFleetAgentQuery.GetAllAsync() returns the non-archived AgentStatus rows from FleetLivenessPolicy.LivenessTenant (the default tenant), whatever the caller's tenant.
  • Adapter MartenFleetAgentQuery(IDocumentStore) opens store.QuerySession(LivenessTenant) and applies the same ArchivedAt == null filter as MartenAgentQuery. It is a singleton, so the hosted watcher can inject it, and it is on the rc#1044 IDocumentStore exception list.
  • The watcher reads the fleet rows once per tick. For each claim it uses FleetLivenessPolicy.Merge(tenantRow, fleetRow) for both the stale path (ComputeStaleness) and the phantom path (ComputePhantom). The merge takes the later heartbeat. Every other field (status, current issue) still comes from the claim tenant's row.
  • The Releasing and Tick complete log lines carry tenant=, so post-deploy checks can tell tenants apart.

In the default tenant both reads return the same row, so the merge changes nothing there.

The carve-out

The rc#1687 rule says a cross-tenant read must not drive a control decision. This read is the one reviewed exception, documented on FleetLivenessPolicy. It is safe because a later heartbeat can only delay a release, never cause one. The read is on the default tenant's database only, and nothing is written there. It opens no DI scope, so the tenant-scope guard does not apply to it.

Consequence: liveness is masked per agent id

The three fleets share agent ids (for example dev-e-node). The fleet row is keyed by agent id, not by pod or fleet, so any live pod that uses the id keeps the row fresh. A claim held by a dead pod in one fleet is therefore not released by the stale path while another pod with the same id is still heartbeating. It waits until that id goes quiet everywhere. This is accepted: the alternative was releasing every live claim outside the default tenant every 5 minutes. Per-pod liveness needs a per-pod id and is out of scope.

Tests

  • StaleClaimWatcherServiceTests: fresh fleet row keeps a non-default-tenant claim; no row anywhere is still released as no agent record; default-tenant release unchanged.
  • FleetAgentLivenessE2eTests: a real HEARTBEAT through /api/events, claims in a provisioned second tenant, the real watcher. The heartbeating agent keeps its claim. The ghost is released.

Slice C: the heartbeat watchdog

Issue: rig-conductor#2396.

The flap. AgentHeartbeatWatchdog ran its scan once per tenant, but its _downSince registry is process-global and keyed by agent id. A non-default tenant's row for an agent has no heartbeat. For an agent outside the scale-to-zero set (for example ibuild-e), that tenant's pass marked it DOWN and the default tenant's pass marked it RECOVERED. This repeated every 60 s tick, with an #admin post for each transition, until AgentArchiveService archived the stale row.

The decision. Agent liveness is a fleet property, and the heartbeats live in one tenant. The watchdog evaluates agents only in the pass whose tenant is FleetLivenessPolicy.LivenessTenant (FleetLivenessPolicy.IsLivenessTenant). Every other pass returns at once. _downSince has one writer per tick, and AgentDown/AgentRecovered land in the tenant that holds the heartbeats.

The consequence. The watchdog no longer watches a row that exists only in a non-default tenant. That is correct: an agent that never heartbeats anywhere is not a live agent, so it cannot go down.

Tests: FleetLivenessPolicyTests (the predicate), AgentHeartbeatWatchdogTests (a two-tenant fake runner: no flap, a stale agent still goes down once, a row only in the other tenant is ignored), and AgentHeartbeatWatchdogFleetE2eTests (a real HEARTBEAT, a provisioned second tenant, the real MartenTenantWorkRunner, three scans, no AgentDown/AgentRecovered in either tenant).