Skip to content

Agent Heartbeat Watchdog

Context

On 2026-05-12 the rig-planner pod was in CrashLoopBackOff for 4 hours before an operator noticed via the #planner channel going silent. Conductor had a live AgentStatus document for the pod but emitted no alert. This pattern can recur for any agent pod without the watchdog.

On 2026-06-14 the gap recurred in a subtler form (rc#1763): rig-planner kept restarting (a Discord 4014 crashloop) yet emitted a fresh startup heartbeat on every boot, so heartbeatStatus read ok and the recency-only watchdog stayed silent for ~22 h. Heartbeat recency alone cannot see a crashlooping-but-heartbeating pod — the watchdog now also tracks restart rate.

Decision

Add AgentHeartbeatWatchdog — a BackgroundService that polls agent heartbeat recency and emits structured events + Discord pings when an agent goes dark. For the crashloop blind spot (rc#1763) it additionally derives a restart-rate signal from AGENT_STARTED cadence — data the conductor already records, so no k8s API/RBAC is needed — via the pure AgentCrashLoopPolicy: an agent that restarts ≥ AGENT_CRASHLOOP_MIN_RESTARTS times within AGENT_CRASHLOOP_WINDOW_SECONDS is flagged even while its heartbeat reads ok.

Scale-to-zero agents and presence hooks (rig-conductor#2376)

A stale heartbeat means "down" only for an agent that is expected to be running. The pure AgentDownAlertPolicy decides:

Agent Stale heartbeat Crashloop
Always-on, or an id the conductor does not know down (heartbeat-stale) down
In AGENT_SCALE_TO_ZERO_IDS not down down (crashloop)
human-* presence hook not down down

Why: the KEDA minReplicaCount: 0 fleets stop heartbeating whenever they are idle, and the developer hook only heartbeats while someone is typing. For them a stale heartbeat means "scaled to zero" or "not typing", not "crashed". Consumer-idle is not agent-idle. Each AGENT_DOWN is also a Critical phone push (OperatorPushRelay) and a tenant-sink message, so false ones are expensive. Measured before those agents were archived: review-e fired 212 AGENT_DOWN in 23 active days (173 on a stale heartbeat), and human-Stig-Johnny fired 1,133 in 81 days, almost all followed by a RECOVERED. rig-conductor#2376 will un-archive agents that heartbeat again (the wiring lands in rig-conductor#2383, after this gate; archive policy). Without this gate, that would bring the noise back.

The default set mirrors the rig-gitops ScaledObjects with minReplicaCount: 0 (checked 2026-09-24): review-e, dev-e-node, dev-e-node-codex, dev-e-dotnet, dev-e-python, dev-e-python-codex, rig-planner, rig-planner-codex. dev-e-dotnet-codex (minReplicaCount: 1) and ibuild-e (launchd on the Mac Mini) are not in it, so they still page on a stale heartbeat. An unknown id also pages, so a new always-on agent fails loud.

This changes behaviour for the scale-to-zero agents that were already visible (dev-e-dotnet, dev-e-node, dev-e-node-codex, rig-planner). Their AGENT_DOWN/RECOVERED pair on every idle period stops. A crashloop still pages. A hung pod that holds a claim still has the claim released by StaleClaimWatcher, which posts to #admin when DISCORD_ADMIN_WEBHOOK_URL is set. To restore the old behaviour, set AGENT_SCALE_TO_ZERO_IDS=none.

GET /api/agents is unchanged: heartbeatStatus still reads down for an idle scale-to-zero agent. That field describes heartbeat age, not an alert.

Status bucketing

The GET /api/agents response now includes a heartbeatStatus field derived from lastHeartbeatAt:

Status Condition
ok age < threshold / 2
degraded age ≥ threshold / 2 and < threshold
down age ≥ threshold

Threshold is controlled by AGENT_DOWN_THRESHOLD_SECONDS (default 600 = 10 min). With the default, ok = < 5 min, degraded = 5–10 min, down = ≥ 10 min.

Event stream

Events are written to {agentId}!watchdog (e.g. dev-e-dotnet!watchdog), a sentinel stream that keeps the agent's main stream and issue streams uncluttered — consistent with the !ci and !overrides sentinel-stream conventions used elsewhere in conductor.

AGENT_DOWN

{
  "agentId": "dev-e-dotnet",
  "lastHeartbeatAt": "2026-05-12T10:00:00Z",
  "staleSeconds": 720.5,
  "reason": "heartbeat-stale"
}

reason is heartbeat-stale (no heartbeat within the threshold) or crashloop (the pod restarted ≥ AGENT_CRASHLOOP_MIN_RESTARTS times within AGENT_CRASHLOOP_WINDOW_SECONDS). When both are true, heartbeat-stale wins (the stronger, original signal), unless the stale heartbeat does not count for that agent (scale-to-zero or human-*, see above); then the reason is crashloop. The field defaults to heartbeat-stale, so existing consumers are unaffected.

Emitted once per down-transition. Never re-emitted while the agent stays down.

AGENT_RECOVERED

{
  "agentId": "dev-e-dotnet",
  "heartbeatRestoredAt": "2026-05-12T10:15:00Z"
}

Emitted once when a previously-down agent returns to ok or degraded.

Discord alert

On AGENT_DOWN, a message is posted to #admin. For a stale heartbeat:

🔴 **Agent DOWN** — `dev-e-dotnet` has not sent a heartbeat in **12m**.
Pod may be in CrashLoopBackOff. Check: `kubectl logs -l agent=dev-e-dotnet --tail=50`
_(rig-conductor#793)_

For the crashloop case (heartbeat OK but the pod keeps restarting):

🔴 **Agent CRASHLOOPING** — `dev-e-dotnet` restarted **5×** in the last **30m** (still heartbeating on startup, so its heartbeat status reads OK).
Check: `kubectl logs -l agent=dev-e-dotnet --previous --tail=50`
_(rig-conductor#1763)_

Both hints use a kubectl logs -l agent=<id> label selector — pods are not in a per-agent namespace (the #1763 culprit rig-planner-rig-agent-runtime-0 lives in the rig-agent namespace). The crashloop hint adds --previous to read the just-crashed container's logs.

On AGENT_RECOVERED:

✅ **Agent RECOVERED** — `dev-e-dotnet` heartbeat restored. _(rig-conductor#793)_

Discord posting is skipped when DISCORD_ADMIN_WEBHOOK_URL is unset (events still emitted).

The conductor's own heartbeat (rig-conductor#2356)

The watchdog also watches rig-conductor itself, so the conductor writes its own HEARTBEAT every 60 s. ConductorSelfHeartbeatService does it, a hosted service that replaced a loop in Program.cs.

Old loop ConductorSelfHeartbeatService
Scope one scope, so one Marten session, for the pod's whole life a fresh scope for every beat
A failed write catch { }: nothing logged logged as a warning; the next beat runs on schedule
A hung write stops the loop for good abandoned after 30 s. Each attempt runs on the thread pool and owns its scope, so a write or a Marten session disposal that stalls never blocks the loop, the next beat or shutdown. The abandoned write's late outcome is logged
Successes not logged the first, then every 10th
Start ApplicationStarted (rc#1659) still waits for ApplicationStarted; command-mode runs such as db-assert start no hosted services

Why it changed: after the 2026-09-23 22:06Z rollover, the new pod's first HEARTBEAT collided with the outgoing pod's on the rig-conductor stream. Four Marten concurrency failures were logged, and then nothing. The healthy conductor's lastHeartbeat stopped, and #admin got "Agent DOWN — rig-conductor". A collision can still happen during a rollover, but now only that one beat fails, with a log line, and the next one is written.

Log lines: [SelfHeartbeat] HEARTBEAT for rig-conductor written (N since start), and on a problem … failed; next try in 00:01:00 or … did not finish within 00:00:30; abandoned, later followed by an abandoned HEARTBEAT for rig-conductor finished later or … failed later. SubmitEvent takes no cancellation token, so a timed-out write cannot be cancelled; it finishes or fails on its own, and only then is its scope disposed.

Configuration

Env var Default Description
AGENT_DOWN_THRESHOLD_SECONDS 600 Seconds without a heartbeat before agent is considered down
AGENT_CRASHLOOP_WINDOW_SECONDS 1800 Window (s) over which AGENT_STARTED restarts are counted for the crashloop signal
AGENT_CRASHLOOP_MIN_RESTARTS 3 Restarts within the window that flag a crashloop (cleanly separates a 5-min-backoff loop, ~5–6 starts, from ≤ 2 deploy rolls)
AGENT_SCALE_TO_ZERO_IDS the 8 KEDA min-0 ids above Comma-separated agent ids whose stale heartbeat does not page (rig-conductor#2376). Replaces the default list. none makes every agent page on a stale heartbeat again.
DISCORD_ADMIN_WEBHOOK_URL — Discord webhook for #admin alerts

State management

In-process ConcurrentDictionary<string, DateTimeOffset> tracks which agents are currently down. A pod restart resets state; the watchdog re-detects any still-stale agents on the first scan after restart (≤ 60 s), re-posting Discord and re-emitting AGENT_DOWN. This is acceptable for an alerting loop.

A crashloop clears only after AGENT_CRASHLOOP_WINDOW_SECONDS passes with no further restart, so the AGENT_RECOVERED for a crashloop can lag the actual recovery by up to one window (~30 min by default). This is expected — the window is precisely what distinguishes a stopped loop from a momentary gap. The RecentStartsAt history is kept on the AgentStatus projection (inline Apply(AgentStarted), last 10 starts) so the signal survives without a projection rebuild.

Multi-tenancy (PR-4)

AgentHeartbeatWatchdog runs its scan once per active tenant via ITenantWorkRunner.ForEachTenantAsync — each tenant in its own DI scope with ITenantContext.TenantId set, so the IAgentQuery read and the AGENT_DOWN/AGENT_RECOVERED IEventStore writes bind to that tenant's database. Single-tenant today ⟹ one pass for invotek (behaviour-neutral). The _downSince down-transition registry stays process-global — agent pods are shared cluster resources, and the dedupe must fire AGENT_DOWN exactly once per agent.

Since rig-conductor#2396 only the pass for FleetLivenessPolicy.LivenessTenant (the default tenant, where the repo-less heartbeats land) evaluates agents; every other tenant's pass returns at once. A second tenant's heartbeat-less row for the same agent had made the two passes flap AGENT_DOWN/AGENT_RECOVERED every tick. See fleet-wide agent liveness, Slice C.

Implementation files

  • src/ConductorE.Api/Services/AgentHeartbeatWatchdog.cs — service + ComputeHeartbeatStatus + crashloop alert
  • src/ConductorE.Core/Domain/AgentCrashLoopPolicy.cs — pure restart-rate policy (rc#1763)
  • src/ConductorE.Core/Domain/AgentDownAlertPolicy.cs — pure "who counts as down" policy and the scale-to-zero default set (rc#2376)
  • src/ConductorE.Core/Domain/Events.cs — AgentDown (with Reason), AgentRecovered records
  • src/ConductorE.Api/Adapters/MartenProjections.cs — AgentStatus.RecentStartsAt via inline Apply(AgentStarted)
  • src/ConductorE.Api/Program.cs — service registration + HeartbeatStatus field on /api/agents
  • src/ConductorE.Api/Services/ConductorSelfHeartbeatService.cs — the conductor's own HEARTBEAT (rc#2356); tests in tests/ConductorE.Api.Tests/ConductorSelfHeartbeatServiceTests.cs
  • tests/ConductorE.Api.Tests/AgentHeartbeatWatchdogTests.cs + tests/ConductorE.Core.Tests/AgentCrashLoopPolicyTests.cs + tests/ConductorE.Core.Tests/Domain/AgentDownAlertPolicyTests.cs — unit tests