RunPod pod control

Lets the rig turn the on-demand self-hosted-AI pod on and off, so control lives in one place (Rig Cockpit today, the Discord bot later) instead of every surface calling RunPod directly. Non-confidential fast-lane of the self-hosted-AI epic (dashecorp/infra#426, follows #428; tracked by #434).

Surface

Since rig-conductor#2034 (= #2008 part 2c) the three routes live in src/ConductorE.Api/Endpoints/RunPodPodEndpoints.cs and run on the REST v2 client + RunPodWakeService. Every handler is an internal static method unit-tested against the scripted FakeRunPodV2Client (tests/ConductorE.Api.Tests/RunPodPodEndpointsTests.cs).

Endpoint	Effect	Status codes	Response
POST /api/pods/{name}/start	Wake: RunPodWakeService.WakeAsync — ensure a pod attached to RUNPOD_VOLUME_ID is RUNNING on any acceptable GPU (start the stopped one, or create a replacement and terminate the stale one). The wake runs on the application lifetime, not the request: a wake may take minutes while Cloudflare Access closes the origin connection at 100 s (524), so a disconnect never aborts a half-done start/create — the caller's retry joins the same in-flight run (singleton gate) and GET shows the progress meanwhile	200 AlreadyRunning / Started / Created · 202 Waiting (start/create in flight — poll again) · 409 problem Failed (nothing in flight; detail = why) · 503 unconfigured · 404 unknown name	{ name, outcome, podId, gpuId, pricePerHour, proxyUrl, message } — gpuId/pricePerHour only for a pod this wake created; proxyUrl = https://{podId}-{RUNPOD_POD_PORT}.proxy.runpod.net
POST /api/pods/{name}/stop	Stop every pod on the volume (a stale pod and its replacement may both be billing after a Waiting wake)	200 (even if a pod refused — see per-pod ok) · 404 no pod on the volume · 503 volume lookup unreadable (nothing stopped) or unconfigured	{ name, ok, pods: [{ podId, ok }] } — ok is the conjunction
GET /api/pods/{name}	Lookup + ObservePodAsync of every pod on the volume; reports the serving pod (RUNNING with a GPU), else the in-flight one, else the first (RunPodVolumePods.Rank — GET /v2/pods order is unspecified, so a stale Exited pod may be listed before its replacement). This is the field Rig-E reads its inference endpoint from (#2008 part 3).	200 always (also unconfigured / blip) · 404 unknown name	{ name, configured, status, isRunning, podId, proxyUrl, podsOnVolume, datacenter, lookup, readySince } — status is the v2 enum upper-cased (PROVISIONING STARTING RUNNING EXITED ERROR TERMINATED) or UNKNOWN; isRunning = RUNNING with ≥ 1 GPU attached (the only state that serves — proxyUrl is set whenever a pod is named, so check isRunning too); lookup ∈ FOUND / NOT_FOUND / UNREADABLE / UNCONFIGURED — an UNKNOWN status on UNREADABLE is a control-plane blip, not a verdict; readySince (#2052) is when the idle monitor first saw vLLM answer /metrics since the pod came up — null while booting or down, so isRunning:true, readySince:null means "waking, not usable yet"

{name} is the friendly pod name (RUNPOD_POD_NAME, default qwen, case-insensitive) and never a RunPod id — the pod is always resolved through our volume; an unknown name → 404 on every route, before any RunPod call. Unconfigured (RUNPOD_API_KEY unset or RUNPOD_VOLUME_ID blank) → 503 on start/stop, 200 configured:false on GET, no RunPod call. Rig Cockpit reaches these through rig-cockpit-worker, which proxies /api/* generically — no worker change needed. Auth is the same posture as the rest of /api/* (Cloudflare Access at the edge + the worker's operator-session check); since rc#2369 start/stop carry the operator-token gate (X-Rig-Operator-Token; start also accepts the wake-only RIG_WAKE_TOKEN that Rig-E holds; enforced in production since 2026-09-25, rc#2435; see 2026-09-24-operator-route-gate.md), which replaces the X-Operator-Discord-Id idea as the next hardening step for these cost-incurring writes.

Idle auto-stop

RunPodIdleMonitor (a BackgroundService, mirrors ReconciliationService) every CHECK_SECONDS resolves the pods on RUNPOD_VOLUME_ID (IRunPodV2Client.LookupPodByVolumeAsync), scrapes each one's vLLM /metrics (IRunPodV2Client.GetActivityAsync — https://{podId}-{RUNPOD_POD_PORT}.proxy.runpod.net/metrics, a browser-like User-Agent because Cloudflare fronts the proxy and 403s default UAs, no API key: the proxy is not the API) and applies the pure RunPodIdlePolicy.Advance phase machine to the aggregate: once the pod is ready (first successful scrape since it came up), after IDLE_MINUTES of consecutive idle observations — zero running and waiting requests and no advance in the monotonic work counter since the previous scrape (#2075) — it calls StopAsync(podId) on every pod observed up; a pod that is not ready within RUNPOD_BOOT_GRACE_MINUTES is stopped as a dead model. Confirmed Provisioning/Starting also enters this budget (#2101); it need not first reach Running. Living in the always-on conductor, it replaces infra/runpod/idle-stop.sh.

Every pod on the volume is classified (RunPodVolumePods, shared with GET). A stale pod and its replacement coexist after a Waiting wake or a refused terminate, and GET /v2/pods order is unspecified — evaluating only the first pod would, with the dead one listed first, read NotRunning every tick, reset the streak, disarm the cap and never scrape the pod that is billing. Per pod: /metrics → Busy/Idle, else control plane → UnreachableButRunning/Booting/NotRunning/Unknown. Aggregate priority Busy > Idle/UnreachableButRunning > Booting > Unknown > NotRunning, named by the first pod that produced it — so a Busy pod anywhere wins and a sibling's Idle never stops it. An idle stop stops every pod observed up (the aggregate being Idle means none is busy); the runtime cap arms when any pod is up and stops every up pod. With more than one pod it logs a Warning per tick.

Fail closed on cost (rig-conductor#1998)

Each tick is classified as a PodSignal before the policy runs:

Signal	When (v2 terms)	Effect
Busy	any pod's /metrics shows running/waiting requests, or its monotonic work counter advanced since the previous scrape (#2075)	reset the idle streak
Idle	a pod's /metrics shows zero running/waiting and no counter advance (and no pod is Busy)	accrue toward a stop
UnreachableButRunning	a pod's /metrics unreachable but ObservePodAsync says Running (and no pod is Busy)	accrue toward a stop
Booting	a pod's /metrics unreachable and the control plane positively reports Provisioning or Starting	accrue boot grace, arm the cap and retain the pod as a stop target
NotRunning	volume lookup NotFound (no pod attached), or every pod's observe says Exited / Error / Terminated	reset (genuinely down)
Unknown	volume lookup Unreadable; or no pod is up and some pod's /metrics is unreachable and its observe is Unreadable / NotFound / Unknown	hold the streak (neither accrue nor reset)

Why the counter, not just the gauge (#2075)

vllm:num_requests_running + vllm:num_requests_waiting are instantaneous gauges, and the monitor samples them once per CHECK_SECONDS (60 s by default). A Rig-E chat turn occupies vLLM for a few seconds, so a sample lands inside one roughly 13% of the time. An agent being actively used therefore read Idle on nearly every tick, the streak never reset, and the pod was stopped a fixed IDLE_MINUTES after becoming ready no matter how much it was used — mid-conversation. It was an idle timeout in name and a hard session cap in behaviour.

Measured 2026-09-11: readySince 08:55:22Z, a turn answered at 09:01:51Z, pod stopped ~09:10:22Z — exactly 15 minutes after ready, with the answered turn moving the deadline by zero seconds.

So GetActivityAsync returns a PodActivity(Active, Completed) from ONE scrape, where Completed sums the monotonic families in RunPodV2Client.WorkCounterPrefixes (vllm:request_success_total, vllm:generation_tokens_total, vllm:prompt_tokens_total) across every label series. RunPodIdlePolicy.ClassifyActivity compares it with the previous scrape:

• a request in flight now ⟹ Busy (a long generation must never read idle)
• the counter advanced since the last scrape ⟹ Busy — this is what makes a turn that began and ended between two ticks visible
• the counter went backwards ⟹ Busy: the process restarted, so the baseline is meaningless and this tick carries no idle evidence
• a counter but no baseline yet ⟹ Busy. This is the first scrape for a pod. Work done before it is already inside the counter with nothing to measure it against, so the tick seeds the baseline and must not ALSO bank idle evidence against it — that is what makes the Booting -> Ready edge enter Ready with IdleStreak = 0 rather than 1.
• no counter at all in the exposition ⟹ the gauge alone, i.e. exactly the pre-#2075 behaviour. An older vLLM exposing none of the families must still be stoppable, so this case stays Idle.

Only the direction of the sum is used, never its magnitude, which is why mixing counter families is safe — and necessary, since the pod runs the floating vllm/vllm-openai:latest tag and series names move between releases. Completed is null, never 0, when no counter is found: a constant zero across ticks would read as idle and re-create the bug.

Baseline lifetime

The per-pod baseline is kept across an unreachable /metrics tick and dropped when the process behind the id is known to have ended — a confirmed-down/startup observation, or a stop the control plane accepted. Both halves matter and they pull in opposite directions:

• Keeping it across a blip is required because a baseline tick is non-idle. If a blip dropped it, every tick after one would be a fresh seed, so a pod whose /metrics flaps would alternate accrue/reset, never reach the streak, and bill indefinitely.
• Dropping it on a down edge is required because RunPod reuses a pod id across stop/start. A pod that comes back is the same id running a new process with counters from zero. Comparing its first scrape against the dead process's totals is wrong either way, and when the two coincide — 0 vs 0 is the common case — it reads Idle, so Ready is entered with IdleStreak = 1 and the pod is stopped an interval early. That is the #2075 bug re-entering through the lifecycle.

Confirmed startup also clears that pod's published readiness, even when a busy sibling wins the aggregate. Entries for pods that have left the volume are pruned each tick, so replacements cannot grow the map.

The UnreachableButRunning row is the fix. Previously an unreachable /metrics reset the streak, so a running pod with a dead vLLM billed forever — the exact failure the monitor exists to prevent. It now falls back to the RunPod control plane: a pod that is RUNNING but not serving is billed, so it accrues toward a stop like idle. A failed volume list is Unknown, never "nothing to stop" — the same tri-state rule the wake path uses.

Independently, a wall-clock cap (MAX_RUNTIME_MINUTES, default 6 h) stops any pod that has been continuously up longer than the cap, regardless of what the streak can see — the backstop for a wedged vLLM reporting a phantom busy request, or a run of Unknown ticks. The uptime clock is in-process, so a conductor restart re-arms it (acceptable for a backstop). Set MAX_RUNTIME_MINUTES=0 to disable.

Idle starts at model-ready — boot grace (rig-conductor#2052)

RUNNING is not usable: vLLM loads the model from the network volume for minutes first. The first real wake (2026-09-09):

event	UTC
POST /api/pods/qwen/start	09:37:51
pod RUNNING with GPU; monitor sees /metrics 502 → UnreachableButRunning	09:40:41
vLLM /health 200 (model loaded) — ready	09:46:57
idle stop (idle 00:15:00)	09:55:40

Second measured wake (2026-09-12) — 3 minutes, on an A100

The 2026-09-09 trace above is one data point on one GPU, and it is the source of the hardcoded READY_BY_MIN = 9 the chat quotes. A second measured wake, after the price-cap fix below, was less than half that:

event	UTC
POST /api/pods/qwen/start → 201 Created, NVIDIA A100 80GB PCIe @ 1.59 USD/h	23:15:47
readySince (model loaded and serving)	23:18:39

~2 min 52 s, wake to serving. So "about 9 minutes" is not a property of the system, it is a property of the GPU that happened to be allocated. Treat READY_BY_MIN as a stale constant, not a measurement — nothing updates it and the chat promises it unconditionally.

The flat streak counted the six minutes of model load as idle, so the pod was usable for 8.7 of the ~16 billed minutes — for wake-on-chat the difference between "wake, answer, stay up for a conversation" and "wake, answer once, stop". The monitor now keeps a RunPodIdleState(Phase, IdleStreak, BootingTicks) per tick:

Phase	Enter	Per tick	Stop
Down	NotRunning (from any phase), or a stop RunPod accepted — both counters cleared, readySince cleared	—	never
Booting	positive Booting from any phase, or UnreachableButRunning while not ready	BootingTicks+1 across Provisioning, Starting and Running without metrics	StopBootGrace once BootingTicks ≥ ceil(RUNPOD_BOOT_GRACE_MINUTES / CHECK_SECONDS) — the #2041 dead-model guard; 0 disables it, and then only the wall-clock cap stops a pod that never comes up
Ready	first Busy/Idle since up — logged once: [RunPodIdleMonitor] pod {id} ready after {elapsed} (boot grace {grace}); elapsed includes positively observed allocation/startup since #2101; publishes readySince	Busy → streak 0; Idle or UnreachableButRunning (a vLLM that died after serving, #1998) → streak+1	StopIdle once IdleStreak ≥ ceil(IDLE_MINUTES / CHECK_SECONDS) — exactly the pre-#2052 window, just started at ready. A successful stop returns to Down (RunPodIdlePolicy.AfterStop)

Unknown never advances anything (a blip neither boots, idles nor readies). The ready tick itself counts as the first idle observation when the scrape is Idle, so with the defaults a woken pod that nobody talks to stops on the 15th consecutive idle scrape counting the ready scrape (≈ 14 minutes after the ready tick at CHECK_SECONDS=60) (≈ 21 minutes after RUNNING on the timeline above), and the boot grace (20 min ≈ 2× the measured 9-min wake→ready; RUNNING→ready alone was ~6 min) is what stops a pod whose vLLM never answers. A stop RunPod accepted is the Down edge (idle, boot-grace and cap alike): a re-wake inside the next tick — wake-on-chat, or the cockpit stop→start flow — starts a new boot and a new readySince instead of inheriting the stale Ready phase (which would bill the new model load as idle, the #2052 bug re-entering). A refused stop resets only the counter that fired and keeps the phase, so a full window elapses before the next attempt and "ready" is logged once per boot. The 6 h cap is unchanged and runs from the first up observation, phase-independent. Readiness lives in the in-process IPodReadinessRegistry singleton (written by the monitor, read by GET); a conductor restart forgets it until the next tick re-observes the pod.

2101 closes only the startup-accounting hole from #2091: a positively observed new startup after Ready

clears stale readiness and starts a fresh boot budget; mixed startup phases share that budget and Unknown pauses it. The cap starts on the first positive running/startup observation and Unknown never arms or resets it. These remain per-volume, in-process guards, not independent per-pod timers or a guaranteed spending ceiling. Busy siblings still defeat boot-grace stops; the independent cap can stop the whole volume.

Wake (#2008 part 1a): GPU selection

Verified 2026-09-08: POST /v2/pods places one gpu.id and never falls back, start cannot move a stopped pod off its machine, and the catalog has no datacenter filter. The wake path is therefore driven by pure Core policies (src/ConductorE.Core/UseCases/), tests first; the HTTP adapter (RunPodV2Client, part 2a), the wake service (part 2b) and the endpoints (part 2c) are described below.

Policy

Decides

GpuSelectionPolicy.Select(catalog, criteria)

Ranked GpuCandidate list from GET /v2/catalog/gpus?include=AVAILABILITY&product=POD, filtered client-side on dataCenters[].id == the volume's datacenter (EU-RO-1). Eligible = capacity ≠ NONE in our datacenter; allowlisted when an allowlist is set (the allowlist is authoritative for capability and replaces the VRAM guard — an allowlisted 24 GB card is eligible); otherwise ≥ MinVramGb (32) only when VRAM is known (unknown VRAM passes — the catalog may carry no VRAM, so the default deployment ranks on price alone); and always a known price.secure ≤ MaxPricePerHour (1.00). Order: cheapest, then higher datacenter availability, then id. Every rejected GPU carries a GpuExclusionReason for logging. Set RUNPOD_GPU_ALLOWLIST when a cheap-but-too-small card must be kept out.

Config (deploy/k8s/deployment.yaml)

All read once at startup: the v2 client (RUNPOD_API_KEY, RUNPOD_V2_BASE_URL, RUNPOD_POD_PORT), RunPodWakeOptions.FromEnvironment() (everything else RunPod-prefixed) and the monitor timings.

Env	Default	Used by	Meaning
RUNPOD_API_KEY	unset ⇒ unconfigured	v2 client	Restricted RunPod key, via secretKeyRef rig-conductor-secrets/runpod-api-key (optional: true). Only ever placed in the Authorization header of api.runpod.io/v2 calls — never logged, never sent to the pod proxy.
RUNPOD_VOLUME_ID	bjn7d94ej9 (set)	wake, stop, GET, monitor	The qwen-weights network volume — the only thing that identifies "our" pod (GET /v2/pods matched on mounts.network[].volumeId, every pod on it). Blank ⇒ unconfigured.
RUNPOD_TEMPLATE_ID	cnz5fxyadi (set)	wake	Template supplying image/env for POST /v2/pods when a replacement is created. Blank ⇒ the wake refuses to create (Failed).
RUNPOD_DATACENTER_ID	EU-RO-1 (set explicitly)	wake, GET datacenter	The volume's datacenter — a pod can only be created there.
RUNPOD_POD_NAME	qwen	routes	The friendly {name} the three routes are keyed on.
RUNPOD_POD_PORT	8000	v2 client, wake, GET	vLLM port in https://{podId}-{port}.proxy.runpod.net (metrics scrape + proxyUrl).
RUNPOD_MIN_VRAM_GB	32	wake	GPU capability floor (only enforced when the catalog carries VRAM).
RUNPOD_MAX_PRICE_PER_HOUR	1.00 (code) — deployed: 2.00	wake	Secure-cloud USD/h cap, inclusive; dot decimal only (1,50 is rejected, not read as 150). This is the wake's single point of failure — see the incident below. Do not lower it to the code default without reading that first.
RUNPOD_GPU_ALLOWLIST	unset	wake	Optional comma-separated GPU ids; authoritative for capability when set.
RUNPOD_V2_BASE_URL	https://api.runpod.io/v2	v2 client	Override for tests / a mock.
IDLE_MINUTES / CHECK_SECONDS	15 / 60	monitor	Idle threshold and poll interval (RequiredIdleStreak = ceil).
MAX_RUNTIME_MINUTES	360 (set)	monitor	Wall-clock cap; 0 disables.
RUNPOD_BOOT_GRACE_MINUTES	20 (set)	monitor	Stop an observed starting/running pod not ready within this (RequiredBootGraceTicks = ceil); 0 disables — an unready pod is then stopped only by the cap.

An unparseable or out-of-range guard value never widens the guard: the default applies and one line lands in RunPodWakeOptions.Warnings, which Program.cs logs at Warning right after Build() ([RunPodWake] options: ...). The runpod-api-key value itself is SOPS/gitops-managed in the infra repo (not this repo).

Disabled-until-configured / fail-open: when RUNPOD_API_KEY is unset or RUNPOD_VOLUME_ID is blank, the v2 client no-ops (reads Unreadable / [] / null, writes false), start/stop answer 503, GET answers configured:false, and the idle monitor exits at startup — so the conductor boots safely before the secret is provisioned.

v1 retired (#2008 parts 2c and 2d)

The REST v1 RunPodClient (IRunPodClient, rest.runpod.io/v1) was unregistered from DI in part 2c (#2041) and its file and tests were deleted in part 2d (rig-conductor#2040 / PR #2043), completing #2007; the endpoints, the wake service and the idle monitor all run on IRunPodV2Client. RUNPOD_POD_ID is no longer read anywhere: there is no explicit-id pin on v2, the volume is the only identity (a pinned id is exactly the stale-pod failure mode #1998 removed). The v1 /metrics parser (ParseActiveRequests) was ported verbatim to RunPodV2Client.

History — volume-anchored resolution on v1 (rig-conductor#1998, #2013, #2014, #2026): a stopped pod keeps its network volume but is recreated with a new id, so a hardcoded RUNPOD_POD_ID went stale (the deployment once pinned a pod terminated in August). v1 resolved the pod with GET /v1/pods?networkVolumeId=<id>&includeNetworkVolume=true (v1 omits the networkVolume object without the include flag — #2013) and cached the id; production briefly ran with RUNPOD_POD_ID=bsyo7ucc9d6nkm (#2011/#2012) for the first unattended idle stop, and the pin was removed in #2026. On v2 there is no include flag to forget: GET /v2/pods always embeds mounts.

Wake (#2008 part 1c): wake decision + termination guard

Pure Core (src/ConductorE.Core/UseCases/WakePolicy.cs), tests first. Part 1a (#2017) ranks GPUs; part 1b walks the create attempts; the HTTP adapter is RunPodV2Client (part 2a) and the orchestration is RunPodWakeService (part 2b), below.

Policy	Decides
WakePolicy.Decide(observation)	Lookup is tri-state (PodLookup.Found / NotFound / Unreadable): Unreadable→WaitAndRepoll (a failed GET/list is never "no pod"); affirmative NotFound→CreateReplacement. Found: Running+GPU→Done · Exited/Error before a start→StartExisting · Exited/Error after a start, or Running with 0 GPUs, or Terminated→CreateReplacement · Provisioning/Starting/Unknown→WaitAndRepoll (never spawn a second pod on an in-flight start or an API blip).
TerminationGuard.MayTerminate(staleId, replacementId, staleVolume, ourVolume, replacementStatus)	The single public entry point. The stale pod is terminated only when both ids are non-blank and differ (never the replacement itself — both sit on our volume), its volume is exactly ours, and the replacement is RUNNING.

Hard requirement for PR 2 (the wake adapter): deriving Starting / Provisioning

Superseded by the REST v2 client (part 2a): the paragraph below is written in v1 terms (desiredStatus, no STARTING value). On v2, Pod.status is a real enum (PROVISIONING | STARTING | RUNNING | EXITED | ERROR | TERMINATED) and runtime is null unless RUNNING; RunPodV2Client.ObservePodAsync derives Running only from RUNNING and runtime.uptime > 0, else Starting; GpuCount = runtime.gpus[].Length, falling back to gpu.count only when the array is absent. The invariants (failed lookup ⇒ Unreadable, never NotFound; Running with 0 GPUs ⇒ replace) are unchanged. See the "REST v2 client" section.

RunPod's desiredStatus is only RUNNING / EXITED / TERMINATED (RunPodClient.cs reads it as UNKNOWN when absent) — there is no STARTING/PROVISIONING value. Right after start (or create) it flips to RUNNING while the GPU is still attaching; a naive mapping would read Running with 0 GPUs and WakePolicy would create a replacement — two pods billed. The adapter must map desiredStatus == RUNNING with the runtime block absent (or uptimeInSeconds missing/0) to PodStatus.Starting (Provisioning for a pod it just created), and only RUNNING with a present runtime block to PodStatus.Running. GpuCount is read from the runtime block and is only meaningful in Running. Likewise a failed/non-2xx pod lookup must map to PodLookup.Unreadable, never to NotFound.

REST v2 client (#2008 part 2a, rig-conductor#2030)

src/ConductorE.Api/Services/RunPodV2Client.cs (IRunPodV2Client) is the outbound adapter part 2b's wake loop drives. It talks to RunPod REST v2 (https://api.runpod.io/v2, override with RUNPOD_V2_BASE_URL; bearer = RUNPOD_API_KEY) over the existing named "runpod" HttpClient, and does transport + JSON shaping only — every decision is a Core policy. The v1 RunPodClient is retired from the hot path (see "v1 retired" above); v2 is the only API that exposes PROVISIONING/STARTING, runtime.gpus[] and mounts.network[], which the wake contract needs. Every call is pinned to its exact method + path + query string in tests/ConductorE.Api.Tests/RunPodV2ClientTests.cs (the v1 include-flag bug, #2013, slipped because a test faked the response shape).

Method	Request (verified against the v2 OpenAPI spec)	Returns
GetCatalogAsync	GET /v2/catalog/gpus?include=AVAILABILITY&product=POD (product is required with the include)	GpuCatalogEntry[] from gpus[]: id, price.secure, availability (NONE\|LOW\|MEDIUM\|HIGH, absent → None), dataCenters[].{id,availability} (absent → empty), memory → VramGb. Empty on any failure — no candidates means nothing is billed.
ObservePodAsync(podId)	GET /v2/pods/{id}	PodObservation (StartAttempted always false — caller's concern). status maps 1:1 (PROVISIONING/STARTING/EXITED/ERROR/TERMINATED; unknown value → Unknown). RUNNING is Running only when runtime is an object with uptime > 0; runtime: null, {} or uptime 0 → Starting. GpuCount = runtime.gpus[] length (GPUs actually attached; [] → 0 → WakePolicy replaces), falling back to gpu.count only when the array is absent. NotFound only for a 404 carrying RunPod's problem-details body (spec NotFoundError: application/problem+json {title, status:404, detail}, meaning "not found or not accessible to the caller"); a bare 404 (wrong RUNPOD_V2_BASE_URL without /v2, a proxy page, non-JSON) and every other non-2xx, transport exception or timeout → Unreadable, never NotFound. A 2xx with an unparseable body is Found+Unknown. 2b invariant: an observe-404 on a remembered pod id is never a create trigger by itself — WakePolicy.Decide(NotFound) is CreateReplacement, so 2b must first re-confirm absence with LookupPodByVolumeAsync(...).Lookup == NotFound (the only affirmative "no pod is attached to our volume") before POST /v2/pods; otherwise a permission-scoped key or a misrouted 404 bills a second pod next to a live one.
LookupPodByVolumeAsync(volumeId)	GET /v2/pods (v2 has no volume filter — only includeClusterPods — and always embeds mounts, so there is no include flag to forget)	PodVolumeLookup(Lookup, PodIds): every pod whose mounts.network[].volumeId matches, in list order (each id once) → Found; listed with no match → NotFound (empty list); non-2xx / exception / bad JSON → Unreadable (empty list). More than one match logs a Warning — a stale pod and its replacement coexist after a Waiting wake, and the wake service needs all of them (a first-match lookup would re-drive the stale one and create a third pod, or never terminate it). This is the only affirmative absence — the gate before any create. There is deliberately no string? convenience: a collapsed null would read a failed list as "no pod".
CreatePodAsync(PodCreateRequest)	POST /v2/pods with {name, templateId, cloud:"SECURE", dataCenterIds:[dc], gpu:{id, count:1}, mounts:{network:[{volumeId, path:"/workspace"}]}} (no image — the template supplies it)	PodCreateResult(HttpStatus, PodId, RetryAfter): the raw status is what PodCreateOutcomePolicy.Classify consumes (201/400/402/403/422/429/503 pass through untouched); PodId from id on 2xx; RetryAfter from the 429 header. Non-2xx logs the problem-details detail. A transport failure returns status 0 (→ Abort): a create that timed out may still have placed a pod, so it is never retried blind.
StartAsync / StopAsync(podId)	POST /v2/pods/{id}/action body {"action":"start"} / {"action":"stop"}	true on 2xx
TerminateAsync(podId)	DELETE /v2/pods/{id} (spec: 204, 409 when the pod belongs to a cluster)	true on 2xx
GetActiveRequestsAsync(podId) (part 2c)	GET https://{podId}-{RUNPOD_POD_PORT}.proxy.runpod.net/metrics — the pod's public vLLM proxy, not the API: browser-like User-Agent (Cloudflare 403s default UAs), no bearer	vllm:num_requests_running + vllm:num_requests_waiting (ported v1 parser); null on non-2xx (Cloudflare 404 while stopped / not routing), transport failure, or unconfigured

IsConfigured is false while RUNPOD_API_KEY is unset (the safe boot state before the SOPS secret is mounted — provisioned as runpod-api-key in rig-gitops apps/rig-conductor/rig-conductor-secrets.sops.yaml); every call then no-ops without a network request — catalog [], observe/lookup Unreadable, create status 0, actions false — instead of a 401 warning per wake tick. Errors never escape (except caller cancellation), pod ids are path-escaped, and the API key is only ever placed in the Authorization header — it is never logged. Part 2b (below) drives it from RunPodWakeService; part 2c (rig-conductor#2034) registered both in DI and rewired the endpoints + monitor onto it.

Wake service (#2008 part 2b, rig-conductor#2033)

src/ConductorE.Api/Services/RunPodWakeService.cs — WakeAsync(ct) → WakeResult is the orchestration that turns the Core policies plus IRunPodV2Client into "ensure a pod attached to our volume is RUNNING, on any acceptable GPU". It holds no decision logic of its own: every branch is WakePolicy / GpuSelectionPolicy / PodCreateOutcomePolicy / TerminationGuard, and every RunPod call is the v2 client. Tests first: tests/ConductorE.Api.Tests/RunPodWakeServiceTests.cs scripts a hand-written FakeRunPodV2Client (tests/.../Fakes/) per scenario and asserts the exact call sequence, so a billable call (start / create / terminate) cannot appear on a path that must not issue one. Wired in part 2c (rig-conductor#2034): registered as a singleton (so the concurrency gate below covers every caller) and driven by POST /api/pods/{name}/start (see "Surface").

Step	What happens	Fail-closed rule
0 Gate	A WakeAsync arriving while another is in flight joins it (Task.WaitAsync(ct) on the running task) and gets the same WakeResult	Two overlapping wakes (an endpoint retry after a Cloudflare 524, a Cockpit double-tap) would both read "no pod" and both POST /v2/pods. The joiner touches the client not at all.
1 Lookup	LookupPodByVolumeAsync(RUNPOD_VOLUME_ID) → every pod on the volume	Unreadable ⇒ Failed, zero writes. Never create on a failed list. NotFound ⇒ step 3 with no stale pod.
2 Existing pods	Per round, ObservePodAsync each pod on the volume → WakePolicy.Decide (with per-pod StartAttempted carried by the service). Precedence: a Done pod wins (AlreadyRunning / Started) and every sibling is stale → step 4; any WaitAndRepoll ⇒ repoll; else one StartExisting ⇒ StartAsync that pod, repoll; else every pod says CreateReplacement ⇒ step 3 with all of them as stale	Bounded by MaxPolls rounds ⇒ Waiting. Never start or create while any pod on the volume is in flight — that is how a Waiting wake followed by a re-wake used to stack a third pod. An observe-NotFound on a remembered id is re-confirmed with the volume lookup first (2a invariant): Unreadable ⇒ Failed, Found ⇒ drive that list instead, NotFound ⇒ create.
3 Replacement	GetCatalogAsync → GpuSelectionPolicy.Select(catalog, criteria from options); walk PodCreateOutcomePolicy.Start/Classify/Advance, one CreatePodAsync per attempt (name = PodNamePrefix + UTC timestamp, gpu.id, template, datacenter, volume); on Created poll the new pod until WakePolicy says Done	Zero candidates ⇒ Failed listing every GpuExclusionReason. Any 2xx with a pod id is Created (the pod exists and bills whatever the exact status — an off-spec 200/202 must not orphan it); a 2xx with no pod id ⇒ Failed("created without id") + Warning. RetrySameCandidate (429 / 5xx, asked of the Core policy, not re-derived) waits min(Retry-After ?? PollInterval, MaxRetryAfter) through the injectable delay, then re-reads the volume before re-posting: a pod there that is not one of the stale ids is adopted as the replacement (a gateway 5xx after RunPod committed the create must not bill a second pod); Unreadable ⇒ Failed (step-1 invariant); NotFound ⇒ retry. Exhausted / RetriesExhausted / Aborted ⇒ Failed(phase). New pod: Done ⇒ step 4; WaitAndRepoll (or Unreadable / not-yet-listed) ⇒ repoll, MaxPolls ⇒ Waiting with the stale pods left intact; any other verdict (Terminated / Exited / Error / RUNNING with 0 GPUs) ⇒ Failed("… not retrying") — nothing is in flight, so it is not Waiting, and no second create walk.
4 Terminate	For each stale pod: TerminationGuard.MayTerminate(stale, serving, ourVolume, ourVolume, servingStatus) ⇒ TerminateAsync(stale)	Only after a sibling / the replacement is RUNNING; refused per pod (kept, Warning logged) when the ids match or the serving pod is not RUNNING. Every pod was resolved through our volume, so the stale volume is ours by construction.

WakeResult(Outcome, PodId, GpuId, PricePerHour, ProxyUrl, Message) with Outcome ∈ {AlreadyRunning, Started, Created, Waiting, Failed} and ProxyUrl = https://{podId}-{RUNPOD_POD_PORT}.proxy.runpod.net; GpuId/PricePerHour are known only for pods this wake created. Every billable action (start, create, adopt, terminate) is logged at Information with pod id, GPU id and price — never the key. Waiting means "something is in flight, poll again"; nothing was terminated. Failed means nothing is in flight either — re-polling will not change it.

Options — RunPodWakeOptions.FromEnvironment() (same directory, read once): RUNPOD_VOLUME_ID, RUNPOD_TEMPLATE_ID (blank ⇒ the wake refuses to create), RUNPOD_DATACENTER_ID (EU-RO-1), RUNPOD_MIN_VRAM_GB (32), RUNPOD_MAX_PRICE_PER_HOUR (1.00), RUNPOD_GPU_ALLOWLIST (comma-separated, case-insensitive, blank ⇒ none), RUNPOD_POD_PORT (8000); code-only bounds MaxPolls (30), PollInterval (10 s), MaxRetryAfter (30 s). Every loop is bounded: observe rounds by MaxPolls per phase, create calls by candidates + 2 (the policy's global retry budget). Invalid values never widen a guard: an unparseable or out-of-range number keeps its default and adds one line to Options.Warnings (RUNPOD_MIN_VRAM_GB="many" is not a non-negative int; using 32) for part 2c to log at Warning on startup. The price cap parses with AllowDecimalPoint only — no thousands separator — so a European-locale "1,50" is rejected (default 1.00 + warning) instead of being read as 150 and admitting every GPU.

Wake (#2008 part 1b): create-attempt outcome walk

Policy	Decides
PodCreateOutcomePolicy.Classify(status)	201→Created · 400/403→TryNextCandidate · 402/422→Abort · 429/5xx→RetrySameCandidate (max 2 as ONE budget across the whole walk — never refilled on a candidate change; retry exhaustion is terminal → RetriesExhausted, not a fall-through — 429 is account-level and 5xx is an outage, neither is per-GPU) · anything else→Abort. Advance/Replay walk the candidate list; total calls ≤ candidates + 2.

Billable-action audit trail (#2050)

On the first real wake (2026-09-09) every [RunPodWake] … billable line had rotated out of kubectl logs within minutes — the conductor was writing ~7 000 lines/min, mostly Npgsql.Command and HttpClient chatter (since turned down, see 2026-09-21-conductor-log-volume.md). Money-relevant facts must not depend on stdout retention, so every billable write the conductor issues is now persisted through IPodActionLog:

Action	Written by	trigger
Start — POST /pods/{id}/start on a stopped pod	RunPodWakeService	wake
Create — a pod created (httpStatus 2xx, gpuId, pricePerHour), or a pod adopted after a transient create failure (detail says "adopted")	RunPodWakeService	wake
Terminate — a stale sibling, after its replacement is RUNNING	RunPodWakeService	wake
Stop	RunPodIdleMonitor / POST /api/pods/{name}/stop	idle · boot-grace · runtime-cap / manual

Every entry carries ok (whether RunPod accepted the call — a refused call is recorded too) and the friendly pod name. Reads, observes and non-2xx create attempts are not billable and are not recorded.

• Storage: PodActionRecord documents in the single-tenant control store (rig_control, IControlStore), same pattern as TenantErasureRecord. The pod is a rig-level resource and the idle monitor runs with no tenant in scope, so a tenant data DB is the wrong home. Append-only, GUID identity, indexed on pod name. Read back with IPodActionLog.RecentAsync (most recent first); an HTTP read route is a follow-up PR.
• Never blocks the action: the record is written after the RunPod call and MartenPodActionLog.RecordAsync swallows failures — a DB blip logs the full entry at Error (a level nothing filters) and the wake/stop carries on. It takes no cancellation token, so an action issued during shutdown is still recorded.
• Tests: RunPodWakeServiceTests, RunPodIdleMonitorTests and RunPodPodEndpointsTests assert the recorded sequence against InMemoryPodActionLog (the default when no log is injected).
• Not the wake outcome: refused or failed wakes that bill nothing leave no row here; see Two durable records.

Last wake outcome (rig-conductor#2541)

Problem. After a failed wake, status stayed EXITED (same as "never asked") and the cause rotated out of the logs in ~6 minutes (#2050). On 2026-10-01 four failed wakes went unnoticed for 24+ hours, and the cause was unrecoverable.

Recording. RunPodWakeService records every wake it runs (a caller that joins an in-flight wake adds nothing) as two events on the stream runpod-pod!{name} in the default tenant's event store (MartenPodWakeHistory, an IPodWakeHistory):

Event	When	Carries
PodWakeRequested	before RunPod is touched	podName, attemptId, requestedAt
PodWakeCompleted	when the wake returns or throws	the above + completedAt, outcome, wakeOutcome, reason, podId, gpuId, pricePerHour, candidatesConsidered, lastProviderStatus

outcome is LastWakePolicy.Classify of how the wake ended: Succeeded (AlreadyRunning / Started / Created), Refused (a fail-closed guard declined, nothing billed — unconfigured, unreadable lookup, template unset, no eligible GPU under the price cap with every exclusion in reason), TimedOut (Waiting: not RUNNING within the poll budget), Failed (RunPod rejected or broke the attempt, or the wake threw — wakeOutcome: "Exception").

Events in Postgres survive restarts and log rotation. No new document type, so no new table or schema-fence drift. Writes are best-effort, capped at 10 s, and skip the wake token: a history outage only logs a Warning, and a wake aborted by shutdown still records its outcome.

Not yet served. Exposing the newest event as lastWake on GET /api/pods/{name} is the follow-up PR for #2541.

Two durable records: why both exist

RunPodWakeService writes to both the billable-action audit trail (#2050) and this wake history. They are not duplicates. The action log has one row per billable Start / Create / Terminate / Stop call: "what did we ask RunPod to bill us for?". The wake history has one Requested/Completed pair per wake: "what happened to the last wake, and why?". It also covers the wakes the action log never sees: refusals, non-2xx creates, and crashes. The 2026-10-01 incident left zero billable calls. Folding outcomes into PodActionRecord would break #2050's rule that every row is something RunPod may charge for. The records agree (Created wake ⇔ Create ok: true; Refused ⇔ no row; TimedOut may have a billed Start/Create). Link them by pod name and the wake's requestedAt..completedAt window.

Changelog

• 2026-10-02 — #2541 (recording): every wake is persisted as PodWakeRequested / PodWakeCompleted on runpod-pod!{name} (default tenant), so its outcome survives a restart and log rotation. Kept separate from the #2050 action log on purpose — see Two durable records.
• 2026-10-02 — #2050: billable actions persisted to rig_control (IPodActionLog). A read route (GET /api/pods/{name}/history) is a follow-up.
• 2026-09-09 — #2052: idle window starts at model-ready (first successful /metrics scrape), separate RUNPOD_BOOT_GRACE_MINUTES (20) stops a pod that never becomes ready, GET gains readySince, RunPodIdlePolicy.Advance phase machine (Down/Booting/Ready); an accepted stop is the Down edge (AfterStop) so a re-wake within one tick boots again and clears readySince.

The price cap is the single point of failure (2026-09-11 incident, rig-conductor#2093)

A wake failed, and the cockpit reported "waking, ~9 minutes" for 15 minutes afterwards. The wake had actually failed in 12 seconds:

22:01:11Z  start requested for pod iau67u7tkxw50y (status Exited) — billable
22:01:21Z  no pod on volume bjn7d94ej9 can serve — creating a replacement
22:01:23Z  create on NVIDIA RTX PRO 4500 Blackwell returned 400 → Exhausted
22:01:23Z  wake failed: ended Exhausted after 1 of 1 candidate(s) and 0 retries

Why only one candidate out of 48. The weight-cache volume lives in EU-RO-1, so the datacenter is pinned, and that alone excluded ~44 GPUs as NotInDataCenter. Two more fell below the 32 GB VRAM floor. Everything then rests on the price cap — which was not set in deployment.yaml, so the 1.00 code default applied. A live probe minutes later showed two capable cards sitting available and rejected on price alone:

AMD Instinct MI300X OAM:                    PriceAboveMaximum
NVIDIA RTX PRO 6000 Blackwell Server Ed.:   PriceAboveMaximum

For scale, RunPod lists an L40S at 1.09 USD/h. A 1.00 cap rejects nearly every card that can hold a 32B model.

After setting RUNPOD_MAX_PRICE_PER_HOUR=2.00, the very next wake returned 201 Created on an NVIDIA A100 80GB PCIe at 1.59 USD/h — squarely in the band the old cap was refusing. Capacity was never the problem.

Raising the ceiling does not raise the bill: the walk still ranks cheapest first, so the usual card wins whenever it has capacity. Exposure stays bounded by IDLE_MINUTES and MAX_RUNTIME_MINUTES.

How to diagnose a failed wake in one command

Do not ask Rig-E. The chat fires the start with a 3 s abort and never reads the response, and has no failure branch, so it says "about 9 minutes" whatever happened. The conductor reports correctly — its 409 body names every excluded GPU with a reason:

kubectl -n rig-conductor run probe --rm -i --restart=Never \
  --image=curlimages/curl:8.10.1 --timeout=420s -- \
  -s -i -X POST http://rig-conductor-api.rig-conductor.svc.cluster.local:8080/api/pods/qwen/start

200 names the GPU and price it created on. 409 is the wake failing, and detail says why.

Conductor log retention used to be minutes (#2050). The billable actions themselves no longer depend on it — every start / create / terminate / stop is persisted to rig_control (mt_doc_podactionrecord) with GPU, price and HTTP status. For the reasons behind a failed wake (excluded GPUs, the create walk), still pull the logs promptly.