Skip to content

Operator Task Release Hold and Status

Issue2104 adds two pure Core helpers with tests. MergeGate applies the release hold (below), live for dashecorp/dashe-website. The signed release route (C9c, Operator Task Routes) calls ReleaseHeldAsync, and the task status route projects its stages with them (below). MergeGate.Merge() returns a MergeOutcome (C9a, MergeGate Merge Outcome); the hold returns Held.

Release hold policy

OperatorReleaseHoldPolicy (tests) decides whether MergeGate may merge a Rig-E task PR.

  • HeldRepos is {dashecorp/dashe-website}. IsHeldRepo compares case-insensitively, because GitHub repo names are. The caller checks it first, so every other repo makes no extra call.
  • LinkedIssueNumbers(prBody, repo) returns every issue of repo that the body links with a GitHub closing keyword, once each and in order. It accepts close, closes, closed, fix, fixes, fixed, resolve, resolves and resolved in any case, an optional colon, and then #N, owner/repo#N or https://github.com/owner/repo/issues/N naming this repo. It ignores other repos, a bare #N without its own keyword, and words such as discloses.
  • The caller reads each linked issue. A body with a rig-e-task marker (OperatorTaskIssueTemplate.IsTaskIssueBody) makes the PR a task PR. Then Decide(isTaskPr, lookupFailed, headSha, approvedSha, reviewedAtHead, heldSha) returns, in this order. approvedSha is the verified approval (below), never the bare Valkey value, and heldSha is release-hold.sha:
Input Decision Reason
Not a task PR, and every lookup succeeded Allow not_task_pr
Head is not 40 lowercase hex characters Hold invalid_head
review-e-bot's latest verdict is not APPROVED at exactly the head Wait (nothing recorded) review_not_at_head
The PR's reviews could not be read Wait (fail closed) review_lookup_failed
The verified release approval and the hold both equal the head Allow; PinsHead is true approved_at_head
An issue lookup failed Hold (fail closed) lookup_failed
A task PR with no verified approval Hold awaiting_release
A task PR with an approval of the head but no hold at it (the hold expired or was deleted) Hold, recorded again not_held
A task PR with an approval of another head Hold approval_for_other_head

The approval is unforgeable without the task key (rc#2104 review). Valkey runs without auth and the Dev-E pods get VALKEY_URL, and POST /api/merge needs no auth, so a bare release-approval sha would let an agent merge a task PR and make MergeGate post an "Operator release approved" audit comment. ReleaseHeldAsync, reached only through the signed release route, therefore also writes the signed request's requestId (lowercase D form) and mac: OperatorReleaseHoldPolicy.ApprovalMac, lowercase hex HMAC-SHA256 under COCKPIT_TASK_SIGNING_KEY of UTF8("rig-cockpit-release-approval-v1\n" + lower(repo) + "\n" + pr + "\n" + sha + "\n" + requestId). MergeGate, the release retry and the status route count an approval only through VerifiedApproval, which checks that MAC in constant time and returns the approved SHA and request. A value without the MAC, a MAC copied from another PR, SHA or request, a requestId that is rewritten, deleted or not in the canonical form, and any approval while the key is missing are no approval. The audit comment follows only a pinned merge, so it too needs a verified approval, and it names that approval's request, never the task issue's marker, which anyone with write access can edit. The domain line differs from the request signature's rig-cockpit-task-v1, so no signed request doubles as an approval.

reviewedAtHead uses the status route's own rule (OperatorTaskSnapshotReading.ApprovalAtHead): review-e-bot's latest APPROVED, CHANGES_REQUESTED or DISMISSED review, APPROVED with commit_id equal to the head. Only the App's login review-e-bot[bot] counts. The plain user login review-e-bot is unregistered and anyone could claim it, so its reviews are ignored: they neither approve a head nor cancel the App's approval. MergeGate's review-approved flag is PR-wide, outlives a push and a CHANGES_REQUESTED, and POST /api/merge sets it for any caller, so it never suffices: without the review at the head, a head a model pushed after one approval would reach the operator's Approve button as soon as CI is green. Since rc#2053 the guard in Merge() applies the same rule to every PR in every repo, before this policy runs (decision). The hold keeps its own check, so its outcomes are unchanged. A Wait records no hold, so the card stays in_review; Review-E's next approval (its webhook) or the status nudge runs Merge() again.

An approval of exactly the head allows the merge even when a lookup failed. The operator approved that exact SHA through the signed release route, so the release can still go through when GitHub is flaky. PinsHead tells the caller to take the merging lock and send sha = head on the merge, so a moved head cannot merge.

MergeGate hold (C9b)

MergeGate.Merge() (source, tests) applies the policy after the guard (check runs, and since rc#2053 Review-E's approval at the head) and the PR fetch, before the merged: pre-set, so every path into Merge() is held. A repo outside HeldRepos returns before any call.

  1. GET /repos/{repo}/issues/{n} with GITHUB_TOKEN for each linked issue. ClassifyLinkedIssues takes the first issue with a marker as the task issue. A non-2xx answer or an exception is a failed lookup, and the first failed number stands in for the task issue.
  2. For a task PR or a failed lookup, HGET release-approval sha requestId mac (verified with the task key), HGET release-hold sha and GET /repos/{repo}/pulls/{pr}/reviews?per_page=100 with GITHUB_TOKEN, then Decide. A Valkey error there holds. Wait: return Deferred(reason) with nothing written.
  3. Hold: HSET release-hold sha heldAt when the held SHA changes; a re-hold of the same head (every check_run, review webhook and nudge) keeps the first heldAt, which dates approval_needed. The first time per head (SETNX release-hold-waiting), emit MERGE_GATE_WAITING [operator-release] and HUMAN_GATE_TRIGGERED(repo, taskIssue, pr, "operator-release"), with the task issue named explicitly rather than looked up in the projection. Return Held(reason); review-approved and ci-passed stay.
  4. Approved at the head: SETNX merging: or return Deferred("merging"). The squash PUT sends sha = head, so a head that moved gets 409 (Failed). A failed PUT deletes the lock. After the merge, MergeGate posts the audit comment, off the request path, once per merged SHA (SETNX release-audit:), as the repo tenant's dev-e-bot App, naming the request the verified approval names (details).
  5. Not a task PR, such as a dependabot PR: it merges as before, with no lock and no sha.

HUMAN_GATE_TRIGGERED moves the issue to waiting_for_human. The SLO enforcer never re-dispatches that state, so a held PR does not send Dev-E back after 30 minutes (B3), and StuckIssueDetector allows 24 h. A new head, such as a fix round, holds and emits again. A later hold of the same head reads the task issue's projection state and emits HUMAN_GATE_TRIGGERED again (not MERGE_GATE_WAITING) when the issue left waiting_for_human for a live state (OperatorReleaseHoldPolicy.ReassertsHumanGate). Review-E triggers MergeGate twice (the runtime's POST /api/merge and the review webhook), so the webhook's REVIEW_PASSED can land after the first hold and move the issue to ready_to_merge, where the StuckWatcher files a stuck issue after 15 min and the SLO enforcer re-dispatches after 30; the webhook's own SetConditionAndTryMerge, which follows its REVIEW_PASSED, then puts it back. A terminal state or a missing row is left alone. The event's existing side effects apply: the "escalated" emoji, a message-state change and an operator push.

Key (repo lower-cased) Fields TTL Written by
release-hold:{repo}#{pr} sha, heldAt 7 d the hold
release-approval:{repo}#{pr} sha, at, requestId, mac 7 d ReleaseHeldAsync
release-hold-waiting:{repo}#{pr}:{head} – 7 d the first hold at a head
merging:{repo}#{pr} – 60 s an approved merge
release-audit:{repo}#{pr}:{sha} – 7 d the audit comment after a pinned merge
release-nudge:{repo}#{pr} – 5 min the status route's nudge (C11b) and its release retry
  • ReleaseHeldAsync(repo, pr, headSha, requestId), for C9c: Held("not_held") unless release-hold.sha equals headSha. Then it reads the reviews again: Held("review_not_at_head") when review-e-bot's latest verdict no longer approves headSha, Failed("review_lookup_failed", status) when they cannot be read, in both cases with nothing recorded. Without a usable task key it answers Failed("approval_key_unavailable") and records nothing (the route refuses earlier with 503 not_configured). It then records release-approval with the request id and its mac, which stays even if this attempt does not merge, so the next check_run completes it. A blocked flag gives Deferred("blocked"), a Valkey error Failed("release_state_unavailable"), and otherwise it returns await Merge().
  • RetryReleaseAsync(repo, pr, headSha), for the status route's release retry (below): runs Merge() only when both release-hold.sha and the verified release-approval.sha equal headSha and there is no blocked flag, otherwise returns null. It does not rewrite the approval; the guard (check runs and the review at the head) and the pinned PUT still apply.
  • A squash PUT that throws (the request may or may not have reached GitHub) is Failed("merge_put_failed") with no status and frees merged: and merging: as a refused PUT does, so a retry can merge; if GitHub did merge, the retry's PUT is refused and nothing merges twice.
  • RecheckHeldAsync(repo, pr), for the status route's nudge (below): runs Merge() unless merge-gate:{repo}#{pr} has blocked, otherwise returns null. Since rc#2053 it no longer needs the review-approved flag: the guard drops that flag when a read shows no approval at the head, and a read that raced Review-E's approval would otherwise disable this recovery. The nudge runs only after it read that approval on GitHub, and the guard re-reads it. The hold still applies. Nothing sets blocked today (Block() has no caller), so that check is inert.
  • Not stopped (infra#456): merges outside MergeGate (the operator override, an admin gh pr merge), an edited marker, and a PR that links no issue.

Stage and timeline projector

OperatorTaskStageProjector (tests) turns one OperatorTaskStatusSnapshot into {stage, detail, terminal, timeline[]}. The snapshot holds the store state and expiry, and the Proposed time. It also holds what the status route read: the issue, the PR linking it (the newest merged one, else the newest open one, else the newest), the Valkey release-hold and release-approval hashes, the deploy.yml run on the merge commit, and the dashecorp.com probe. The GitHub parts and the probe come from the C11a readers (Operator Task Snapshot Reader). WireName gives the snake_case JSON names.

No detail line names an actor the route did not observe (rc#2265, the card says what was observed). implementing used to read "Dev-E is implementing issue #N" from the agent-ready label, which the rig applies itself before any dispatch — a specific, named, false claim that sat on the operator's card for 42 minutes on task ff18ab60 while the dev-e lane was busy elsewhere. The only implementer this route can observe is the login that opened the pull request, and an author a read did not answer is left unsaid.

Stage Condition Detail
proposed Proposed, before expiry Waiting for your confirmation
expired Proposed or Approved, at or after expiry (an Approved task can no longer start an attempt) Proposal expired, ask Rig-E again
submitting Approved before expiry, or AttemptStarted Creating the GitHub issue
unknown OutcomeUnknown Issue creation outcome unknown, not retried
queued Submitted, no agent-ready, no PR Waiting for Planner-E; Planned (tier:tN), waiting for agent-ready; No plan after 10 min; or GitHub status not read yet
implementing agent-ready label or its event, no PR; a draft PR; or a PR closed unmerged while the issue is still open <agent> claimed issue #N once IssueStatus.AgentId was observed; else Issue #N is agent-ready, no pull request yet; No pull request 10 min after agent-ready (only from the label EVENT time); or Draft pull request #N by the login that opened it
in_review Open PR with no hold or approval at its head, or with one there that review-e-bot's latest verdict does not approve <reviewer> is reviewing pull request #N once IssueStatus.ReviewAgentId was observed; else Pull request #N by the login that opened it in review
approval_needed release-hold sha equals the head, approval does not, and review-e-bot's latest verdict approves the head Approve the release of the short SHA
releasing the verified release-approval sha equals the head, review-e-bot approves it, not merged Merging the short SHA
merged Merged, nothing below yet Merged as the short merge SHA
deployed The deploy run succeeded, and no probe has read the page without the new href at or after the 15-minute limit Deployed, waiting for dashecorp.com to serve it
verified The probe read the task template's expected href after the merge (± one trailing slash), and the deploy run did not fail Live on dashecorp.com
failed Issue closed with no PR; PR closed unmerged and the issue closed; deploy run completed without success; a probe at or after 15 min past the merge read the page without the expected href the reason

After the merge a failed deploy run comes first, then a live anchor, then the 15-minute limit (rc#2253). The probe used to be primary — a live anchor was verified even if the run failed — which was defensible only while the anchor could tell before from after. Once the page may already serve the expectation, "primary" means a task whose Cloudflare deploy failed is reported live. For the same reason a failed task dates no deployed or verified timeline entry, however the page looks.

What counts as "the expected href" is the template's, derived from the NewAnchor it asks Dev-E to write, and it is the probe that judges it: OperatorTaskVerifySnapshot carries Satisfied beside the Href it read, so the projector names no href of its own. It was DashecorpAnchor.ReleasedHref, a const that dashecorp.com has served since 2026-09-21 — so the condition was already true before any task ran and every merged task PR reported success whatever it contained. The other half of the fix is the propose route's pre-state gate: a task may only be proposed from the template's from state, so reaching the to state means a change actually shipped.

"Not live" is never read from the clock alone: it needs a probe that read the page at or after the limit (any href but the expected one, or no card). With no such probe (the first status read came later, or every probe was unreadable) the task stays deployed or merged and keeps probing. expired, verified and failed are terminal, so the status route can stop reading GitHub there.

The timeline has one {stage, at, ref} entry per stage reached, in pipeline order, each from its own timestamp: the store Proposed time; the expiry; issue created_at (issue#N); the agent-ready label event (issue#N); PR created_at unless it is a draft (pr#N); release-hold heldAt and the verified release-approval at (short SHA), each only when its SHA is the PR's current head; PR merged_at (short merge SHA); the successful run's updated_at (actions), else the first live probe (probe); the live probe's checkedAt (the href); and for failed the issue or PR closed_at, the failed run's updated_at or the checkedAt of the probe that read the page after the 15-minute limit. submitting and unknown have no store timestamp, and a stage whose timestamp was not read is left out (implementing needs the label event time, not only the label). A hold or approval of an older head adds no entry, just as it sets no stage (an open PR stays in_review). The chat keys its lines by request and stage, so a stale approval_needed line would also hide the one for the current head once MergeGate holds it. The reader fetches that event on any read where the time is still unknown, whatever the PR state, so a first read after the PR opened (the tab was closed, or a restart dropped the in-process memo) still dates implementing. A chat that stopped polling adds one line for each entry it has not shown yet.

Status route (C11b)

CockpitTaskStatusRead (tests) builds the snapshot for the projector from the store and three ports: the snapshot reader and probe, and ValkeyOperatorTaskReleaseState, which reads the two hashes above and runs the nudge. The route answers the result as JSON (contract). Only a Submitted task with an issue number reads anything.

  • Walked end to end. CockpitTaskLifecycleWalkE2eTests drives one task through every stage below in order — including MergeGate holding the PR itself rather than merging it — and asserts the stage, detail, timeline and all 15 status keys at each one (why).
  • Memo (m14). GitHub, and the probe with it, is read at most once per 30 s per requestId, in process. The same window rate-limits the reconciliation list of an unknown outcome (routes). The last read passes to the next as previous, so the plan and the agent-ready time are read once. Valkey is read on every status.
  • Terminal. Once a read without a failed part projects expired, verified or failed, that answer is kept and nothing is read again: no GitHub, probe or Valkey call. A restart costs one more read. A failed deploy run is therefore final: a re-run that succeeds later is not seen.
  • A failed read (Failed non-empty) keeps the last known issue, PR, review or run for the part that did not answer, so one GitHub fault never moves the stage back. With the PR list unanswered and no PR known, a closed issue is not read as "closed with no pull request". A stage seen through a failed read is shown but never kept as terminal.
  • Probe. It runs with the GitHub read from the merge until the anchor is live, or until one probe read the page at or after 15 min past the merge; the projector then answers failed, "Not live 15 min after the merge", dated by that probe. A status first read after the limit (the tab was closed) still probes, so a live site is verified, not failed. An unreadable probe keeps the last result and the next GitHub read probes again.
  • Nudge (m2). When review-e-bot's latest verdict is APPROVED at exactly the head of an open, ready PR, more than 2 min ago, and release-hold.sha is not that head, the route calls TryNudgeAsync. It takes release-nudge:{repo}#{pr} with SET NX for 5 min and, only if it got it, runs MergeGate.RecheckHeldAsync off the request. The guard and the hold still apply, so a nudge can hold the PR but never merge it without a release approval. This covers an approval that arrived while a check was pending, when no event reran Merge() and no hold was recorded.
  • Release retry. When release-approval.sha and release-hold.sha both equal the head of an open, ready PR, review-e-bot still approves that head, and the approval is more than 2 min old, the route calls TryRetryReleaseAsync instead. It takes the same release-nudge key and runs MergeGate.RetryReleaseAsync off the request. This covers a release that answered 202 for a guard fetch error, or 502 for a failed PR fetch or PUT: the approval was recorded, no check_run is pending to finish the merge, and the Worker only releases at approval_needed, so the card would otherwise stay releasing forever.