Context budgeting for the openai-api provider

The failure

Asking Rig-E "what can you tell me about the rig?" from the phone returned a red ERROR — upstream status 502, rendered directly beneath a green banner reading "GPU brain is ready — model loaded and serving."

Both panels were correct. The pod was RUNNING, the model was loaded, and the request was rejected for being one token too long.

14:06:31  OpenAI API call turn=0, messages=4
14:06:54  OpenAI tool call: get_api_health
14:06:54  OpenAI tool call: get_api_agents
14:06:54  OpenAI tool call: get_api_streams_status
14:06:54  OpenAI tool call: get_api_stuck-issues
14:06:54  OpenAI tool call: get_api_issues
14:06:55  OpenAI tool call: get_api_events_recent
14:07:01  OpenAI API call turn=1, messages=13
14:07:02  400 This model's maximum context length is 16384 tokens ... at least 16385 input tokens

It is deterministic, not cumulative

This was first described as something that bites "a few questions into a conversation", as though context accumulated. It does not. The turn loop appended each tool result with JSON.stringify(result) and no bound whatsoever, then posted the lot on the next call.

What separates a working question from a broken one is fan-out, not history:

question	tool calls	outcome
"Are you awake?"	1-2	answers fine
"What can you tell me about the rig?"	6	502, on the first attempt

The same broad question failed identically twice, fifteen minutes apart, in different pod states (2026-09-10 21:53Z and 2026-09-11 14:07Z). Both landed on exactly 16385 tokens against a 16384 ceiling, because vLLM reports the first count past the limit rather than the true total — so the real overshoot is unknown and larger than one token.

The consequence is worth stating plainly: the single most natural question to ask this agent was the one guaranteed to fail.

The fix

src/agent/providers/context-budget.js trims tool results to fit before the call, rather than trying to recover from the rejection. There is nothing to recover: vLLM refuses the whole request, so no partial answer exists, and a retry posts the same payload.

Largest-first. The biggest payload is cut to the size of the next largest, then the total is re-measured. With six results where one is enormous, shaving all six equally would destroy five small useful ones to save the outlier.

Measured from inside the cluster on 2026-09-11, that is not a hypothetical distribution:

endpoint	bytes	~tokens	share
/api/issues	316,876	~90,536	97.0%
/api/agents	7,738	~2,210	2.4%
/api/events/recent	789	~225	0.2%
/api/stuck-issues	702	~200	0.2%
/api/streams/status	406	~116	0.1%
/api/health	57	~16	0.0%
total	326,568	~93,305	5.7× the window

One payload is 97% of the fan-out and the other five together fit with room to spare. Proportional shaving would have destroyed five useful results to save the only one that is actually the problem.

A correction worth keeping. The first draft of this document asserted that get_api_events_recent was probably the largest, because it took 8.2 s and 6.8 s in the two incidents while everything else returned in milliseconds. It is 0.2%. Payload size was inferred from response duration, and the two are unrelated: /api/events/recent is slow and tiny, /api/issues is fast and enormous. One command overturned it. Measure the thing; do not infer it from a number that happens to be nearby.

Pessimistic estimation. CHARS_PER_TOKEN = 3.5, below the ~4 of English prose, because tool output is JSON — dense with braces, quotes and ISO timestamps, which tokenize poorly. Estimating low on chars-per-token means estimating high on tokens. That asymmetry is deliberate: over-estimating trims a little more than needed and costs some detail, while under-estimating reproduces the bug.

Output headroom. max_model_len bounds input plus output, so DEFAULT_RESERVE_OUTPUT_TOKENS is withheld. Budgeting the whole window for input would fit the request and leave the reply no room to finish.

The tools schema counts. Six tool definitions are sent on every call. Measuring only the messages would under-count by a meaningful margin.

In-band truncation markers. A trimmed payload says so inside its own content, and tells the model the data is incomplete rather than to infer the remainder. A silently shortened result is worse than an error, because the model will answer confidently from half the evidence.

A floor. Below MIN_TOOL_RESULT_CHARS a payload stops being evidence and becomes something to hallucinate around. If everything is at the floor and it still does not fit, the module reports fits: false and gives up rather than looping — a hung chat reply is worse than an oversized request that gets a clean 400 back.

Why the overflow is answered in words

chat.js deliberately returns a generic 502 {error: 'the agent provider failed'} for provider failures, because provider error text routinely carries env var names, credential paths and prompt fragments, and that response crosses a network boundary to a phone. That rule is right and stays.

A context overflow is a narrow, principled exception, and it does not weaken the rule:

• The sentence sent is a constant in chat.js, never the provider's text. Nothing can ride out on it.
• It is a 200 with a normal assistant reply, not an error, because it is a conversational outcome rather than an infrastructure fault. The brain is up; it was asked for something that does not fit.
• It is the one 4xx here an operator can act on — by asking a narrower question — so it is the one worth spending words on.

The reply names the cause, states that the GPU is healthy, and suggests the narrower question that will work.

That contradiction between panels is most of what made this expensive. A red 502 under a green "ready" banner tells the operator the brain is down when it is running and healthy, and on 2026-09-10 that reading cost a full misdiagnosis of the GPU.

Configuration

Value	Default	Meaning
llm.maxContextTokens	16384	Must match vLLM's --max-model-len for the served pod
llm.reserveOutputTokens	2048	Withheld from the input budget for the reply

If the pod's --max-model-len changes, change llm.maxContextTokens with it. They are two statements of the same number in different places, and the failure when they disagree is this bug.

Raising --max-model-len on the pod is not an alternative to this module. It costs VRAM for the KV cache and only moves the ceiling, so the same question against a busier rig reaches it again. At 5.7× over, it would also need to roughly six-fold — which the served AWQ pod cannot give.

What this does NOT fix

Trimming 90,536 tokens down to the ~13,000 that remain means the model sees roughly 14% of the issues. It is told, in-band, that the rest is missing, so it will say so rather than invent it — but the answer's ceiling is set by what the endpoint sends, not by what trimming can salvage.

/api/issues returns every issue with its full markdown body. Dropping body from that default projection is the actual cure, tracked at dashecorp/rig-conductor#2080. This module stops the 502; it does not make the answer good, and the two should not be confused when this closes.

Source

File	Role
src/agent/providers/context-budget.js	estimateTokens, fitToolResults, isContextOverflowError
src/agent/providers/context-budget.test.js	The real six-tool failure shape, largest-first, the floor, the tools overhead
src/agent/providers/openai-api.js	Budgets each turn; classifies the overflow 400
src/chat.js	CONTEXT_OVERFLOW_REPLY — the words sent instead of a 502

Issue: #696. Related: #685 (model-loading vs pod-asleep), #695 (the wake reply that reported time since the last message rather than wake duration).