m@berryhill: ~/berryhill.dev/posts/adaptive-reasoning-choose-compute-not-defaults.md
~ homeposts/about.md
m@berryhill in ~/posts$ cat adaptive-reasoning-choose-compute-not-defaults.md 
---
title:  Adaptive reasoning: choose compute, not defaults
date:   2026-08-22
topic:  ai-agents
read:   10 min
words:  2,214
slug:   adaptive-reasoning-choose-compute-not-defaults
views:  live post
tags:   [ai-agents, agent-ops, reasoning, reliability]
---
essay · long read

Adaptive reasoning: choose compute, not defaults

Reasoning compute is a routing decision, not a default. Pick per-problem effort and prove the choice with token, latency, and accuracy gates.

by matthew berryhill
·
posted 2026-08-22
·
updated 2026-08-22
·
topic ai-agents
table of contents
  1. Write down the budget
  2. What the paper's policy learned
  3. A router needs gates
  4. The Adaptive Router Scorecard
  5. Refresh the policy when traffic moves
  6. Run the comparison
  7. Source

A classification call and a hard planning problem should not automatically receive the same reasoning budget. Yet that is what many systems do: choose one reasoning setting, apply it to every request, then treat the resulting token bill and latency as properties of the model.

They are policy choices.

An August 2026 arXiv v1 preprint by Gijs Kassenaar, Zhao Yang, and Vincent François-Lavet, "Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation" (arXiv:2608.20256v1), tests a different approach.(paper) The model selects one of three effort modes through its first generated token: NoThink, Short, or Long. The selection is learned inside Group Relative Policy Optimization (GRPO). There is no separate router model sitting in front of the response.

In the authors' setup, mean response length on held-out MATH500 fell from 4,796 to 2,811 tokens, a 41% reduction. Accuracy moved from 0.796 to 0.782. On GSM8K, which was not in the training corpus, the same policy used 76% fewer tokens than the Long-only baseline without retraining.(paper)

Those results come from one 1.5B distilled model trained on MATH.(paper) They do not establish production economics for hosted frontier models, multi-turn agents, or tool-using workflows. They do establish a useful operating question: how much reasoning should this call buy?

Write down the budget

Most teams set a reasoning budget once. They select a model, accept its default reasoning.effort or thinking.budget_tokens, and move on.

That works when the workload is uniform. Mixed workloads expose the tradeoff. Easy calls consume tokens they do not need. Hard calls hit the same cap even when more compute could improve the outcome. A fixed setting quietly redistributes cost and risk across traffic classes.

Always-think vs Never-thinkTwo side-by-side failure modes. Always-think on the left over-spends on every call with high cost and high latency but flat correctness. Never-think on the right under-spends on every call with low cost and low latency but turns one bad call into a production incident. The router scorecard in the middle is the missing artifact between them.ALWAYS-THINKCoT or extended thinking on every callCostballoonsLatencyballoonsCorrectnessflat at bestHigh token meter. Impressive cost column.NEVER-THINKCheap single-pass on every callCostlooks greatLatencylooks greatCorrectnessone bad callUntil the cheap call costs a production incident.ROUTER SCORECARDthe missing middle
Both extremes hide the same decision: the operator never named the budget.

A default can be defensible. An invisible default cannot. Name the token ceiling, latency budget, accuracy floor, and fallback for each traffic class before tuning the selector.

What the paper's policy learned

The policy emits a mode token at the start of its response:(paper)

  • NoThink uses no internal scratchpad.
  • Short allows a brief reasoning trace under a small token cap.
  • Long allows a larger reasoning trace under a higher cap.

GRPO trains that self-routing choice with a reward that balances correctness against token use. The mode caps are part of the training regime.(paper) Copying the three labels into an API wrapper will not reproduce the learned policy.

The three-mode adaptive routerA horizontal flow. Each incoming problem reaches the router on the left; the router selects one of three discrete effort modes — NoThink, Short, or Long — with named per-mode token caps. The router is learned, not thresholded.SELECT ONE EFFORT MODE PER PROBLEMProblemone inbound callone verifierROUTERGRPO-trained policythree discrete modesNOTHINKno scratchpad · single-passSHORTbrief inline trace · small capLONGfull CoT · larger capPer-mode cappart of the regime,not a suggestionWhat the shaped reward enforcesPenalize tokens spent. Reward accuracy. The policy learns to drop to NoThink on easy problems and reach for Long only when the problem justifies it.
The useful mechanism is first-token self-routing inside the trained policy, not a second model guessing difficulty.

The MATH500 result is a tradeoff, not a free optimization. Across three seeds, the policy cut mean response length by 41% while accuracy fell 1.4 percentage points.(paper) Whether that exchange is acceptable depends on the workload's accuracy floor and the actual cost of latency and tokens.

The GSM8K transfer result is encouraging but narrower than a deployment claim. It shows that the learned policy transferred to one adjacent math distribution.(paper) A different difficulty mix, verifier, prompt convention, or model can change the mode distribution and the outcome.

A router needs gates

Before deploying an adaptive reasoning policy, answer four questions:

  1. What is the hard per-call token ceiling for this traffic class?
  2. What latency budget must the call meet?
  3. What accuracy loss, if any, is acceptable against the fixed-budget comparator?
  4. Which condition sends the call back to the last validated policy?

The comparison also needs a representative evaluation slice, a fixed-budget baseline, a verifier, and an outcome signal. A team may not need a new public benchmark, but it still needs a controlled test on its own workload.

This is the same discipline required for agent audit budgets: compare the new policy against a control rather than trusting the policy's confidence.

The Adaptive Router Scorecard

The scorecard below is an operator checklist for promoting or retiring an adaptive reasoning policy. It is not part of the paper and it is not a universal points system.

Field Question it answers
effort_mode Which mode did the policy select for this call?
token_spend How many tokens did the call use against its cap?
latency Did it finish inside the traffic class budget?
accuracy_delta How did the outcome compare with the fixed-budget policy?
fallback_rules Which condition restores the last validated policy?
audit_trail Can the decision and its inputs be replayed?
refresh_cadence When will the policy be retested on current traffic?
The Adaptive Router ScorecardSeven gate fields in a single column. Each field is a gate, not a metric. Promotion requires all seven. Retirement triggers on any one.THE ADAPTIVE ROUTER SCORECARDSeven gates. Promotion requires all seven. Retirement triggers on any one.1effort_modeWhat mode did the router pick — NoThink, Short, or Long — and why?2token_spendHow many tokens did the call actually cost, against the per-mode cap?3latencyDid the call finish inside the latency budget for this traffic class?4accuracy_deltaHow did this call compare to the fixed-budget baseline on the same problem?5fallback_rulesWhich conditions force the call back to a fixed budget or to a safer mode?6audit_trailIs the decision and its inputs logged so the policy can be replayed?7refresh_cadenceWhen will the policy be re-evaluated against the current workload?
Promotion requires evidence across all seven fields. One failed gate is enough to keep the policy out of production.

The fields close the gaps this operating model exposes. Another workload may need more. The important part is that every promotion decision leaves a replayable record instead of a vague claim that adaptive reasoning "seems cheaper."

Refresh the policy when traffic moves

A selector is a snapshot of a workload. Retest it when:

  • the difficulty mix changes and mode shares drift;
  • a new traffic class changes the latency budget;
  • the verifier, prompt convention, or tool adapter changes the meaning of a correct outcome.

Keep the previous validated policy as the fallback while the new comparison runs. The audit trail should connect each mode choice to token spend, latency, verifier result, and fallback behavior.

Run the comparison

Start with one reasoning-heavy traffic class.

  1. Record the current fixed budget and hard per-call ceiling.
  2. Build a representative held-out slice with a verifier and outcome signal.
  3. Run the fixed policy and adaptive policy on the same slice.
  4. Compare token spend, latency, accuracy delta, and fallback rate.
  5. Promote only if the adaptive policy stays inside the accuracy floor and improves the budget that matters.
  6. Log every mode decision so the result can be replayed.
  7. Retest when the workload changes. Name the fallback first, because recovery is also a budget decision.

Do not debate adaptive reasoning in the abstract. Put one traffic class through the scorecard and keep the fixed policy until the evidence says to move.

Source

  • Gijs Kassenaar, Zhao Yang, and Vincent François-Lavet, "Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation," arXiv:2608.20256v1, August 2026 arXiv v1 preprint. The study trains a first-token, three-mode self-routing policy inside GRPO on MATH using a 1.5B distilled base model. Reported evaluations include held-out MATH500 (4,796 to 2,811 mean tokens; 0.796 to 0.782 accuracy across three seeds) and GSM8K transfer (76% fewer tokens than the Long-only baseline without retraining).
#tags # ai-agents # agent-ops # reasoning # reliability
m@berryhill in ~/posts$
$ cd ../ · back to posts/