Agent Audit Budgets Need a Control Group

An audit budget tells you how many AI-agent outputs a person can inspect. It does not tell you which outputs deserve those inspections.

The moment a team sorts that queue—by confidence, consequence, task type, or anything else—it has chosen an allocation policy. That policy should have to prove it reduces risk. A plausible signal is not enough.

My operating rule is: confidence is a hypothesis about where errors live, not an audit policy. Replay the policy against an independent random control and a coverage-oriented baseline. Promote it only when the comparison shows measurable risk reduction without hiding a critical blind spot.

The operating read

• Every sorted audit queue is an allocation policy, even when the team has not named it.
• The selector must compete on common traces, under a common budget and verifier.
• Random sampling supplies an independent control; coverage-oriented sampling tests blind spots.
• The five-part scorecard reviews lift, slices, coverage, verifier quality, and drift.
• Negative, inconclusive, stale, or invalid evidence triggers fallback—not a forced winner.

The audit queue is already a policy

Human review is usually described as a safety layer. That description skips the allocation decision inside it.

Suppose a fleet produces 10,000 outputs and the team can inspect 200. Reviewing “the lowest-confidence 200” sounds like a neutral use of the budget. It is not neutral. It makes several claims at once:

• reported confidence separates failures from successes;
• that relationship holds for the current models, tasks, prompts, and versions;
• the bottom of the confidence ranking contains the most important errors;
• repeatedly selecting that region does not leave another failure class unobserved.

Each claim may be reasonable. None becomes true because the queue is easy to sort.

This is the step beyond my earlier argument that confidence should not allocate an audit budget on its own. A mixed risk score and a protected random reserve are still proposed policies. The next question is not whether their logic sounds better. It is whether they beat credible alternatives on the same work.

Confidence creates false independence

A confidence-ranked queue often appears to generate its own evidence. The policy selects low-confidence outputs, reviewers find errors there, and the team concludes that low confidence was useful.

But the team observed only the region the policy chose. It does not know how many consequential errors were left in the confident majority, how much of the apparent yield came from one repetitive task class, or what an independent sample would have found at the same cost.

That is false independence: the selector chooses the evidence later used to justify the selector.

A control group breaks that loop. It gives the team an outcome stream the production ranking did not choose. Random sampling is not automatically the best production policy, and it may spend scarce reviews inefficiently. Its value is independence. It establishes what the audit process would have found without trusting the challenger’s ranking.

A coverage-oriented baseline answers a different question. Instead of sampling every eligible output with equal probability, it deliberately spans observed work classes or failure neighborhoods. Those labels are local operating constructs—task families, deployment versions, shared tools, retrieval sources, or another defensible partition. The point is not to claim that this baseline is optimal. It is to test whether a high-yield selector is repeatedly harvesting one cluster while leaving critical territory untouched.

What the preprint actually tested

A July 2026 v1 preprint by Cesare Zavattari, Alessandro Tommasi, and Giuseppe Prencipe, “One Human, N Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence”, studies this allocation problem under explicit assumptions.

The paper compares random auditing, confidence-ranked auditing, round-robin selection, a correlation-aware diversity_bayes policy, and an oracle reference with perfect error knowledge. The authors describe diversity_bayes as a knowledge-gradient-style instantiation. They do not establish it as an optimal production selector, and the oracle is an undeployable upper reference.[^1]

In the authors’ synthetic model, confidence-ranked auditing becomes worse than random after a modeled miscalibration threshold. That result is conditional on the model’s audit budget, calibration, error correlation, persistence, and verifier assumptions. It does not mean confidence ranking is always worse than random.[^1]

The threshold also moved with the audit fraction. At modeled audit fractions of 0.02, 0.05, 0.10, and 0.20, the reported flip thresholds were 0.93, 0.89, 0.84, and 0.75. The authors’ explanation is specific: tighter budgets remained in the extreme low-confidence tail, while larger budgets entered poisoned mid-ranks. That direction belongs to this modeled sweep; it is not evidence that tighter budgets are universally safer.[^1]

The paper then applies one locked, direct-answer confidence-elicitation protocol to five open-weight models. Their verbalized confidence was near constant. Point estimates were at or beyond the modeled flip, while the confidence intervals straddled it. Both parts matter. One protocol can reveal a problem without establishing that every open-weight model—or every way of eliciting confidence—has the same problem.[^1]

One proprietary model, gpt-4o-mini, provides a useful counterweight: its confidence was informative and landed below the modeled threshold. That is one proprietary robustness point, not a hosted-model survey or proof that proprietary models form a more reliable class.[^1]

The correlation result is similarly bounded. On aligned GSM8K and HotpotQA items in the authors’ decomposition, shared item difficulty explained co-failure better than lineage, and the reported confidence intervals for lineage-excess correlation included zero. In this setup, lineage added no measurable correlation. The study does not establish that lineage never matters, especially when production agents see different inputs.[^1]

Finally, a preliminary 15-agent recorded-trace replay preserved the ordering oracle, then diversity-Bayes, then confidence-ranked approximately equal to random on GSM8K and HotpotQA. The oracle again used perfect error knowledge and was not deployable. The effects were modest on GSM8K and larger on HotpotQA. A small benchmark replay is evidence worth testing against; it is not proof of production effectiveness.[^1]

The source has important limits. Its transfer advantage depends on persistent error profiles, verifier quality, and correlation strength. Correlation was estimated on aligned items and may overstate transfer when agents process different inputs. The confidence result comes from one elicitation protocol. The trace replay is preliminary and small. The preprint has not been treated here as peer-reviewed, independently replicated, or representative of a production fleet.[^1]

A control group changes the operating question

The useful production question is not “Should we trust confidence?” It is:

At the same audit budget, does this predeclared selector reduce more residual risk than an independent control, across the slices and failure classes that matter, using a verifier we can trust?

Answering that question requires a policy tournament. This tournament is my operator synthesis, not a method validated by the preprint.

Start with a common, versioned set of recorded fleet traces. Freeze the eligibility rules, audit budget, verifier, outcome definitions, consequence labels, and comparison window before scoring the selectors. Then replay at least these lanes:

1. The current production selector. This is the policy the team is already relying on, whether or not it has been named.
2. A confidence-ranked selector. This isolates what the confidence signal contributes.
3. An independent random control. This supplies a comparison stream that no ranking policy selected.
4. A coverage-oriented selector. This spans predeclared work classes or failure neighborhoods and tests for concentrated blind spots.

Every lane receives the same eligible traces and the same number of audits. Every selected item goes through the same verifier rules. Every lane reports the same outcomes:

• consequential errors caught;
• residual risk left after review;
• uncertainty around the difference from random;
• performance by model, task class, version, and time window;
• distinct failure classes reached;
• concentration of audits in any one class;
• verifier errors, missing labels, and ambiguous decisions.

Without that common contract, the tournament is theater. A selector cannot claim a win from a larger budget, an easier slice, a cleaner verifier, or a handpicked time window.

Correlated failures change coverage

Yield and coverage are not the same result.

Imagine a confidence-ranked lane finds 30 errors while the random lane finds 20. If 28 of the confidence-ranked errors are variants of one low-consequence failure and the random lane finds a rarer failure capable of triggering an irreversible action, raw yield gives the wrong winner. This is a hypothetical, not a measured fleet result, but it shows why the outcome definition must be written before the replay.

Correlation makes the same mistake less visible. In a plausible fleet configuration, several agents fail together because they share difficult inputs, prompts, retrieval data, tool adapters, policies, or dependencies. Auditing many members of one visible cluster can raise the caught-error count without buying much new information about the rest of the fleet.

That does not justify assigning fake probabilities to guessed clusters. It just means the comparison should report where the budget went and what distinct failure territory it reached. A coverage-oriented baseline is useful precisely because it can lose on raw yield and still expose a class the ranking ignored.

Coverage labels should remain contestable. If the team cannot label failure neighborhoods consistently, that uncertainty belongs in the scorecard. A neat cluster dashboard built from unstable labels is not evidence of coverage.

The Five-Part Audit Allocation Scorecard

The Five-Part Audit Allocation Scorecard below is my operating recommendation. It reviews comparative evidence; it is not a weighted point system, a universal threshold, or a scorecard from the paper.

1. Baseline lift

Ask: Does the challenger catch more consequential errors or leave less residual risk than the random control at the same budget?

Report caught-error lift and residual-risk difference with a predeclared uncertainty rule. Use severity-weighted risk only when the consequence labels and weights were defined before evaluation. Inconclusive evidence does not beat the control.

2. Calibration slices

Ask: Where does the priority signal actually separate errors from successes?

Measure tail precision and selector lift by model, task class, deployment version, and time window. A fleet-wide average can hide a selector that works for one stable lane and fails after a prompt or model change. Do not promote a global win that contains an unacceptable regression in a predeclared critical slice.

3. Failure coverage

Ask: Did the budget inspect distinct failure neighborhoods, or repeatedly sample one cluster?

Report unique failure classes reached, audit concentration, and consequential classes missed. Compare the challenger with the coverage-oriented lane. Require no unacceptable loss of critical-failure coverage, even when the challenger has higher raw yield.

4. Verifier quality

Ask: Can the audit process reliably identify the failures used to score the selectors?

Track verifier false negatives, false positives, or adjudicated agreement by slice. Keep missing and ambiguous labels visible. If verifier quality falls outside its predeclared operating range, fail closed. Apparent selector lift built on unreliable labels is not promotion evidence.

5. Drift and fallback

Ask: Does the selector keep earning its allocation after the system changes?

Recompute baseline lift, calibration slices, failure coverage, and verifier quality on rolling, versioned windows and after material changes to models, tasks, prompts, tools, or policies. Name the fallback before deployment. It will usually be the independent random policy or the last selector that passed the local evaluation contract.

The promotion rule is strict by design: a challenger earns budget only when it beats the random control under the predeclared uncertainty rule, avoids unacceptable critical-slice and coverage regressions, uses a verifier inside its valid range, and has a live drift monitor with a named fallback.

The fallback rule is equally important: if the evidence is negative, inconclusive, stale, or invalidated by drift or verifier failure, retain or restore the independent control or last validated selector. Do not force a winner because the organization wants one queue.

Human oversight must earn measurable risk reduction

Putting a human in the loop does not make an agent system safe by itself. The system still decides where that person looks, what evidence they see, how their decision is scored, and what happens when the policy stops working.

Treat those choices as an evaluation contract:

1. name the current allocation policy;
2. predeclare the eligible work, budget, verifier, slices, and outcomes;
3. replay competing selectors on the same recorded traces;
4. retain an independent random control and a coverage-oriented baseline;
5. promote only from comparative evidence;
6. monitor the same scorecard after deployment;
7. fall back when the selector no longer earns its authority.

The goal is not to prove that confidence, randomness, or coverage is universally best. It is to stop confusing an intuitive queue with a validated control.

Human attention is expensive and limited. Every allocation policy should be able to show what risk it reduced, what it missed, and when it will give the budget back.

Source

[^1]: Cesare Zavattari, Alessandro Tommasi, and Giuseppe Prencipe, “One Human, N Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence”, arXiv:2607.28317v1, July 30, 2026 preprint.