Benchmark Agent Decisions, Not Just Final Scores

A final score tells you where an agent ended. It does not tell you whether the agent earned the result.

An iterative agent can reach a strong endpoint after invalid configurations, weak diagnoses, repeated experiments, and one lucky change. Another can make sound decisions from the evidence available, stay within budget, and still finish behind because the run ended before the next useful intervention. If evaluation collapses both histories into one terminal metric, it hides the difference between reliable behavior and a fortunate outcome.

Terminal performance still matters. The practical mistake is treating it as the whole benchmark.

For agents that repeatedly observe, decide, act, and update, the more diagnostic unit is the decision sequence: what the agent saw, what it inferred, what it changed, what it expected, and what happened next. Operators need that sequence if they want to distinguish evidence-led progress from invalid action, wasted budget, and accidental wins.

A good endpoint can hide a bad run

Consider two hypothetical agents tuning the same model under the same experiment budget.

Agent A improves the final metric. Along the way, it misreads a failed run as a performance regression, repeats a configuration it has effectively tested already, proposes one configuration the system cannot execute, and finally lands on a strong result after changing several variables at once.

Agent B improves less. It identifies a bottleneck in the logs, changes one relevant control, predicts the direction of the result, rules out an unproductive region, and leaves a clear next experiment when the budget expires.

An outcome-only ranking puts Agent A first. An operator deciding which behavior to trust has a harder question.

Did Agent A discover a repeatable path, or did the benchmark reward the last number while forgiving the route? Did Agent B underperform, or did it create more useful evidence per intervention? Which agent would you give a larger budget, a more expensive tool, or authority to run unattended?

The final metric contributes to those answers. It cannot produce them by itself.

What AgentHPOBench actually evaluates

A July 31, 2026 arXiv v1 preprint by Tianyu Huai and seven coauthors introduces AgentHPOBench, a benchmark for evaluating LLM agents as sequential hyperparameter optimizers.[^agenthpobench]

According to the abstract, the benchmark contains 30 executable machine-learning tasks across seven research categories. Each begins with a validated baseline run. An agent then makes several sequential interventions, observing accumulated configurations, metrics, and logs before proposing the next valid configuration. The authors evaluate 12 widely used agents and conventional hyperparameter-optimization baselines under a unified protocol.[^agenthpobench]

The authors report that current agents show measurable experimental optimization ability across domains. They also report limitations in sustained iterative refinement, complex log diagnosis, and consistent progress toward reported reference performance.[^agenthpobench]

Those are the authors' abstract-level findings. This article does not treat the preprint as peer-reviewed, independently reproduced, or proof that one evaluation method is universally superior. It also does not infer model rankings, costs, effect sizes, or causes that the abstract does not establish.

The useful shape of the benchmark is narrower: the agent does not get one shot. It receives evidence, makes an intervention, sees a consequence, and has to decide again. That structure exposes an operating problem hidden by final-answer evaluation. Once actions are sequential, earlier decisions change the evidence available to later ones.

The decision sequence is the diagnostic unit

An iterative experiment is not a bag of independent attempts. It is a chain:

evidence → diagnosis → intervention → consequence → updated evidence

A weak diagnosis can waste the next run. An invalid intervention can consume time without producing interpretable evidence. A change to several variables at once can improve the metric while making the cause harder to identify. A failed experiment can still be useful if the agent predicted a discriminating result and updates correctly afterward.

This is why the endpoint and the path answer different questions.

The endpoint asks: How good was the final result?

The path asks:

• Did the agent use the evidence available at that moment?
• Was its diagnosis consistent with the metrics and logs?
• Did the next intervention test that diagnosis?
• Was the proposed action valid and executable?
• Did the observed consequence match the expectation?
• Did the agent update rather than repeat the same mistake?
• How much budget did the decision consume?

Neither view should replace the other. A clean decision process that never improves the target is not enough. A strong target reached through invalid or uninterpretable moves is not enough either.

Separate outcome quality from process quality

A useful evaluation report should keep at least two lanes visible.

Outcome quality describes where the run finished. Depending on the task, that can include terminal performance, distance from a reference, task completion, constraint satisfaction, or another predeclared result.

Process quality describes how the agent moved. It can include validity, evidence use, diagnosis quality, intervention discipline, response to consequences, and budget consumption.

Keeping these lanes separate prevents one score from laundering the other.

A high outcome should not erase invalid actions. A tidy trace should not excuse a poor endpoint. A lucky result should remain a result, but it should not be mistaken for evidence that the policy behind it is reliable. A useful failure should remain a failure, but the evidence it creates can still matter when deciding whether the agent deserves another run.

I would not force these dimensions into a universal weighted score. The weights would smuggle product risk and resource economics into a number that looks more objective than it is. Instead, define the operating contract for the specific system:

1. which terminal outcomes are required;
2. which actions make a run invalid regardless of outcome;
3. which process evidence must be present;
4. which budget limits apply; and
5. which tradeoffs require human review.

That produces a benchmark result an operator can interrogate instead of merely sort.

Use an Agent Decision Trace

For every consequential intervention, I would preserve an Agent Decision Trace with six entries:

1. Context — The experiment state when the decision was made: task, current configuration, remaining budget, applicable constraints, and relevant prior attempts.
2. Evidence used — The specific metrics, logs, errors, comparisons, or prior consequences that informed the decision. References are better than a vague claim that the agent “reviewed the results.”
3. Decision — The diagnosis and the intervention selected. Keep the explanation close enough to the action that a reviewer can tell what was actually chosen.
4. Expected effect — What the agent expected to change and why this intervention would test the diagnosis. An expectation makes the decision falsifiable.
5. Observed consequence — What happened after execution, including failures, side effects, and evidence that remained ambiguous—not only the headline metric.
6. Validity and budget — Whether the action was valid and executable, what it consumed, and whether the information gained justified another step under the remaining budget.

This trace is my operator synthesis. It is not presented as AgentHPOBench's measured rubric or as a framework validated by the paper.

The fields create a simple review loop. Compare evidence used with the decision. Compare the expected effect with the observed consequence. Check validity independently of performance. Then inspect whether the next decision incorporates what the prior intervention taught.

That last transition matters. A single plausible explanation is cheap. Reliable iterative behavior appears when the agent revises its next move after reality disagrees.

Account for validity and budget

Invalid actions should not disappear into an average.

If an agent proposes a configuration outside the allowed space, calls an unavailable tool, violates a resource constraint, or creates an experiment the harness cannot interpret, the run has produced operational evidence. Silently converting that action into a default configuration or simply moving to the next attempt makes the benchmark look cleaner while hiding a deployment-relevant failure.

Budget needs the same visibility. Iterative agents spend more than tokens. They can consume accelerator time, API calls, wall-clock time, human attention, scarce test environments, and opportunities to run a better experiment.

The right question is not only “Did the metric improve?” It is “What did this intervention buy?”

An unsuccessful run can buy information if it rules out a diagnosis or narrows the next decision. A successful run can buy little understanding if several variables changed and the agent cannot explain which evidence should guide the next step. The accounting does not need a universal currency, but it does need a declared budget and a record of what each consequential decision consumed.

This also reveals two different failure modes:

• stagnation: valid interventions continue but do not create useful progress or diagnosis;
• thrashing: the agent spends budget revisiting regions, repeating mistakes, or reacting to noise without a stable update rule.

A terminal score can hide both until the budget is gone.

Instrument selectively

Decision-level evaluation does not require retaining every token or turning every experiment into permanent surveillance.

Preserve the minimum evidence needed to reconstruct consequential choices. That usually means references to the relevant configuration, metrics, log excerpts, decision, expectation, result, validity check, and budget ledger. Raw prompts, full internal traces, sensitive datasets, and routine no-op steps may need stricter access, shorter retention, redaction, or no retention at all.

The selection rule should be explicit:

• capture actions that change experiment state;
• capture diagnoses that justify those actions;
• capture evidence needed to test the diagnosis;
• capture validity failures and budget effects;
• sample lower-consequence steps only when needed for drift or debugging.

This gives reviewers a decision record without pretending that more telemetry automatically creates more truth.

The trace should also be versioned with the harness, task, model, tools, and policy that shaped the available actions. Otherwise a later reviewer may grade a past decision against evidence or constraints the agent did not have.

Grade whether progress was earned

The practical benchmark report I want is not a single leaderboard column. It is a review surface that lets an operator see:

• the final outcome;
• the sequence of consequential decisions;
• the evidence available for each decision;
• invalid or uninterpretable interventions;
• expected versus observed effects;
• repeated errors and useful updates; and
• budget spent to produce the result.

Then the evaluation can distinguish four cases that one terminal score blurs:

1. Strong outcome, strong process: the result is good and the path supplies evidence worth trusting.
2. Strong outcome, weak process: the result counts, but the behavior needs investigation before receiving more authority.
3. Weak outcome, strong process: the run failed its target, but it may have produced a sound diagnosis or next experiment.
4. Weak outcome, weak process: neither the endpoint nor the trace supports continued confidence.

This is not an argument to stop measuring final performance. It is an argument to stop asking the final number to certify behavior it cannot see.

As agents take longer sequences of actions, the benchmark has to preserve enough of the sequence to tell whether progress came from evidence, whether the interventions were valid, and whether the budget produced learning instead of motion.

A good endpoint is valuable.

A good endpoint reached by a decision process you can trust is the result you can build on.

[^agenthpobench]: Tianyu Huai et al., “AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers”, arXiv:2607.29626v1, July 31, 2026 preprint. Paper-specific claims in this draft are limited to the arXiv abstract and attributed to the authors.