m@berryhill: ~/berryhill.dev/posts/skill-libraries-need-an-optimization-budget.md
~ homeposts/about.md
m@berryhill in ~/posts$ cat skill-libraries-need-an-optimization-budget.md 
---
title:  Skill Libraries Need an Optimization Budget
date:   2026-09-13
topic:  AI agents
read:   14 min
words:  3,097
slug:   skill-libraries-need-an-optimization-budget
views:  live post
tags:   [AI agents, agent skills, agent infrastructure, llm ops, optimization]
---
essay · long read

Skill Libraries Need an Optimization Budget

Skill evolution is a billed optimization loop. A production skill library declares iterations, sampling, and validation cost up front, then treats overshoot as failure.

by matthew berryhill
·
posted 2026-09-13
·
updated 2026-09-13
·
topic AI agents
table of contents
  1. The operating read
  2. Skill evolution is a billed loop, not a free upgrade
  3. What the preprint actually paid
  4. The costs that are easy to undercount
  5. What the preprint supports — and what it does not
  6. The operator rule
  7. Set the budget before the loop starts
  8. Iteration is a feature, but it is also a meter

A skill library that is allowed to evolve without a budget does not become better for free. It becomes expensive.

Every iteration runs the inference agent on the training split. Every proposer call queries a large model to draft a candidate skill. Every maintainer call reads traces and rewrites the wiki. Every accepted proposal triggers a fresh validation rollout. Every benchmark score is averaged across three independent runs, because a single optimization trace is too noisy to trust.

Add up those costs across iterations, datasets, and model families, and the bill arrives long before the skill library is useful.

The useful question is not whether skill evolution works. The recent evidence says it does. The useful question is what it costs, and whether the cost has a budget.

The operating read

  • Skill evolution is not a free side effect; it is a billed optimization loop.
  • The recent WikiSkill preprint runs training rollouts, proposer calls, maintainer calls, validation rollouts, and three independent repetitions per benchmark per model.
  • A production skill library needs an envelope for iterations, sampled traces, wiki growth, and validation cost — declared before the first pass.
  • Overshoot is a failure signal, not enthusiasm. The budget is the contract.

Skill evolution is a billed loop, not a free upgrade

An evolving skill library is an optimization process. Each iteration has at least four expensive steps:

  1. Run the inference agent on the training split with the active skill set.
  2. Sample traces and have a maintainer agent consolidate them into the wiki.
  3. Have a proposer agent read the wiki plus traces and draft one skill update.
  4. Evaluate the candidate on a held-out validation split; keep it or roll back.

Each of those steps queries a language model and consumes tokens, latency, and compute. None of them is optional. A skill library that skips any of them either accepts stale behavior or accepts untested behavior.

The hidden cost is repetition. A single optimization trace is noisy. A reported average score that is not averaged across independent runs is an anecdote with a confidence interval drawn around it.

That cost compounds with the number of models, benchmarks, and editing choices under test. A serious evaluation multiplies all four.

What the preprint actually paid

The recent arXiv v1 preprint, "WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution" by Liyan Tang, Cyrus Rashtchian, Chun-Sung Ferng, Andrew Tomkins, Da-Cheng Juan, and Tu Vu, is the clearest current accounting of that loop. The paper is dated August 27, 2026, and its results are limited to the benchmarks and models it tested. It is a useful reference point because the authors wrote down the loop and measured its outputs.

The measured scope is explicit:

  • Five benchmarks: LiveMathematicianBench, SealQA, SpreadSheetBench, OfficeQA, and ALFWorld.
  • Five inference models: Qwen-3.5-4B, Qwen-3.5-9B, Qwen-3.6-27B, Gemma-4-31B, and Gemini-3.5-Flash.
  • Three independent runs of the full evolution process per model per benchmark, averaged for the reported scores.
  • Three competing baselines plus a no-skill baseline.
  • A held-out test split, distinct from the validation split used for gating.

The numbers reported in Table 1 are not a single trace. They are the mean of three evolution runs, with paired bootstrap significance at p < 0.05 across 1,000 iterations. That is the standard a serious skill-library claim needs to clear.

The proposed method shows consistent gains. Within the Qwen family, average improvement grows with model scale: +12.3 points for the 4B model, +17.5 for the 9B model, and +23.9 for the 27B model. Skill evolution also compensates for model scale: a Qwen-3.5-9B agent with WikiSkill-evolved skills reaches 47.4% average accuracy, exceeding a Qwen-3.6-27B agent without skills at 39.4%.

The point is not the magnitude of the gains. It is that producing the gains required a defined experimental scope, a defined repetition count, and a defined statistical test. The preprint earns its claims by paying for them.

The costs that are easy to undercount

Three cost lines tend to disappear when teams plan a skill library:

Trace sampling. Full training traces will not fit a proposer's context window. The preprint samples a subset of successful and failing traces per iteration to keep the proposer usable. The sampling rate is a knob. Raising it raises quality; lowering it lowers cost. Neither setting is free, and the cost is paid every iteration.

Wiki growth. The wiki is persistent. It accumulates patterns, evolution logs, and a skill-impact tracker across iterations with no hard size cap. A larger wiki helps the proposer; it also slows the maintainer, enlarges the proposer's context, and makes the next iteration more expensive than the last. The preprint does not report a measured wiki size across runs; the cost is implicit.

Validation rollouts. Every accepted or rejected proposal triggers a held-out evaluation. That cost is paid once per candidate, not once per accepted skill. A proposer that drafts noisy candidates pays a validation cost without producing accepted behavior.

There is a fourth cost the preprint does not measure: the human operator time to triage proposals, audit failures, and decide whether the library is converging or drifting.

A real skill-library budget has to enumerate all four.

Skill evolution is a billed loop Five numbered steps in each iteration, each with a cost column on the right, plus a repetition step showing three independent runs per reported score. The repetition note uses an amber accent; the rest use green. Skill evolution is a billed loop Per-iteration costs and repetition cost. 1 Training rollouts Inference agent × training split. rollouts 2 Trace sampling Subset chosen to fit context. samples 3 Maintainer + proposer Wiki update and skill draft. LLM calls 4 Validation rollout Held-out split, accept or roll back. rollouts 5 Independent repetition Three runs per reported score. ×3 runs Budget the loop · Mean of runs is the price of a claim Costs compound across iterations. Wiki growth is persistent and uncapped. Diagram is an operator model, not a measurement from the preprint.
Each iteration pays four costs, and a single reported score pays for three full repetitions. The wiki is the only line that grows without stopping.

What the preprint supports — and what it does not

The preprint supports four claims that hold within its measured scope:

  • Skill evolution beats no-skill baselines in most model–benchmark pairs in the five models and five benchmarks tested.
  • The wiki layer matters: the ablation removing the wiki shows that persistent knowledge accumulation is a real contributor, not bookkeeping.
  • Cross-model transfer is possible: skills evolved by one model can outperform self-evolved skills on another model, including across model families.
  • Larger models benefit more from skill evolution within the Qwen family tested.

The preprint does not support these claims:

  • A production cost model for skill evolution in operator environments.
  • A wiki-size budget or compression policy for indefinite operation.
  • A bound on the number of iterations needed to reach a target skill quality.
  • A claim about long-running agents that accumulate skills over weeks or months.
  • A claim about adversarial or noisy environments where the wiki can be poisoned.
  • A claim about skill-versioning, retrieval, or supersession in a production skill catalog.

The authors also report a failure mode worth naming: redundant tool calls that can exhaust Gemini-3.5-Flash's interaction budget before task completion. That observation is exactly the kind of cost signal a budget exists to surface.

A production skill library is not the same artifact as a benchmark that demonstrates skill evolution works. A benchmark can run for a week and report a clean number. A production library has to run forever and stay inside a budget.

The bandit acceptance gateTop: wiki and traces feed a proposer. The proposer drafts a candidate and the validation rollout scores it on a held-out split. The gate compares the new score to the best validation score; accept or roll back. The wiki accumulates on every iteration regardless of the gate outcome. A return rail loops from after the gate back to the proposer.The bandit acceptance gateHeld-out score is the only rule that accepts a candidate.Wiki + tracesPersistent contextProposerDraft candidatereadValidation rolloutScore on held-out splitCompare to best scoreΔ vs prior best validationImproves?Δ > 0 ⇒ acceptACCEPTSkill set updatedROLL BACKSkill set keptyesnoWiki persists regardless · Gate stops the skill set, not the looploop · return to proposer
Each iteration is a held-out bandit trial: keep the candidate if it improves on the best validation score, otherwise roll back. The wiki accumulates either way.

The operator rule

Define the budget before the first iteration. Treat overshoot as failure.

A practical budget has five lines, declared up front:

  1. Iterations. Pick a maximum. Stop when the budget is reached, not when the proposer stops proposing.
  2. Trace samples per iteration. Pick a number that fits the proposer's context. Lower it if the proposer is over budget; raise it only if the cost is approved.
  3. Wiki size. Set a soft cap and a hard cap. Prune or compress on the soft cap. Stop accepting new patterns at the hard cap.
  4. Validation rollouts per candidate. One is the floor. More is fine if the cost is approved; never unbounded.
  5. Repetitions. State the run count that a reported score must average across. One run is not a score.
Five lines declare the budget envelopeFive numbered budget lines listed vertically: iteration cap, trace samples per iteration, soft and hard wiki caps, validation rollouts per candidate, and repetitions per reported score. Each line ends with the operator rule for that cost. The footer names the budget as a contract that must be declared before the first iteration.Five lines declare the budget envelopeName the budget up front. Overshoot is information about the loop.1IterationsPick a maximum. Stop on budget, not on enthusiasm.max N2Trace samples per iterationFit the proposer's context. Lower when over budget.n samples3Wiki sizeSoft cap + hard cap. Prune, then stop adding.soft / hard4Validation rollouts per candidateOne is the floor. More only if the cost is approved.≥ 15RepetitionsState the run count. One run is not a score.≥ 3 runsBudget is the contract · declare it before the first iterationCompare used vs budgeted every iteration. If any line overshoots, the nextiteration gets a smaller envelope. Cost is information, not enthusiasm.
A budget is five named lines declared up front. Overshoot on any line is information about the loop, not enthusiasm about the result.

The point of the budget is not to be stingy. It is to make cost a visible, reviewable choice instead of an accidental outcome.

The same rule applies to gating. The preprint gates on validation score: accept a candidate if it improves on the best validation score, otherwise roll back the skill set to the last accepted state. The wiki persists regardless of acceptance. That is a useful pattern. It is also a pattern that can keep iterating forever if no one enforces a budget envelope.

Set the budget before the loop starts

Two habits make a skill library sustainable:

Name the budget in the same document as the acceptance criteria. A skill library without a declared budget is an optimization process with no stop condition. The acceptance rule says what counts as better; the budget rule says what counts as done. Both belong on the same page.

Compare used cost against budgeted cost every iteration. Rollouts consumed, samples drawn, LLM calls billed, validation rollouts paid, wall-clock minutes spent. If a single iteration overshoots its share, the next iteration gets a smaller envelope. Overshoot is information about the loop, not enthusiasm about the result.

A skill library that ships today without a budget will eventually be discovered by its finance team, a quota limit, a latency spike, or a wiki that becomes too large to read. None of those are pleasant discoveries. Each of them is the cost that was never named.

Iteration is a feature, but it is also a meter

Skill evolution is one of the few techniques that genuinely compounds. A persistent wiki, a proposer that reads it, a maintainer that consolidates traces, a validation gate that rolls back bad candidates — that combination has produced consistent gains in the recent preprint, and it transfers across model families.

It is also an optimizer. Every iteration is a paid experiment. Every proposer call is a billed decision. Every validation rollout is a held-out trial. The meter is always running.

The useful operator stance is not "let the loop run." It is "let the loop run inside the envelope we agreed to, and tell me when it stops making progress per dollar."

The synthesis

Skill libraries that improve under iteration are real. The preprint evidence is strong within its measured scope. What the preprint does not give us is a production cost model or a budget rule.

The operator rule is simple:

A skill library without an optimization budget is a research project, not a production system. Declare the envelope. Enforce it. Update it from observed cost.

Iteration is a feature. Iteration is also a meter. Both are true, and both belong in the same one-line brief.

A prior post argued for a six-check promotion contract before evidence earns behavioral authority: compile experience into skills, not memory. The promotion contract is what a single skill needs to earn its place. The optimization budget is what the loop needs to earn its keep.

Source

#tags # AI agents # agent skills # agent infrastructure # llm ops # optimization
m@berryhill in ~/posts$
$ cd ../ · back to posts/