The useful question is not how much experience an agent can remember. It is which experience deserves to change what the agent can do next.
Those are different decisions. Retaining an event preserves evidence. Turning that event into reusable behavior grants it authority over future work.
Collapse the two and every successful shortcut, accidental workaround, stale constraint, and confident mistake becomes a candidate for permanent behavior. Separate them and experience can accumulate without silently rewriting the agent.
My operating rule is simple: raw traces are evidence, not capability. Repeated experience should cross an explicit promotion boundary before it becomes an executable skill.
The operating read
- Retention preserves evidence; promotion grants future behavioral authority.
- WikiSkill separates raw traces, persistent knowledge, and executable skills.
- My production contract adds six checks: evidence, recurrence, scope, acceptance, version custody, and monitoring.
- Roll back bad behavior without deleting the history that explains it.
The memory debate starts one layer too early
Most discussions about agent memory ask what to retain, how to retrieve it, and how long it should persist. Those are important questions. But they do not answer the harder one: what may the agent do differently because of what was retained?
A trace might show that a command failed, a customer preferred a certain format, or a deployment recovered after a particular intervention. Keeping that trace makes later inspection possible. It does not establish that the same intervention should run whenever a superficially similar situation appears.
The distinction is behavioral authority.
Memory can inform a decision. A skill can shape or execute the decision. Because the second has consequences, it needs a stronger assurance standard than the first.
This does not make memory obsolete. It gives memory a more honest job: preserve context and evidence without pretending every retained fact is a validated procedure.
Raw experience is evidence, not capability
An execution trace is local. It records what happened under a particular model, tool state, prompt, environment, and set of constraints. It may contain a successful strategy. It may also contain irrelevant detail, an unobserved side effect, or a workaround that succeeded for the wrong reason.
Even repeated success needs interpretation. Three traces can repeat because they expose a general pattern. They can also repeat because the same hidden defect was present three times.
That is why direct trace-to-behavior learning is dangerous. It removes the stage where an operator asks:
- Which parts of this experience caused the outcome?
- Does the pattern recur outside one task or environment?
- Where should the behavior apply—and where should it stop?
- What known-good behavior could it break?
- How would we detect that it has become stale?
A trace can motivate those questions. It cannot answer them merely by existing.
What WikiSkill contributes
A recent research architecture makes this separation concrete. In their August 27, 2026 arXiv v1 preprint, “WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution,” Liyan Tang, Cyrus Rashtchian, Chun-Sung Ferng, Andrew Tomkins, Da-Cheng Juan, and Tu Vu divide agent learning into three connected layers:
- Raw experience: immutable execution traces containing complete task trajectories.
- Accumulated knowledge: a persistent wiki that consolidates strategies, failure modes, recurring patterns, and evolution history.
- Executable skills: procedural instructions that can affect later task performance.
The separation matters more than the storage format. Raw experience is preserved. A knowledge layer interprets experience across iterations. Skill changes are proposed from that accumulated knowledge rather than copied directly from one trajectory.
WikiSkill also treats knowledge and skills differently when a proposed change fails. Candidate skill changes are evaluated on a validation split. Improvements are accepted; degrading changes revert to the most recent successful skill state. The wiki history remains, including evidence about the failed proposal.
That is a useful design choice: rollback behavior without deleting the lesson. A rejected procedure can still teach the system why that procedure was rejected.
The paper's ALFWorld case study shows the distinction in practice. An early abstract proposal—roughly, choose actions that move toward the goal—was rejected. It was too general to produce a validated improvement. A later proposal targeting a concrete repetition loop was accepted, then refined as recurring evidence made the pattern more precise.
The lesson is not that one benchmark proves a production policy. It is that accumulated knowledge can remain broad while executable behavior stays conditional.
The paper is a recent preprint, not peer-reviewed consensus or evidence of production reliability. It directly evaluates skill evolution in its own benchmark setup. It does not evaluate production monitoring, semantic versioning, large-catalog skill retrieval, or indefinite knowledge growth.
WikiSkill supplies the architecture; the six checks below are my operator contract for deciding when evidence earns behavioral authority.
The Trace-to-Skill Promotion Contract
A promotion boundary should be strict enough to block anecdotes but practical enough to use. I would require six checks.
1. Linked evidence
Can the candidate point back to the traces, outcomes, and recurring patterns that motivated it?
A skill without evidence is folklore encoded as procedure. Keep the motivating traces, the observed outcomes, and the interpretation that connected them. The link should survive later edits so an operator can ask why the skill exists.
Evidence linkage does not prove the skill is correct. It makes the claim inspectable.
2. Recurring pattern
Is this a repeated, generalizable pattern rather than a one-off anecdote?
Recurrence is not a magic count. The required evidence should rise with consequence. A formatting preference may need little proof. A skill that mutates production data should need repeated evidence across materially different cases.
The important move is to define the pattern, not merely count similar-looking traces. “This command worked three times” is repetition. “This recovery procedure works when these three preconditions are present” is the beginning of a reusable claim.
3. Bounded specification
Are the trigger, scope, procedure, non-scope, failure boundary, and exit condition explicit?
A useful skill says when it applies and when it does not. It names the state it may inspect, the actions it may take, and the condition that ends execution. It also says what should happen when required evidence is absent or contradictory.
This is where broad advice becomes operable. “Investigate the failure” is guidance. “Read the current deployment state, compare the active revision, stop before mutation if they differ, and escalate with both receipts” is a bounded procedure.
4. Held-out acceptance
Does the candidate improve or preserve the intended outcome on cases other than the traces that produced it?
Testing only against the motivating traces rewards memorization. A candidate should face cases that include normal operation, the target failure mode, adjacent but out-of-scope conditions, and known regressions.
WikiSkill uses separate training, validation, and test splits and gates candidate skill changes on validation improvement. In production systems, I would extend that idea into a standing acceptance suite tied to the skill's declared scope.
This is the same separation I use for agent corrections: reflection can propose a change, but verification must authorize it.
5. Versioned promotion
Is the exact promoted artifact identifiable, along with its evidence, diff, test result, owner, and prior active version?
“Updated the skill” is not enough. An operator should be able to name what became active, what it replaced, and why. The version can be semantic or another unambiguous identifier; what matters is custody of the state transition.
WikiSkill records proposals, diffs, validation scores, and acceptance outcomes. A release registry and explicit version policy are production hardening I would add, not requirements established by the paper.
6. Monitoring
After promotion, can the operator detect regression, drift, stale evidence, or scope violations?
A held-out test answers whether the candidate was acceptable before promotion. Monitoring asks whether it remains acceptable under live conditions.
Useful signals include trigger frequency, success and fallback rates, policy violations, unexpected tool calls, latency or cost changes, and cases where operators override the skill. Monitoring should connect back to the active version so a regression can be attributed instead of merely noticed.
The promotion decision can be stated compactly:
Promote a skill only when its evidence is linked, its pattern recurs, its behavior is bounded, its acceptance tests pass, its active version is identifiable, and its live effects are observable.
A durable skill needs provenance and a kill switch
Promotion is not the end of the lifecycle. A skill needs two invariants after it becomes active: rollback and supersession.
Rollback means a degrading executable skill can return to the most recent accepted version without deleting the traces, knowledge, tests, or audit history that explain what happened. WikiSkill directly demonstrates this separation between reversible skill state and persistent wiki evidence.
Supersession means a newer accepted version identifies the prior version it replaces. The older version remains inspectable but is no longer canonical. This is an operator requirement I would add; the paper supports iterative refinement but does not define a formal superseded state.
Together, these rules prevent silent accumulation. The system does not keep appending instructions until contradictions become invisible. It knows which version is active, which evidence authorized it, and which state can be restored.
A practical kill switch is the operational expression of rollback. It should disable or revert the skill without requiring the failing skill to diagnose itself. That control belongs outside the behavior it governs.
Memory still has a job
The alternative to indiscriminate memory is not amnesia. It is a layered model with different assurance requirements:
- Working context helps the agent complete the current task.
- Historical traces preserve what happened.
- Accumulated knowledge consolidates patterns across traces.
- Executable skills encode behavior that has passed a promotion contract.
These layers may share infrastructure, but they should not share authority by default.
A note that says “the last deployment failed during migration” is useful context. A knowledge entry connects that event to similar failures. A skill defines the checked recovery procedure. Only the skill should be eligible to drive repeatable action—and only after its scope, evidence, tests, version, and monitoring are explicit.
That boundary also improves correction. New evidence does not need to erase old evidence. It can challenge the current interpretation, trigger a candidate revision, and leave the active behavior unchanged until the revision passes.
Compile slowly; execute confidently
Start with one consequential behavior that currently lives in prompts, memory snippets, or operator habit.
Collect the traces that motivate it. Name the recurring pattern. Write the trigger, scope, procedure, non-scope, failure boundary, and exit condition. Build acceptance cases that were not used to invent it. Promote one identifiable version. Monitor its effects. Make rollback real before the first failure.
That process is deliberately slower than telling an agent to remember what worked.
It should be. Compilation is where uncertain experience becomes governed capability. Once a skill has crossed that boundary, the agent can execute it with more confidence because the operator can still answer the questions that matter: Where did this behavior come from? What was tested? Which version is active? Is it still working? How do we stop it?
Preserve experience generously. Grant behavioral authority carefully.
Compile slowly; execute confidently.
Source
- Tang, Liyan, Cyrus Rashtchian, Chun-Sung Ferng, Andrew Tomkins, Da-Cheng Juan, and Tu Vu. “WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution.” arXiv:2608.27454v1, submitted August 27, 2026.