A memory budget tells an AI system how much it can keep.
It does not tell the system what deserves to stay.
That distinction matters because “give it more context” has become the default answer to memory failures. Sometimes more capacity is the right answer. But capacity only delays the decision. Once the working set reaches the bound, the system still needs a rule for what survives.
That rule is the memory policy.
A useful policy has to answer three questions: what signal predicts future value, when the system has enough evidence to decide, and which workload outcome proves the rule helped. Without those answers, retention is an accumulation instinct wearing an architecture diagram.
A new preprint about KV-cache eviction makes this unusually clear because its positive and negative results point in the same direction. Delaying an eviction decision can help when near-future behavior reveals what will be reused. The same technique does not become a general improvement when that signal stops separating useful memory from everything else.[^paper]
The evidence is narrow: bounded model working memory, specifically KV-cache selection. It is not a validated policy for episodic memory, retrieval databases, durable user records, or an agent’s complete continuity layer.
The broader operating lesson is mine: retained memory has to earn its cost.
A memory budget is not a memory policy
During autoregressive generation, a KV cache lets the model reuse prior key-value calculations instead of recomputing them for every new token. That can improve generation speed, but the cache consumes memory as the sequence grows.[^hf]
An eviction method decides which cached items remain when that memory is bounded.
Many methods have to score an item with the evidence available at or near arrival. The paper, Eviction as Estimation, organizes this decision around commit lag: the time between an item arriving and the system making its keep-or-evict decision.[^paper]
At one end, the system commits immediately. It is fast and online, but it knows little about what happens next.
At the other end, an offline policy can inspect the future. It has better evidence, but it is not a deployable online decision rule.
Between them is a more practical idea: wait for a bounded observation window, watch how the item participates in near-future work, and then commit.
The paper’s Retrospective Memory Management method, or RMM, does this with an H-step provisional buffer. An arriving cache item remains provisional for H steps. The method measures its “demonstrated utility”: attention the item receives during correct near-future predictions. It then keeps the provisional item if that score outranks the weakest committed item. Otherwise, it evicts it.[^paper]
The delay is not free. The committed cache holds K items, while the provisional window adds H more during operation. Waiting for evidence consumes memory too.
That is the first useful correction to the usual memory conversation: delaying a decision can improve the evidence, but it changes the resource envelope.
The controlled win and the benchmark loss belong together
The controlled experiments show why the idea is interesting.
In a constructed 16-fact reasoning probe, the model had to combine two relevant facts among fourteen distractors. When the reasoning was correct, demonstrated utility placed both needed facts near the top roughly 0.8 of the time. Accumulated attention and recency were around 0.1.[^paper]
In another constructed streaming task, reusable facts carried a clear reminder before they mattered again. With a memory budget of 40 items, RMM stayed near full-retention performance across a trajectory ten times larger than the budget. Its re-query score was about 0.70, compared with an unbounded ceiling of 0.80. Recency scored 0.00.[^paper]
That is good mechanism evidence. The tasks were built to create sharp, time-separated reuse and make the useful signal visible.
Then the independent benchmark results remove the temptation to oversell it.
On the paper’s LongBench single-turn tests, the authors report RMM as statistically on par with H2O overall, not better. On HotpotQA, H2O scored 0.309 versus RMM’s 0.299 while keeping 50% of the cache, and 0.316 versus 0.293 while keeping 25%. On TREC, both scored 0.725 at the 50% budget; RMM led 0.600 to 0.550 at 25%.[^paper]
The streaming LoCoMo result was worse. With a 1,800-token budget across eight conversations, RMM scored 0.147. H2O scored 0.176, SnapKV 0.209, and the full cache 0.218.[^paper]
The controlled result did not turn into a general benchmark advantage.
The authors’ diagnosis is that the correctness weight became nearly uniform on natural text in their runs. The model predicted roughly 85% to 90% of tokens correctly, so correctness-weighted attention stayed close to accumulated attention. The methods separate when reuse is sharp, localized, and connected to the model’s near-future behavior. When correctness is diffuse, the extra signal adds little distinction.[^paper]
That mixed result is not an embarrassment to hide in the limitations section. It is the most operationally useful part of the paper.
A retention signal is not good in the abstract. It is good in a workload where it predicts later reuse better than the baseline.
The deployment gap is the policy gap
There is another boundary that matters.
The paper determines correctness with teacher forcing: at each position, it can compare the model’s top prediction with the next token that actually occurred. A live system does not have that future answer at decision time.
The authors explicitly identify this as a deployment gap. A confidence score or verifier might serve as a proxy, but the paper did not test either one.[^paper]
So “observe future utility” is not an implementation instruction. Production cannot directly observe future usefulness. It needs an available proxy.
That proxy might be retrieval frequency, successful task completion after recall, verifier acceptance, user correction rate, recency under a specific workflow, or something else entirely. Each candidate can fail. Frequency can reward repetition instead of importance. Recency can erase durable facts. A verifier can reproduce the model’s blind spots. User interaction can be sparse or noisy.
The operator’s job is not to choose a sophisticated-sounding signal. It is to show that the signal separates valuable memory under the actual workload.
The Memory Policy Card
The paper does not propose a general agent-memory framework. The following card is my operator synthesis for any memory layer with a real bound.
1. Bound
What is finite, and what is its explicit ceiling?
Name the layer before discussing the policy. A KV cache, context assembly step, retrieval index, episodic store, and durable user record have different costs and failure modes.
The bound may be bytes, tokens, records, latency, retrieval fan-out, GPU memory, or a privacy retention window. “As much as possible” is not a bound.
2. Observe
Which available signal might indicate that an item will be useful again?
This is a proxy, not direct knowledge of the future. Name when it is available and what could make it misleading.
For a support agent’s episodic store, the signal might be whether a prior resolution is retrieved and contributes to a verified successful outcome. For durable user preferences, usefulness alone may be insufficient; consent, sensitivity, and expiry can dominate the retention decision.
3. Delay
How long can the system wait for evidence before committing?
Immediate decisions are cheap but information-poor. Delayed decisions can gather evidence but require provisional capacity, add latency, or complicate reconciliation.
Specify the window: five turns, one task, 24 hours, or a bounded number of events. If the system can revise the decision later, say what triggers reconsideration.
4. Evict
Which deterministic rule removes an item when the bound is reached?
A policy needs more than a score. It needs tie-breaking, protected classes, expiry behavior, and a safe fallback when the signal is missing.
For example: remove the lowest verified-reuse score after seven days, except for user-pinned records; break ties by oldest access; expire sensitive records regardless of score; fall back to least-recently-used when the verifier is unavailable.
That may not be the right policy. It is at least testable.
5. Measure
Which workload outcome will show whether the policy improved the system?
Choose a baseline and measure the whole tradeoff. Compare against recency, accumulated attention, no retention, full retention where feasible, or the current production rule.
Track task quality plus the costs the policy can move: latency, compute, memory, throughput, and retrieval noise. KV-cache implementations make these tradeoffs concrete. Offloading can reduce GPU memory while hurting throughput; quantization can reduce cache memory while adding latency.[^hf]
A policy that preserves quality but doubles latency is not automatically a win. Neither is a policy that lowers memory use while quietly damaging continuity.
Write the card before buying more capacity
Here is the compact version:
| Field | Decision to write down |
|---|---|
| Bound | The finite layer and its capacity or cost ceiling |
| Observe | The available proxy for future value, including failure modes |
| Delay | The observation window before commitment |
| Evict | The exact removal rule, exceptions, ties, and fallback |
| Measure | The baseline and workload outcomes: quality plus operating cost |
This card does not prove that a bounded system should stay small. Sometimes the correct result is that the memory budget is too low. Sometimes the simplest capacity increase beats a clever policy.
But make that a measured conclusion.
The RMM paper’s strongest lesson is not that delayed eviction wins. Its own benchmarks do not support that claim. The lesson is that a policy can look excellent in the regime that exposes its signal and ordinary in the regime that does not.
That is the standard memory systems should have to meet.
Define the bound. Name the observable proxy. Choose the decision lag. Write the eviction rule. Compare it with a baseline on the workload that matters.
Then decide whether the system needs a better policy, a larger budget, or both.
[^paper]: Maruthi Vemula and Neeraj Praneeth Gajula, “Eviction as Estimation: A Fixed-Lag Smoothing View of Test-Time Memory, and When Measuring Beats Accumulating,” arXiv v1, sections 3–9, Tables 2–3, and Figure 2, https://arxiv.org/html/2607.24667 [^hf]: Hugging Face, “Cache strategies,” including cache offloading and quantized cache, https://huggingface.co/docs/transformers/kv_cache