The deployment passed its easy checks.
The records copied. The service started. Retrieval returned results. Nothing looked lost.
But a persistent agent does not remember by possessing records. It remembers by turning prior experience into useful behavior. Change the model underneath that process and the bytes can survive while the behavior changes.
That is why I treat a foundation-model replacement as a memory migration, not a database copy.
The acceptance question is not “Did the store move?” It is “Can the new coupled system still write, retrieve, interpret, and repair memory within the limits the product requires?”
A new controlled study gives that question useful evidence. It also shows why the answer cannot be reduced to one portability score. My operating response is a Memory Migration Test: inventory the whole memory contract, establish own-store baselines, test both writer-reader directions, rebuild embedding indexes cleanly, preserve governed repair evidence when allowed, set thresholds before looking at migration results, and keep rollback executable.
A memory store is not the memory system
A memory record sits inside a chain:
experience → writer → stored representation → retrieval and delivery → reader → behavior
For retrieval-augmented memory, an embedding model and vector space sit inside that chain too. Resource limits constrain every stage. Raw evidence may or may not remain available when a derived store fails.
Change one component and the meaning of the handoff can change even if its serialized form does not.
In “Does Your Agent's Memory Survive a Model Upgrade?”, Ankit Goyal and Jaideep Ray compare four persisted-memory formats under controlled migrations: long-context raw history (LC-RAW), retrieval-augmented generation (RAG), model-written NOTES, and a fixed-schema knowledge graph (KG-fixed).
The source is a September 4, 2026 arXiv v1 preprint under review. It is not peer-reviewed and it is not production validation. The benchmark uses 48 scripted synthetic histories with 160 randomized-code, exact-scored questions per history. It tests one similarly sized, sub-10B open-weight pair: Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct-1M. Real conversations are less tidy; larger models, subjective tasks, and longer histories may behave differently (Goyal and Ray, Sections 3.1–3.2 and 4.6).
Within those boundaries, the study exposes three migration risks that matter operationally.
Direction can dominate the result
The NOTES result moved in opposite directions.
When Llama read notes written by Qwen, accuracy improved by 9.91 percentage points relative to Llama reading its own notes. When Qwen read notes written by Llama, accuracy fell by 13.28 points relative to Qwen reading its own notes. The symmetric average was -1.47 points, and the paper's symmetric H1 hypothesis was not supported (Goyal and Ray, Table 3 and Sections 4.1–4.2).
The average hides the operational fact. One direction helped; the other hurt.
This does not show that model-written notes are generally fragile. It shows that, in this synthetic workload and one model pair, note portability depended on which model wrote and which model read. The measured percentages are properties of that configuration, not reusable loss rates for other agent systems.
A migration test therefore needs two rows:
- Old → new: hold the new reader fixed; compare the inherited old-writer store with the new reader's own-store baseline.
- New → old: hold the old reader fixed; compare the new-writer store with the old reader's own-store baseline.
The reverse direction is not academic. It tests whether a rollback can still interpret state written after cutover. If the new system writes memory the old system cannot use, “we can switch back” may be true for the binary and false for the agent.
Never average these rows into a portability pass. A strong direction cannot buy permission for a failed one.
Matching dimensions do not make embedding spaces compatible
The study also tested an embedding migration in one RAG setup. A 50/50 index mixing BAAI/bge-large-en v1.0 and v1.5 vectors reached 47.53% accuracy. Full re-embedding into v1.5 reached 54.47%. The mixed index therefore trailed full re-embedding by 6.95 percentage points, even though both models produced 1,024-dimensional vectors (Goyal and Ray, Sections 3.2 and 4.3).
The correct comparison matters: the mixed index underperformed the fully rebuilt index. It did not underperform the old index, which scored 42.57% (Goyal and Ray, Tables 2 and 4).
This was a specific 50/50 migration between two versions of one embedding family. The RAG pipeline used event-based chunks, cosine similarity, top-k=8, single-stage dense retrieval, no reranker, and no lexical or hybrid search. Its roughly 40% retrieval miss rate and measured mixed-space penalty belong to that setup, not to RAG as a category. Other embedders and retrieval designs may behave differently (Goyal and Ray, Section 4.3 and Appendix F).
The engineering implication is still sharp: dimensional equality is only shape compatibility. It says nothing about semantic coordinate compatibility.
My operating rule is stricter than the experiment itself: build a complete new index in a separately versioned embedding space, validate it, then switch routing atomically. If a gradual transition is unavoidable, keep spaces siloed and route by a known version. Do not search one blended vector space and hope relevance metrics notice the damage later.
That atomic cutover rule is my operator synthesis. The paper measures a mixed-index condition; it does not test a production cutover system.
Structure can reduce dependence, but the schema earned no universal crown
The workload-specific KG-fixed store showed near-zero writer-swap change: +0.0004 ± 0.0020 in the study (Goyal and Ray, Sections 4.1, 4.2, and 4.6). That is evidence that a stable, task-fit structure can reduce dependence on the writer model in this configuration.
It is not evidence that every knowledge graph is portable or superior.
The schema was designed for the scripted workload. KG-fixed and NOTES also differed in both representation and reader access: one exposed structured items through a task-fit access method; the other supplied model-written prose. The comparison does not isolate format alone, and the paper's H4a test did not support categorical superiority (Goyal and Ray, Sections 3.3, 4.2, and 4.6). A schema that excludes the facts your real users need can be perfectly stable and still be wrong.
The useful lesson is not “use a graph.” It is “make the contract explicit enough to test.”
Source evidence changes what can be repaired
Derived memory is compression. Once the writer omits a fact, store-only rewriting cannot reliably recover evidence that is no longer present.
The study's repair experiment makes this concrete. In one direction, Qwen reconstructed NOTES from retained raw history and brought 34 of 48 histories to the study's 90% target. Store-only repair reached 0 of 48 in each direction. In the reverse direction, neither raw-history nor store-only repair reached the target (Goyal and Ray, Section 4.4 and Appendix E).
That finding is direction- and resource-bounded. The repair model changed with the direction, so this is not a clean comparison of model capability. Llama attempts also hit output-token limits; larger allowances could change the outcome. The 90% target and the recovery counts belong to this benchmark, not to production systems generally (Goyal and Ray, Section 4.4 and Appendix E).
Still, the failure mode is familiar: if the derived store is the only copy, omission becomes destruction.
When policy permits, I want a provenance-linked source ledger: an encrypted or equivalently protected source copy, least-privilege access, explicit retention and deletion obligations, and a tested reconstruction path. The test must use the actual repair model under real context, output, cost, and time limits.
“When policy permits” is not a footnote. Raw histories create privacy, security, access, retention, and deletion duties, which the paper also identifies as a limitation of recovery through retained history (Goyal and Ray, Sections 4.4 and 4.6). If policy requires deletion or forbids retention, delete as required and state the consequence plainly: the system does not possess that recovery path. Fail closed on claims of recoverability rather than quietly retaining data “just in case.”
The governed ledger and fail-closed rule are my operator synthesis. The paper provides bounded evidence that retained history changed repair options in one direction; it does not grant a general license to retain user data.
The Memory Migration Test
The paper does not evaluate the complete contract below. This is my eight-part operator synthesis for deciding whether a model replacement has earned production authority.
1. Inventory the coupled components
Record and version all eight parts of the memory contract:
- writer model and prompt/version;
- reader model and prompt/version;
- embedding model and vector-space version;
- memory format and schema version;
- chunking, retrieval, ranking, and delivery configuration;
- raw or source evidence retained for repair;
- resource limits and budget; and
- baseline and acceptance metrics.
This inventory prevents “the model changed” from becoming a vague diagnosis. It lets you locate loss during writing, retrieval, delivery, reading, or repair.
2. Establish own-store baselines
Measure the old reader on the old writer's store and the new reader on the new writer's store. Use task-relevant, independently auditable metrics.
An inherited-store score without an own-store baseline is just a number. The baseline tells you what that reader can do when it receives memory from its native writer under the same evaluation conditions.
3. Test old → new
Hold the new reader fixed. Compare its behavior on the inherited old-writer store with its behavior on its own freshly written store.
This is the forward cutover test. It asks whether the new model can use the history the product already owns.
4. Test new → old
Hold the old reader fixed. Compare its behavior on the new writer's store with its behavior on its own store.
This is the behavioral rollback test. Run it separately. Do not let the old → new result stand in for it, and do not average the directions.
5. Rebuild embeddings as a complete versioned index
Build the new index outside the active route. Verify document counts, source coverage, version metadata, retrieval quality, and query behavior. Cut over atomically only after it passes.
If atomic replacement is impossible, maintain isolated indexes and explicit version routing. Never allow old and new vectors to become an unlabeled mixed space.
6. Keep a governed retained-source ledger
Where policy allows, preserve source evidence with provenance, strong protection, least-privilege access, retention limits, deletion enforcement, and a reconstruction test.
Where policy does not allow it, record that constraint as a real loss of repair capability. Do not promise recovery from evidence the system is prohibited from keeping.
7. Predeclare acceptance thresholds
Choose thresholds before observing migration results. Define them relative to each reader's own-store baseline and the product's risk budget.
At minimum, cover:
- answer or task quality;
- retrieval quality and miss behavior;
- evidence coverage;
- latency and cost; and
- repair success under production resource limits.
The preprint used study-specific controls, including a planned five-percentage-point action threshold, an own-store eligibility floor, calibrated target-retention rates, and a NOTES evidence-coverage minimum changed from 90% to 60% before main result collection. Those are experimental choices, not universal production defaults. Its signed Git tag was an internal analysis lock rather than a full preregistration, and the planned bootstrap analysis followed initial t-tests, although both methods produced the same decisions (Goyal and Ray, Sections 3.4–3.5 and 4.6).
Set your own gates before the migration has a chance to negotiate them. Every required metric in both directions must pass. Averages do not compensate for a broken route.
8. Preserve and exercise rollback
Keep the prior model, store, index, and route available until the migration contract passes and the observation window closes.
Halt or reverse cutover on any directional threshold breach, index-validation failure, source-integrity failure, or repair path that cannot execute under its declared limits. Preserve the failed evidence, then diagnose writing, retrieval, delivery, reading, and repair separately.
Rollback is not “redeploy the old model.” It is restoring a compatible coupled system.
Diagnose stages without pretending you proved causality
The paper's diagnostic interventions are useful because they ask where loss enters the chain. But they do not establish a complete causal account. Style rewriting was incomplete, and KG-fixed was excluded from one isolated-item intervention because injection did not reproduce its normal access method (Goyal and Ray, Section 4.6 and Appendix D).
That is the right posture for production debugging too.
A retrieval miss can look like a reader failure. A writer omission can look like a retrieval failure because the missing evidence was never available to retrieve. A resource ceiling can look like a weak repair model. Stage-level interventions narrow the search; they do not automatically isolate one cause.
Keep numerical penalties, diagnostic shares, repair counts, and costs attached to the configuration that produced them. The study is a reason to run your own migration test, not a table of constants to paste into an architecture standard.
Continuity is an earned property
A health check proves the service can answer. A record count proves records exist. Neither proves the agent retained useful continuity.
Behavioral memory survives an upgrade only when the entire contract survives: the old history remains usable by the new reader, the new state remains compatible with a real rollback, embedding spaces stay coherent, repair evidence exists only under legitimate governance, and every predeclared production gate passes.
That standard is harder than copying a database.
It is also honest enough to operate.
Source
- Goyal, Ankit, and Jaideep Ray. “Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability.” arXiv:2609.05339v1, September 4, 2026. Preprint under review.