A system can review every AI session and still miss the objective taking shape across them.
That is the boundary problem. The control evaluates one conversation. The useful outputs are accumulated somewhere else, then combined into something no individual session had enough context to recognize.
A recent preprint, Magnet: Detecting Cross-Session AI Misuse Through Capability Accumulation, studies this specific failure mode. It is early and narrow research, not proof of a production-ready detector. But it exposes an architecture question that applies well beyond one paper:
If risk is assembled across sessions, why is the session still the unit of detection?
My operating principle is simple: the detection unit should match the capability-assembly unit.
That does not mean retaining everything a person does. I would design a tightly bounded longitudinal safety record that can identify meaningful capability accumulation—and govern that record as carefully as the intervention it supports.
The threat is a capability trajectory
The Magnet authors study harmful objectives divided into benign-looking units handled in isolated sessions. Each output may appear permissible on its own. The concern becomes visible when the outputs fill complementary roles in a larger objective and can be composed outside any one conversation.
At an architecture level, the threat model has four stages:
- Decomposition: a larger objective is divided into requests that appear benign or ambiguous independently.
- Distributed acquisition: separate sessions produce outputs that demonstrate or supply different capabilities.
- Correlation gap: session-level controls cannot see that those capabilities belong to one accumulating trajectory.
- Composition: the outputs are combined beyond the view of the individual-session control.
This is not a recipe for an attack. It is a description of where the safety boundary breaks.
The important object is not a repeated topic. It is a capability trajectory: a sequence of demonstrated capabilities whose combination changes the risk. Two conversations about the same subject may be harmless. Two outputs that fill materially complementary roles may deserve more scrutiny even when each passed its local check.
That distinction matters because a crude longitudinal control will overreact to recurrence. A useful control has to reason about composition.
What the preprint shows—and what it does not
Magnet tracks capabilities grounded in model outputs and tool-call effects. In the paper's design, an expert-authored taxonomy maps each session to a binary capability vector. Those observations are combined into a per-entity inventory, and a higher-level detector evaluates the accumulated set.
The demonstrated correlation boundary is one user ID. The research does not solve correlation across accounts, providers, platforms, or fragmented actors. It also excludes memory-enabled agents because memory would break the independent-session assumption used in the study.
The evaluation is deliberately bounded. It covers two harm domains, nine models, a non-adaptive actor, and a known, expert-authored capability taxonomy. Results vary by model and task. The taxonomy is closer to a library of known structures than a universal map of harmful capability. The paper is an arXiv v1 preprint; peer review and independent reproduction are not established.
Those limits do not erase the architecture problem. My read is that they tell us where to stop claiming evidence.
Here is the claim boundary I would carry into production discussions:
- The source supports: individually benign-looking sessions can contribute to a harmful objective that becomes visible through accumulated capabilities; the authors demonstrate a detector built around one user ID and a predefined capability taxonomy.
- A reasonable synthesis is: session-scoped detection can miss risk that appears only when complementary capabilities are considered across sessions.
- My operator recommendation is: production systems should use a bounded capability ledger rather than copy the paper's permanent, monotonic accumulation design.
- The source does not establish: production readiness, general detection accuracy, privacy safety, universal effectiveness, or solved identity correlation.
That separation is not academic caution. It prevents an interesting detector design from becoming an excuse for an unbounded surveillance system.
A bounded capability ledger
I would define a bounded capability ledger as:
A limited record of the capability signals needed to spot meaningful accumulation—kept only within an authorized identity and time boundary.
A capability signal is not a judgment about a person's character or intent. It is a reviewable indication, grounded in an output or tool effect, that the system produced a meaningful ingredient. Complementary accumulation means multiple signals fill different roles and their combination changes the risk assessment. It does not mean that a topic appeared twice.
The correlation boundary answers who or what the system is permitted to connect. In the preprint, that boundary is one user ID. In a real deployment, it must be disclosed, authorized, technically defensible, and no broader than the safety purpose requires. A platform should not quietly turn account-level safety review into cross-platform identity resolution.
The risk horizon answers how long a signal can remain relevant. It should be tied to a specific threat model, not to the storage team's ability to keep data forever.
The ledger should support a decision such as contextual review or a temporary safety intervention. It should not become a narrative of everything a person has asked, researched, or built.
That design is intentionally different from Magnet's research mechanism. The paper permanently retains newly observed components in a monotonic inventory. That makes the experiment easier to reason about. It is not the retention policy I would ship.
The governance contract is part of the detector
Longitudinal detection creates longitudinal data. My view is that governance cannot be a policy document added after the model works. It is part of whether the system is safe enough to deploy.
I would require six controls.
1. Purpose limitation
Name the exact safety decision the ledger is allowed to support. Then name the prohibited uses.
A safety ledger should not be reused for advertising, generalized profiling, employee evaluation, engagement scoring, or unrelated product decisions. If the purpose cannot be stated narrowly enough to constrain collection and use, the system is not ready.
Design question: What exact decision may this state influence, and which uses are forbidden?
2. Data minimization
Retain capability-level signals and the references needed for review, not complete conversation histories by default.
In my view, minimization does not make the retained state non-personal or automatically privacy-safe. It reduces the blast radius and forces the team to justify every retained field.
Design question: Can a reviewer understand the signal without storing the person's full behavioral history?
3. Bounded retention
Every signal needs an expiry tied to the risk horizon. Expiration should remove or de-identify the state unless a documented safety reason justifies a narrower extension.
I would reject “permanent because risk may return” as a retention policy. It is the absence of one.
Design question: When does this signal stop changing a safety decision, and how is deletion verified?
4. Access control
Limit who can read, change, export, or query the ledger. Log access and changes. Keep safety review separate from unrelated operational use.
A minimized dataset can still become a powerful profiling surface if too many systems or people can reach it.
Design question: Which roles can access the ledger, for what action, and who reviews the access trail?
5. Contextual human review
Accumulation is a signal, not conclusive proof of harmful intent. I would treat benign research, repeated legitimate work, and taxonomy errors as plausible sources of false positives.
Additional review, temporary tool limits, or an account restriction should require a person to review context, uncertainty, contrary evidence, and plausible benign explanations. The required review can scale with consequence, but it cannot disappear precisely when the consequence becomes serious.
Design question: Which actions require human review before restriction, and what evidence must the reviewer see?
6. Appeal and correction
A consequential decision built from longitudinal state needs a meaningful challenge path. Where disclosure is legally and operationally permissible, an affected person should be able to contest the decision and correct inaccurate signals.
This does not imply that every safety detail can always be disclosed. It means the system needs a real correction mechanism rather than an opaque final answer.
Design question: How can someone challenge the outcome, correct bad state, and receive a reviewed resolution?
These controls align with broader governance concerns in the NIST AI Risk Management Framework and the NIST Privacy Framework, including privacy risk, minimization, lifecycle management, access controls, auditability, human oversight, and mechanisms for review and correction. The six-part contract itself is my production recommendation, not a framework proposed by the Magnet authors or a claim that NIST endorses this exact design.
What still fails
A bounded ledger does not solve the entire threat model.
It can miss activity fragmented across accounts or systems. It depends on a capability taxonomy that may be incomplete, biased, or too coarse.
Expect it to sometimes mistake legitimate research for malicious assembly. Some cases will also require context that a minimized record cannot preserve adequately. Human review can become slow, inconsistent, or ceremonial. Correlation may be technically possible while still exceeding the authorized privacy boundary.
Those are not reasons to return to session-only detection. They are reasons to treat the control as a governed system with explicit residual risk.
I would stop a deployment if any of these conditions were true:
- The safety purpose is too broad to constrain collection and use.
- The detector requires indefinite raw-history retention.
- The correlation boundary is wider than what users were told or what the system was authorized to use.
- A material restriction cannot receive contextual review.
- Inaccurate longitudinal state cannot be corrected.
- The team cannot measure false positives, false negatives, retention behavior, and access abuse.
Either extreme is the wrong response to compositional risk: pretending each session is isolated, or retaining an unlimited behavioral dossier because more state feels safer.
The better design follows the capability trajectory while keeping the data boundary tight.
Before building a cross-session detector, ask two questions together:
What is the minimum longitudinal state needed to detect meaningful capability accumulation?
When must that state expire?
If a team can answer only the first, it has designed a detector. It has not yet designed a governable safety system.
Sources
- Natalie Isak and Matthew Dressman, Magnet: Detecting Cross-Session AI Misuse Through Capability Accumulation, arXiv v1, August 3, 2026.
- National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0).
- National Institute of Standards and Technology, NIST Privacy Framework: A Tool for Improving Privacy through Enterprise Risk Management, Version 1.0.