We tend to evaluate an AI agent as if the model contains the capability and the harness merely gives it somewhere to work.
That assumption breaks when training rollouts pass through the harness itself. The available tools, visible state, error messages, control flow, and success signal become part of the experience being optimized.
The harness is not just packaging. It shapes the learned policy.
That does not mean every agent needs reinforcement learning or that harness-native training guarantees production reliability. It means the evaluation unit is larger than the model. You are testing a model, through a harness, inside an environment—and each part can affect the result.
The operating read
- When a harness participates in training rollouts, its interfaces and feedback become part of the learned operating conditions.
- A strong score can combine task capability with familiarity with one tool surface.
- Reliability needs a transfer test: change the interface, failure mode, or environment and measure what survives.
The wrapper assumption is breaking
A deployment wrapper sits around behavior learned somewhere else. It translates requests, exposes tools, and manages execution after training is complete.
A training harness can play a different role. If the model repeatedly acts through that interface during rollouts, the interface helps determine which actions are easy to discover, which failures are legible, and which paths receive credit.
Consider a service operation available through three interfaces:
- a dedicated tool with a clear name and a narrow schema
- a generic shell command that requires the agent to construct the request
- a skill wrapper that adds instructions, validation, and recovery guidance
The underlying capability may be identical. The work required from the agent is not.
An agent trained repeatedly against the dedicated tool may become reliable at selecting it. That is useful behavior. But if the deployment harness renames the tool, changes the argument structure, or exposes the same operation through the shell, success may drop.
The model did not necessarily lose the underlying capability. The interface changed the route it had learned to use.
This is the difference between three claims that are often collapsed into one:
- Harness familiarity: the agent can operate an interface it has seen repeatedly.
- Task capability: the agent can solve the underlying problem when the route is available.
- Transferable reliability: useful behavior survives meaningful changes in the route, feedback, or environment.
My evaluation standard is stricter: I do not treat performance inside one familiar harness as evidence of transfer.
What the agent is actually learning
The learning loop does not present the model with an abstract task. It presents a model-visible trajectory shaped by the harness.
A task enters the harness. The harness constructs context and exposes actions. The model chooses an action. The environment returns state or an error. The harness decides what the model sees next. Eventually, an evaluator records an outcome and the training system updates behavior from the collected trajectory.
Every transition contains design choices.
Does the model see the full tool output or a summary? Does a failed action return a specific validation error or a generic failure? Does state persist between steps? Can the agent inspect what changed? Does the evaluator reward only terminal task success, or does it also distinguish safe recovery from accidental success?
These choices do not make the harness and the model technically identical. They do make the harness part of the learned operating conditions.
The SWE-agent paper made a related point from the interface side: the agent-computer interface can materially affect software-agent behavior and benchmark performance. Better models do not remove interface design from the system. They increase the value of getting that interface right.
The interface changes the policy
Tool schemas are not neutral labels. They change the search problem.
A narrow tool makes the intended action more obvious and constrains invalid choices. A broad shell exposes more power while forcing the agent to infer commands, syntax, permissions, and output interpretation. A verbose harness supplies useful guidance while consuming context and burying the current task under instructions.
State changes the policy too. If the harness always reconstructs a clean view after every action, the agent may never learn to detect stale state. If every failure includes a precise correction, the agent may look capable until production returns a timeout, a partial response, or no explanation at all.
Even isolation matters. A fresh sandbox per rollout reduces some contamination paths. It does not automatically prove that external services, cached artifacts, or shared dependencies are clean. The question is not whether the harness has an “isolated” label. The question is what actually resets, what persists, and what can leak across attempts.
Reward sits at the end of this chain. If the system credits terminal task success, the recorded trajectory may teach the agent which harness-specific paths correlate with that result. Sometimes those paths represent durable task skill. Sometimes they are shortcuts that disappear when the interface changes.
You cannot tell which one you trained by looking at the familiar-harness score alone.
What harness-native training demonstrates—and what it does not
A recent preprint, OpenForgeRL: Train Harness-native Agents in Any Environment, provides a useful architecture for examining this problem.
OpenForgeRL puts a proxy around model inference so real harnesses can make generation requests during reinforcement-learning rollouts. Remote containerized sandboxes run the work; the proxy records model-visible exchanges; and the system reconstructs those exchanges as trajectories with a terminal reward, typically task success. The useful change is straightforward: it trains through real harness behavior instead of a simplified imitation.
Its ClawEval results show why harness choice belongs in the evaluation. The same trained model performed differently through ReACT, ZeroClaw, OpenClaw, and Codex. In the authors' sampled behavior analysis, reinforcement learning shifted tool use away from a generic shell and toward dedicated service tools, while error recovery improved but remained the weakest analyzed capability.
The cross-harness experiment provides narrow transfer evidence. Training on ZeroClaw raised reported pass@1 on unseen OpenClaw from 11.4 to 14.7 and on unseen Codex from 12.2 to 16.8. Training across ZeroClaw, OpenClaw, and Codex produced larger gains across those three evaluated harnesses.
The boundary is as important as the gain. The evaluated setup shows limited transfer among related tool-use harnesses on the same benchmark and task family—not transfer to arbitrary schemas, permissions, failure modes, long-lived production state, or unrelated environments. These are preprint findings reported by the authors, not independent replication.
The right conclusion is not “harness-native training makes agents reliable.” It is more specific: training through a harness can improve behavior in that operating system, and deliberate interface variation can reveal whether some of that behavior survives elsewhere.
A contract for the harness you train against
Before training or evaluating an agent through a harness, I would write down six fields. I call this the Harness Training Contract. It is an operator model, not a taxonomy from either paper.
1. Tool surface
Which actions, schemas, affordances, and constraints does the agent repeatedly see?
List the tools by capability, not only by name. Record required arguments, validation behavior, permission boundaries, and any generic alternatives. Mark which details are likely to change at deployment.
If success depends on one exact schema, that dependency should be visible before the score arrives.
2. State model
What state is visible, persistent, hidden, summarized, or reconstructed between actions?
Record what the model can inspect, what the harness carries forward, and what the environment changes without telling the model. Include context truncation and summarization because a state transition the agent cannot see is still part of the operating system.
3. Error feedback
Which failures are observable, how specific is the feedback, and does training contain enough failed attempts to teach recovery?
I test recovery separately from happy-path success. Vary malformed inputs, timeouts, permission failures, partial results, and contradictory state. A model that succeeds only when the harness explains every mistake has learned a narrower capability than the score suggests.
4. Isolation
What is reset, persisted, shared, or externally mutable across rollouts?
Name the boundary: container, filesystem, account, service, cache, database, and credentials. Then verify the reset behavior. Isolation is a property to test, not a noun to add to the architecture diagram.
5. Reward boundary
What outcome receives credit, and can a harness-specific shortcut satisfy the evaluator without solving the underlying task?
Terminal success may be the correct reward. It still needs an adversarial reading. Ask whether the agent can exploit fixture behavior, stale state, permissive validation, or an evaluator blind spot. Then separate task completion from the path taken to reach it.
6. Transfer test
Which tools, schemas, harnesses, versions, errors, or environments will change, and does useful behavior survive?
Do not leave this as a final benchmark run. Design the variation before training so the team knows which details should remain stable and which should be deliberately perturbed.
At minimum, change one interface detail, one failure mode, and one environmental assumption. If performance collapses, the result is still useful: you found the edge of the learned operating conditions before production found it for you.
Reliability means transfer, not one benchmark
A capable model still depends on a route for expressing that capability. The harness supplies the route, and training can improve performance inside their combination.
None of those facts is the same as transferable reliability.
Reliability requires useful behavior to survive changes that matter: a renamed tool, a revised schema, a different harness version, weaker error feedback, stale state, partial service failure, or an environment the training setup did not perfectly reproduce.
That is why I would not ask only, “How well did the model score?”
I would ask:
- Which parts of the score came from task capability?
- Which parts came from interface familiarity?
- What changed during the transfer test?
- Which useful behaviors survived?
- Where did recovery fail?
The model is learning the task. It is learning how the harness presents the task, too.
If we train those together, we should evaluate them together—and then change the harness on purpose.