Consider a hypothetical deployment review: an evaluation lead gets a benchmark result. The model name matches the deployment proposal. The endpoint, precision, tool-call parser and retry policy are missing.
Would you treat those as the same tested system?
A benchmark score should travel with a configuration record. A serving route is the path and configuration through which you access a model: the endpoint, serving software and request handling between your application and its answers. The model name alone does not identify that system.
My recommendation: before using a benchmark to choose a deployment, ask what was tested, which requests it could accept, and what happened to the attempts that failed. Those are separate questions. A useful evaluation record answers all three.
The model name is not the tested system
The arXiv preprint IB2: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifier proposes making this distinction explicit. Its system identity includes the model revision, serving route, precision, reasoning effort, observed request envelope, tools, output contract and harness.
That definition gives an evaluation lead something more useful than an argument about which model is best: a way to ask whether a result describes the configuration under consideration.
Suppose a benchmark used a tool-call parser different from the one behind your endpoint. Even if the model identifier matches, the comparison has an unresolved field. That does not prove your deployment will perform worse. It means the benchmark has not answered that deployment question on its own.
The practical distinction is between choosing what behavior to test and identifying the system whose behavior you measured. You need both. The same configuration discipline matters when a model upgrade changes an agent’s memory behavior: keeping one component’s name or stored records intact does not establish continuity of the whole system.
Give every result a configuration record
Here is the record I would request alongside a score. It extends the preprint's system-identity proposal into a deployment comparison aid; it is an operator recommendation, not a validated procurement standard.
| Field | Tested configuration | Proposed deployment |
|---|---|---|
| Model and revision | Exact identifier or unknown | Exact identifier or unknown |
| Endpoint and serving route | Endpoint, access mode, serving software/version where disclosed | Same fields |
| Precision | Reported precision or unknown | Reported precision or unknown |
| Reasoning settings | Effort, budgets and limits | Intended settings |
| Request envelope | Prompts, context, sampling, token and timeout limits | Intended request configuration |
| Tools and tool choice | Schemas, available tools, selection policy | Required tools and policy |
| Output contract | Required schema and formatting rules | Application requirements |
| Harness and parser | Versions and response parsing behavior | Versions and response parsing behavior |
| Retry and fallback policy | Allowed retries, fallback routes and accounting | Intended policies |
| Evaluation date | When this configuration was tested | Date of the deployment comparison |
Write unknown when a field is unavailable. Do not fill it from a model announcement, an assumed provider default or a different endpoint's documentation.
Unknown is a limit on interpretation, not evidence of poor performance. It tells you where to ask for disclosure or run a test on the intended route.
Establish capability coverage before scoring
First declare the request contract. Then establish whether the endpoint supports it. Only then admit it to the corresponding evaluation.
For example, a workload may require a particular tool interface and structured output. An endpoint that cannot accept that contract has a capability-coverage problem for this workload. Calling that a zero score for model ability would collapse two different questions.
The IB2 protocol distinguishes pre-admission binding from admitted execution in sections 4.2 and 4.5. That distinction is useful, but its implementation is not a guarantee: the authors disclose that the PartialCoverage branch was unreachable in the reported implementation, and that the binding gate missed strict-JSON adherence failures after extended reasoning.
Keep these outcomes separate:
| Outcome | What to report |
|---|---|
| Required capability unsupported before admission | The unsupported requirement and coverage/binding outcome; do not automatically label it zero model ability |
| Contract accepted; execution later fails | An admitted attempt, its failure category, and its treatment in each metric |
| Parseable response supplies some work products | Assertion credit under the declared rubric, alongside contract validity and failure status |
| No parseable response | The absence and its scoring treatment; the IB2 rubric assigns no assertion credit when there is nothing parseable to score |
For a deployment decision, unsupported capability can still be disqualifying. You do not need to relabel it as low intelligence to decide that an endpoint cannot serve your application.
Keep admitted failures in the result
Once an attempt has been admitted, do not let it disappear because the response was inconvenient to score.
Report admitted attempts, usable responses, failure categories and the denominator for every metric. If you report answer quality only among usable responses, label that condition and put failure-inclusive results beside it. A reader should not have to infer which attempts were excluded.
There is an important scoring nuance here. Under IB2's section 4.5 rubric, retaining failures does not mean assigning zero to every contract-invalid response. Parseable responses can receive assertion credit for work products actually supplied. Failure class itself does not add or subtract points. An absent parseable response earns zero mechanically because there is no supplied work product to credit.
In your own evaluation, define those rules before comparing results. Keep capability coverage, contract validity, usable-response rate and answer quality distinguishable. They answer different operational questions.
What the preprint can—and cannot—support
The value I take from IB2 is the proposed evaluation contract, not a vendor ranking.
Its evidence has substantial limits. The preprint's limitations and disclosure sections describe a sealed synthetic task bank and controlled tool sandbox, not a measured population of production workloads. There is no human baseline. Each configuration was run once; task-resampling intervals do not establish run-to-run variance.
As the authors disclose in the versioned preprint, access mode, evaluator harness generation and the serving tool-call parser change together in the serving-arm comparison. That comparison cannot isolate the effect of the route, engine, precision or provider. The earlier calibration runs were retrospectively audited against the finalized binding gate; only the replacement route was prospectively bound under the complete gate. Consumed reserve, retry-audit asymmetry and fragile nominal tier boundaries further constrain ranking interpretations.
All authors work for Iterate.ai. The instrument was built against Iterate systems, although no Iterate system appears in the reported cohort. The corpus and raw run evidence remain private, and one route is intentionally unnamed and under-identified. This article does not independently replicate those experiments or identify that route.
The authors also say they found no documented instance of public benchmarks misattributing route limits to model weights. The argument is therefore not that this preprint proves a market-wide reporting failure. It offers a protocol for making configuration and scoring boundaries explicit.
Ask for this record before transferring a score
Copy the configuration table above into your next evaluation review, then append this results record:
Workload and required request contract:
Capabilities supported / unsupported / unknown:
Admission decision and evidence:
Admitted attempts:
Usable responses and definition of usable:
Failure categories and counts:
Contract-valid responses:
Assertion-credit rubric and treatment of invalid responses:
Failure-inclusive metrics and denominators:
Conditional answer-quality metrics and denominators:
Retries and fallbacks included / excluded:
Evidence links and inaccessible evidence:
Differences from the proposed deployment:
Unknown fields and unresolved questions:
Matching the fields does not prove that a score transfers to your workload. It makes the comparison inspectable. Material differences are a reason to evaluate the intended configuration, not to quietly borrow the old score.
Before the next model-selection meeting, ask for this record alongside the leaderboard. You can still use the benchmark. You will be clearer about which decision it supports—and which decision still needs a test.
Source
IB2: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifier. Versioned arXiv preprint. Protocol and scoring definitions: sections 3 and 4; limitations and disclosures: sections 10 and 11. Recommendations in this article are operator analysis, not independent replication.