m@berryhill: ~/berryhill.dev/posts/model-availability-needs-a-risk-contract.md
~ homeposts/about.md
m@berryhill in ~/posts$ cat model-availability-needs-a-risk-contract.md
---
title:  Model Availability Needs a Risk Contract
date:   2026-07-30
topic:  ai-agents
read:   11 min
words:  2,483
slug:   model-availability-needs-a-risk-contract
views:  live post
tags:   [ai-agents, model-risk, reliability, llm-operations, production-ai]
---
essay · long read

Model Availability Needs a Risk Contract

A backup model is not a continuity plan. Define the substitute, thresholds, evidence, authority, and drill before production access changes.

table of contents
  1. Availability is more than uptime
  2. What the Fable 5 announcement actually says
  3. A fallback model is not yet a fallback plan
  4. The Model Availability Contract
  5. What should travel across providers
  6. Run the switch before you need it

A model can be online and still be unavailable to your system.

Anthropic’s June 30 announcement described access changing by model, organization, product, and route after U.S. export controls applied on June 12. Fable 5 was scheduled to return globally through first-party products on July 1. Mythos 5 had returned for a set of U.S. organizations after government approval on June 26, while broader partner access and routes through AWS, Google Cloud, and Microsoft Foundry were still being restored.[^1]

That sequence exposes a gap in the usual uptime model. A green status page says the service is operating. Production availability also depends on whether this organization can use this model through the required route, with the required entitlement and safeguards, for the actual workload.

A fallback model is not a fallback plan unless the evidence contract survives the switch. If your continuity plan is just primary_model = backup_model, you have a second endpoint—not operational continuity.

Availability is more than uptime

For an AI workload, availability is a layered question:

  • Is the service running?
  • Can this organization access the model in this geography?
  • Is the required first-party or cloud route enabled?
  • Does the account have the right plan or credits?
  • Will the provider accept this class of request?
  • Can the model still produce valid outputs, use the required tools, and remain inside the workload’s risk limits?

The answer can be yes at one layer and no at another. A model might return through a first-party product while a cloud channel remains unavailable. A request might reach the provider but be redirected by a safeguard. A replacement might accept the same schema and then fail the structured-output contract that the rest of the system depends on.

Availability belongs to the whole workload path, not the model name.

Availability Is a Layered DependencyA parent node labeled Production availability branches to four required layers: service health, policy and access, safety evidence, and workload substitution. Each layer names the question an operator must answer before treating a model as available.Production availabilityService healthIs the endpoint runningand responding?Policy + accessCan this organization usethis route and model?Safety evidenceDo safeguards remaininside the risk limit?Workload fitCan the workload switchwithout leaving bounds?
A green endpoint is only the first gate. Production availability exists only when every layer holds for the actual workload.

What the Fable 5 announcement actually says

Anthropic’s announcement includes two distinct developments.

First, it describes the interruption and staged restoration of access. It says the company suspended access because it could not reliably verify nationality in real time, then details different return conditions by model, user group, product surface, and cloud channel.[^1]

Second, Anthropic describes safety work. The company says an updated classifier blocks the reported safeguard bypass in more than 99% of cases and may route blocked Fable 5 requests to Opus 4.8. It also says it has started developing a consensus framework with Amazon, Microsoft, Google, and other Glasswing partners for assessing jailbreak severity. The proposed criteria include capability gain, breadth of that gain, ease of weaponization, and discoverability.[^1]

Keep those developments separate. The source provides no basis for saying the proposed severity framework governed or caused redeployment. Anthropic describes it as work in progress, not as a standard already operating across the industry.

The useful operator lesson is narrower: model access can change along several dimensions, and provider-managed routing can silently create a substitution event. Output quality, tool behavior, latency, cost, refusal behavior, traces, and audit evidence may change with it.

A fallback model is not yet a fallback plan

API compatibility is the least interesting part of substitution.

Two models can accept the same request and still behave differently where the workload is fragile. A candidate might preserve average task quality while regressing on a failure slice: the small set of high-consequence cases where errors matter most. It might produce valid prose but malformed JSON. It might call the right tool with the wrong argument shape. It might pass an evaluation performed three months ago even though the model version, system prompt, tools, or policy controls have changed since then.

This is the substitution gap: the distance between “the request ran” and “the workload remained inside its operating limits.”

The evidence contract is workload-specific. A research assistant, an internal coding agent, and a customer-facing financial workflow should not share one universal quality or safety threshold. Each has different consequential failures, latency constraints, review paths, and rollback costs.

The right question is not, “What is our second-best model?” It is, “Under what measured conditions may this model replace this dependency for this workload?”

A Fallback Model Is Not Yet a Fallback PlanThe left column shows a fallback endpoint that shares an API, relies on stale evaluation, and switches by operator guess. The right column shows evidence-backed substitution with a workload contract, versioned evidence, and explicit thresholds, authority, and rollback.API-compatible fallbackEvidence-backed substitutionSame API shapeThe request runs; workload behavior is unknown.Named workload boundaryActions, route, tools, and data controls are explicit.Old or aggregate evaluationAverages can hide consequential failure slices.Versioned workload evidencePrimary and substitute face the same risk slices.Switch by operator guessNo declared activation or reversal boundary.Bounded, authorized, reversibleThresholds, authority, drill, and rollback travel together.
Compatibility answers whether a request can run. A risk contract answers whether the workload is allowed to keep running after the model changes.

The Model Availability Contract

A useful contract has five fields. This is my synthesis, not an Anthropic or NIST template. Together, the fields form a model-availability risk contract: the conditions under which substitution is allowed, evidenced, authorized, rehearsed, and reversed.

The Model Availability ContractFive numbered checklist rows define the operating contract. Substitute names the replacement boundary. Threshold sets permission and rollback limits. Evidence records comparable versioned results. Authority assigns activation and reversal decisions. Drill rehearses and observes the complete switch.FIVE FIELDS THAT MUST TRAVEL WITH THE SWITCH1SubstituteName the workload, model version, route, tools, data controls, and exclusions.2ThresholdSet the quality floor, risk ceiling, degraded mode, error budget, and rollback trigger.3EvidenceCompare both models on the same versioned workload suite and consequential failure slices.4AuthorityAssign who can activate, approve degraded mode, override, reject, and roll back.5DrillExercise activation and rollback; record timing, gaps, hidden dependencies, and remediation.
The contract is not a model shortlist. It is the decision record that makes a substitution testable, authorized, and reversible.

1. Substitute

Name the exact replacement boundary:

  • workload and actions covered
  • candidate model and version
  • provider and route
  • prompt, tool, and configuration differences
  • data-control differences
  • unsupported capabilities

“Use another frontier model” is not a substitute definition. “Use model B through route C for read-only document classification, with tool use disabled” is closer.

2. Threshold

Define the conditions that permit, constrain, or block the switch:

  • task-quality floor
  • safety or policy ceiling
  • latency and cost bounds
  • error budget
  • degraded-mode condition
  • automatic rollback condition

Thresholds should be measured against the actual workload. Aggregate benchmark parity is not enough if one consequential slice regresses.

3. Evidence

Record what demonstrates that the substitute remains inside those thresholds:

  • representative workload suite
  • primary and substitute results
  • high-risk failure slices
  • structured-output and tool-call validity
  • traces and observable failure signals
  • model, prompt, tool, and evaluation versions
  • evidence date and expiry rule

Evidence without a version and timestamp is a memory, not a control.

4. Authority

Decide who can act before an incident:

  • who may activate the substitute
  • who may approve degraded mode
  • who must be notified or consulted
  • who can apply a manual override
  • who can reject or roll back the switch
  • where the decision is recorded

A technically viable fallback can still fail operationally if nobody knows who is allowed to use it.

5. Drill

Rehearse the switch under production-like conditions and record:

  • activation time
  • rollback time
  • observability gaps
  • hidden dependencies
  • failed workload slices
  • manual steps
  • required remediation

A configured fallback that has never been exercised is an architectural hypothesis.

NIST’s voluntary AI Risk Management Framework Playbook provides adjacent support for this operating posture. Its guidance covers contingency processes and redundancy for vital third-party AI systems. It also addresses activation thresholds, assigned authority, post-deployment monitoring, recovery, and incident-response practice.[^2] The five-field contract is my synthesis. The Playbook does not prescribe it and explicitly should not be treated as a checklist.[^3]

A filled example

Consider a hypothetical agent that reads incoming support requests, retrieves account context, and drafts a response for human approval. It cannot issue refunds or change accounts.

Its contract might say:

Substitute: Model B, pinned to a named version, may replace Model A only for retrieval and draft generation. The route must preserve regional data handling. Account-changing tools remain disabled.

Threshold: The candidate must preserve the team’s acceptance rate on a representative test set and meet the workload’s structured-output floor. Latency and cost must stay inside defined bounds. Privacy-sensitive and escalation-required cases must remain within the team’s regression ceiling. Set the numbers from the team’s measured baseline rather than a universal template.

Evidence: Run the same versioned requests against both models. Compare reviewer decisions, invalid output, retrieval citations, tool attempts, privacy failures, escalation misses, latency, and cost. Keep the failure examples, not just the averages.

Authority: The on-call operator may activate the substitute in read-only draft mode. Expanding its actions requires the service owner. Any privacy-threshold breach triggers rollback and escalation.

Drill: Once per release cycle, disable the primary route, activate Model B, process a production-like batch, inspect traces, and roll back. A passing rehearsal means the switch and rollback complete inside the target time, every required signal remains visible, and all measured results stay inside the contract’s thresholds.

That is much more useful than “Model B works with our SDK.”

What should travel across providers

Provider-specific scores do not automatically travel. Neither do safety labels, refusal categories, trace formats, or claims about how one safeguard behaves.

The portable layer should be evidence expressed in the workload’s own terms:

  • the same representative requests
  • the same consequential failure slices
  • the same output and tool contracts
  • the same business and safety thresholds
  • the same human review rubric
  • comparable latency, cost, and error observations
  • a versioned decision record

Providers remain different; a shared workload evidence layer makes the substitution decision inspectable.

Keep both kinds of evidence. Vendor evidence explains the provider’s claims. Workload evidence records what happened in your application and gives the team a defensible basis for its decision.

This boundary also improves observability. If a provider silently routes a request to another model, the model identity and route should be visible enough for you to segment results. Otherwise, a quality regression can look like random application noise.

Run the switch before you need it

Pick one production workflow. Do not start with an enterprise-wide provider strategy.

Write the five fields. Remove ambiguous model aliases. Choose a representative workload suite. Define the failure slices that could hurt users. Compare the primary and candidate. Decide who can switch and who can roll back. Then force the substitution in a controlled window.

Watch for the parts that architecture diagrams tend to hide: credentials tied to one route, prompts tuned to one model, missing telemetry, incompatible tool behavior, human approvals that cannot keep up with degraded mode, and rollback steps that exist only in someone’s head.

The drill passes only if the team can test, compare, authorize, observe, and reverse the switch within its declared limits.

Model access will keep changing for technical and nontechnical reasons. Teams need substitution to be bounded, evidenced, and reversible before the primary dependency disappears.

[^1]: Anthropic, “Redeploying Fable 5”, June 30, 2026, with July 1 update. [^2]: NIST AI Risk Management Framework Playbook, Manage and Govern, especially GOVERN 6.2 and MANAGE 2.4, 3.1, and 4.1. [^3]: NIST, AI RMF Playbook overview.

m@berryhill in ~/posts$
$ cd ../ · back to posts/