Vertical AI Needs Receipts, Not Better Chat

A vertical AI demo can look excellent while avoiding the hardest part of the job.

Give it a domain question. Watch it produce a fluent answer. Ask a follow-up. Let the interface make the exchange feel like expert work.

Then ask the questions an operator has to answer:

Which tools and data did it actually use?
What software environment produced the result?
Where is the artifact, not just the answer?
Can someone inspect the execution and revise it?
Can a qualified operator rerun the work?
Can the job execute where the required data and compute already live?

If the product cannot answer those questions, the chat may be useful, but it is not yet the complete professional system.

That is the boundary I would use to evaluate vertical AI: not the quality of the conversation alone, but the receipts left by the workbench around it.

The operating read

Chat quality is the demo surface, not the complete product boundary.
A professional workbench must fit the domain's real tools and preserve its execution environment.
The durable output is an artifact with lineage, not a response stranded in a transcript.
Reproducibility and compute placement deserve separate tests.
Receipts make work inspectable; they do not guarantee that the method or result is correct.

Chat is the demo surface

Chat compresses complexity. That is one of its strengths.

It gives people one place to ask, refine, and redirect. It can make an unfamiliar system approachable. It can also make an incomplete product look more operational than it is.

Professional work rarely begins and ends in a text box. A scientist may need databases, notebooks, packages, local files, clusters, and a figure that can survive review. An analyst may need governed data, executable queries, versioned models, and a workbook another person can audit. An engineer may need a repository, a runtime, tests, deployment context, and a patch that can be reproduced from the record.

The answer is only one surface of that work.

The product boundary also includes the tools used, the environment that ran them, the artifact that survived, the revisions that followed, and the compute that made execution possible.

I call that boundary a workbench.

A workbench is not just a chatbot with more connectors. It is an execution environment where the AI can participate in domain work while leaving enough structure for a human to inspect, continue, and rerun it.

That distinction matters because domain fluency is increasingly easy to demonstrate. Execution fidelity is much harder to fake.

Chat can make the answer legible. The workbench has to make the work legible.

The Vertical AI Workbench Scorecard

When I evaluate a vertical agent—whether I am considering a vendor or designing the system myself—I want five separate answers.

Inspection and revision belong together under Artifact Lineage.

1. Tool Fit

Operator question: Can the system work through the domain's actual tools, data sources, models, and existing pipelines, or must the operator manually move the work into chat?

A long connector list is not enough. The useful test is whether the product fits the working path of the domain.

Can it read from the sources people already trust? Can it use the package, instrument, model, query engine, or internal service the task requires? Can it reuse an existing Python, R, shell, or notebook workflow instead of forcing the operator to rebuild the process inside a proprietary interface?

The failure mode is swivel-chair AI: the model generates something plausible, then a person spends the rest of the job copying inputs and outputs between the chat and the systems where work actually happens.

Tool Fit is the first score because a product that cannot reach the real workflow is still operating beside the work.

2. Environment Capture

Operator question: Does the system preserve the software, dependencies, configuration, and execution context behind a result?

Code without an environment is an incomplete receipt.

A script may be visible while the language version, package versions, kernel state, configuration, or runtime assumptions remain hidden. The output exists, but the conditions that produced it have already started to disappear.

Environment Capture asks whether the product records enough execution context for another person to understand what ran. It does not require pretending that every external dependency or nondeterministic condition can be frozen perfectly. It does require more than a code block pasted below an answer.

The failure mode is a technically inspectable artifact that no one can execute again without reconstructing the machine around it.

3. Artifact Lineage

Operator question: Can an operator trace an artifact through its inputs, messages, code, commands, environment, review findings, and prior versions?

The durable unit of professional AI work should not be the final chat response. It should be the artifact plus the path that produced it.

That artifact might be a figure, analysis, notebook, report, model, design, query, or patch. Lineage connects it to the relevant execution record: what was requested, which sources and tools were used, what code ran, what changed, and which version is current.

This is narrower than claiming the artifact is correct. Lineage makes the work inspectable; it does not make every decision sound.

The failure mode is a polished result with no reliable way to distinguish executed work from narrated work.

Lineage is useful when it survives revision and gives the next qualified operator a path back to executable work.

4. Reproducibility

Operator question: Can another qualified operator rerun or validate the work rather than merely trust the final answer?

Artifact Lineage and Reproducibility are related, but they are not the same test.

Lineage reconstructs how the artifact was produced. Reproducibility tests whether the preserved code, environment, inputs, and instructions are sufficient to validate or rerun it.

A product can show an impressive history and still fail this test. Logs may be incomplete. The code may not match the commands that actually ran. The environment may be missing. An external source may have changed. A remote job may depend on state the artifact never captured.

The failure mode is a detailed story about the work that still cannot become work again.

5. Compute Placement

Operator question: Can execution happen where the relevant data, tools, and compute already live without creating an unsafe or impractical transfer boundary?

This dimension is easy to overlook when the demo task fits on a laptop.

Real vertical work may depend on a local workstation, an internal network, a GPU machine, a high-performance computing cluster, or data that should not be copied into a new environment. In those cases, compute is not a commodity hidden behind the interface. Its placement changes what the product can do and what risk the operator accepts.

A credible workbench should make the execution location and permission boundary visible. If a remote job runs under the operator's account, that is consequential: the job may be able to access what the account can access. Approval, scope, and job output matter as much as the ability to submit the workload.

The failure mode is a product that can discuss the domain but cannot safely execute where the domain work lives.

The scorecard is deliberately separable: strong lineage cannot rescue weak tool fit, and visible history cannot prove reproducibility.

One current example: Claude Science

Anthropic's Claude Science is useful here because the company presents it as an AI workbench for scientists, not as a new foundation model. It is one product example, not proof that this architecture is universal or that the product succeeds at every scientific task.

The documented product boundary maps to the five scorecard dimensions in concrete ways.

For Tool Fit, Anthropic describes scientific connectors and reusable skills, along with the ability to work with existing Python, R, and shell workflows. Its documentation says the app includes more than 60 scientific databases or preconfigured integrations. That is a product-capability claim, not evidence that every integration fits every lab.

For Environment Capture, the tools and environments documentation describes named package environments and artifact records that include the environment, language version, and installed package versions. Python and R kernels can persist during a session, and environments can be reused. That gives the operator more context than code alone, while still leaving external systems and nondeterministic inputs as separate concerns.

For Artifact Lineage, Anthropic's artifacts documentation describes persisted artifacts, versions, comparisons, and five provenance views: Messages, Code, Execution Log, Environment, and Review. It identifies the execution log as authoritative when it conflicts with the Code view. That is an important design choice. It privileges what ran over what the interface says was supposed to run.

For Reproducibility, the same artifact documentation says figures and other artifacts can carry exact code, environment information, a plain-language description, message history, and execution records intended to support validation and reproduction. Those mechanisms are meaningful. They are not an independent demonstration that every analysis will reproduce across machines, data changes, and research methods.

For Compute Placement, Anthropic's remote-compute documentation describes local execution, SSH-connected Linux workstations and high-performance computing login nodes, Slurm job submission, and local or remote GPU resources. Remote jobs require operator approval and run under the user's account on the selected host. The capability is useful precisely because the permission boundary is real.

The example shows what it looks like when a vertical product expands beyond chat. It does not settle whether the product's output is scientifically correct, whether its reviewer catches every material issue, or whether the architecture is the right fit for another domain.

Receipts are not the same as correctness

A workbench can improve inspectability without guaranteeing a good result.

Claude Science's reviewer makes that boundary visible. Anthropic's reviewer documentation says it checks claims against the approved plan, saved artifacts, recent responses, and execution record. It also says the reviewer does not rerun the analysis or decide whether the chosen method was appropriate.

That is the right distinction to preserve.

A reviewer can check whether a claim matches the recorded work. It cannot automatically establish that the work asked the right question, used the right method, or reached a valid domain conclusion.

The beta also has enterprise-control gaps. Anthropic's current limitations page says Claude Science events do not yet enter the organization's audit log, organization exports do not include device-local data, and administrators cannot yet restrict locally added connectors or remote-compute connections.

Local-first needs qualification too. Anthropic's data-handling documentation says conversation history and artifacts are stored on the member's device, while prompts and responses are processed on Anthropic's servers under its model-traffic policy. Remote-compute traffic is described as going directly to the selected destination.

Those are not reasons to dismiss the workbench model. They are reasons to evaluate the complete boundary instead of turning "auditable" or "local" into a magic word.

This is not another provenance checklist

There are adjacent questions that should stay separate.

Provenance asks where an answer or artifact came from. Post-extraction governance asks what may happen after a generated claim exists. Chain of custody asks what state, authority, and evidence crossed a handoff.

The workbench scorecard asks something earlier and more architectural: can the domain work itself be executed through appropriate tools, preserved with its environment, inspected as an artifact, rerun by a qualified operator, and placed near the necessary data and compute?

A product may need all of those layers. Collapsing them into "trust" makes the evaluation less useful.

How I would test a vertical AI product

Do not begin with a polished benchmark prompt.

Choose one real task from the target workflow. Use the actual source system, actual toolchain, and a representative execution environment. Then ask the product or internal team to carry the task through six stages:

Access the required domain sources and tools.
Execute the analysis or transformation in a visible environment.
Produce a durable artifact.
Revise the artifact after a meaningful change in requirements.
Hand the work to another qualified operator to inspect and rerun.
Show where execution occurred, which permissions it used, and what data crossed the boundary.

Score each stage against the five dimensions.

If the product produces an excellent answer but the operator has to rebuild the workflow, the Tool Fit is weak.

If the code exists but the runtime cannot be reconstructed, Environment Capture is weak.

If the final artifact cannot be connected to what actually ran, Artifact Lineage is weak.

If a second operator cannot validate or rerun the job, Reproducibility is weak.

If the work cannot execute near the required data and compute without an unacceptable transfer or permission model, Compute Placement is weak.

That evaluation will tell you more than another side-by-side chat test.

The product boundary is the strategy

Vertical AI will still need good models and good interfaces. Better chat is not irrelevant.

It is just insufficient as the definition of the product.

The harder advantage is making domain work operable: fitting the existing tools, preserving the environment, producing versioned artifacts, separating lineage from reproducibility, and placing execution where the work can actually run.

That is what turns an impressive response into something a professional can inspect, revise, defend, and continue.

Chat gets the demo started.

The workbench determines whether the work survives.