m@berryhill: ~/berryhill.dev/posts/the-model-answer-is-not-accepted-knowledge.md
~ homeposts/about.md
m@berryhill in ~/posts$ cat the-model-answer-is-not-accepted-knowledge.md 
---
title:  The Model Answer Is Not Accepted Knowledge
date:   2026-08-10
topic:  ai-systems
read:   14 min
words:  3,130
slug:   the-model-answer-is-not-accepted-knowledge
views:  live post
tags:   [ai-systems, knowledge-systems, ai-governance, llm-operations]
---
essay · long read

The Model Answer Is Not Accepted Knowledge

Valid JSON proves shape, not truth. Reliable AI extraction needs evidence, provenance, policy controls, and reversible claim acceptance.

by matthew berryhill
·
posted 2026-08-10
·
updated 2026-08-10
·
topic ai-systems
table of contents
  1. The operating read
  2. Valid JSON is not accepted knowledge
  3. What TRACE-CTI actually tested
  4. A six-state claim lifecycle
  5. Thresholds encode policy, not truth
  6. Make disagreement first-class
  7. Route review by consequence
  8. Build the acceptance boundary

The model returned valid JSON. The extraction step worked. But the claim is not ready to drive a search result, an agent action, or a decision.

Schema conformance proves shape, not truth. A typed record can still be unsupported, contradicted, stale, or produced by a broken run. Treating it as knowledge collapses two different events: a model made a prediction, and a system accepted a claim for use.

The most important interface in an extraction system is therefore not text-to-JSON. It is candidate claim to accepted knowledge. That interface needs evidence, provenance, explicit promotion policy, retained disagreement, and reversible decisions.

The operating read

  • A model response starts as a candidate claim, even when it matches the schema.
  • Promotion binds that claim to evidence, provenance, validation grounds, and a named policy.
  • Agreement changes the support profile; it does not automatically establish truth.
  • Accepted knowledge is eligibility for a defined use, not permanent trust.
  • Revocation removes a claim from the accepted view while preserving its history.

Valid JSON is not accepted knowledge

A common extraction-pipeline failure mode is putting the sharp technical boundary in the wrong place.

Teams validate the response against a schema, reject malformed records, and send the survivors downstream. Schema validation is valuable. It can prove that a field exists, a value has the expected type, or a record can be parsed. It cannot prove that the source supports the statement.

That missing distinction becomes dangerous when extracted records feed systems with consequences. A weak claim moves into search, analytics, agent plans, or a person’s decision. By then, the original model response may be several transformations away and difficult to contest.

A reliable system should preserve two layers:

  • Candidate claims: what an extraction run proposed, with its source and run context.
  • Accepted knowledge: what a named policy currently permits a downstream system to use.

Acceptance does not mean universal or permanent truth. It means the claim is eligible for a defined use under the active policy. That narrower definition makes the decision inspectable—and reversible.

Shape validation versus knowledge acceptance Two paths separate a schema-valid model record from an accepted claim: the first stops at parseable shape, while the second adds retrievable evidence, provenance, policy validation, and permission for a defined downstream use. Shape validation versus knowledge acceptance Schema-valid output ✓ Required fields present ✓ Expected types ✓ Parseable record Not sufficient for promotion Accepted for a defined use • Retrievable evidence • Run + transformation provenance • Supporting + conflicting evidence • Named policy and version • Recorded validation decision Interpretive guard Governed eligibility is not permanent or universal truth.
Parsing tells you whether the record fits the pipe. Promotion decides whether the claim may steer the system.

What TRACE-CTI actually tested

A useful concrete example comes from the v2 arXiv preprint “TRACE-CTI: Auditable Post-Extraction Governance of TTP Claims with Knowledge Graphs” by Federico Valletta, Giacomo Longo, Enrico Russo, and Alessio Merlo.

The authors study post-extraction governance for cyber-threat-intelligence claims mapped to MITRE ATT&CK tactics, techniques, and procedures. Their architecture keeps run-level Predictions, configuration-level GraphAssertions, cross-setup ConsensusAssertions, validation records, and a policy-derived trusted view separate. Consensus provides corroborating evidence; it does not automatically make a claim trusted (TRACE-CTI architecture).

The evaluation is specific. It uses two public English-language cyber-threat-intelligence corpora containing 65 reports and 5,303 sentences. The experimental matrix combines two retrievers with three generator families, then incrementally ingests the results into six graph versions. Those boundaries matter. This is not evidence that the same thresholds or results transfer to every domain, language, model, or production extraction system (TRACE-CTI evaluation).

The paper’s implementation uses Neo4j knowledge graphs. Its governance method is described independently of graph technology, and the authors note that enriched relational or event-sourced systems could encode similar semantics. They did not evaluate those alternative backends. A graph is one implementation, not the admission price for governed extraction (TRACE-CTI methodology and limitations).

The distinction I take from the work is broader than cyber-threat intelligence: retain the model’s observation, derive trust through policy, and never confuse the derived view with the raw evidence.

A six-state claim lifecycle

Here is the backend-neutral operating model I would put between extraction and downstream use. This six-state lifecycle is my synthesis informed by TRACE-CTI, not the paper’s own taxonomy.

The six-state claim lifecycle A six-state rail moves from Prediction through Evidence-bound assertion, Corroboration, Policy validation, and Accepted knowledge to Non-destructive revocation, with evidence, provenance, disagreement, policy version, and supersession history retained across transitions. The six-state claim lifecycle 1 Prediction 2 Evidence-bound assertion 3 Corroboration 4 Policy validation 5 Accepted knowledge 6 Non-destructive revocation restoration requires a new versioned decision Persistent evidence rail 1 Source and location 2 Run and transformation provenance 3 Support and conflict 4 Policy version and validation grounds 5 Revocation and supersession history Revocation removes eligibility; it does not erase history.
Trust is not a field the model fills in. It is a versioned state the system must be able to explain—and undo.

1. Prediction

A prediction is the raw candidate produced by a model or extraction process. Preserve it as an observation. Do not let it enter a trusted downstream view merely because it is syntactically valid or carries a high confidence score.

Promotion requires a retrievable source, the relevant source location, and enough run provenance to reconstruct where the candidate came from.

2. Evidence-bound assertion

An assertion connects the candidate statement to retrievable source evidence and complete provenance. The record should identify the source, extraction run, configuration, and transformation history rather than storing only a polished sentence.

At this state, the system can inspect the claim. It has not accepted it. Lack of corroboration means unvalidated, not false.

3. Corroboration

Corroboration records the support required by the active policy. That support might come from independent sources, repeated extraction setups, authoritative evidence, or another domain-specific rule.

The system should retain both supporting and conflicting evidence. Agreement is evidence for a decision, not a substitute for one.

4. Policy validation

Validation applies explicit acceptance rules for the intended use and consequence level. The decision records which grounds passed, which policy version ran, what disagreement existed, and whether review was required.

A claim used to populate a low-stakes discovery index may face a different policy from a claim that can trigger an automated action. The policy should make that difference visible.

5. Accepted knowledge

An accepted claim is eligible for the governed downstream view used by search, analytics, agents, or decisions. It remains linked to its evidence, provenance, validation grounds, disagreement state, and policy version.

This is authorization for use, not a declaration that the statement is permanently true.

6. Non-destructive revocation

Revocation removes a claim from the accepted view without deleting the history that once made it eligible. The original prediction, evidence, provenance, validation decision, conflicting evidence, and prior membership in the trusted view remain inspectable.

A revoked record should not be silently restored in place. Restoration or replacement requires a new versioned decision linked to the prior record through supersession.

Thresholds encode policy, not truth

TRACE-CTI shows why a support threshold cannot be reported as a free accuracy gain.

At the paper’s fixed graph state v6.0, moving from the view with at least one supporting setup to full agreement across all six setups raises macro-averaged, document-level gold-aligned precision from 25.3% to 90.6%. At the same time, recall falls from 88.2% to 16.3% (TRACE-CTI results and discussion).

All four numbers belong together. The stricter view admits fewer unsupported claims, but it also excludes far more claims that align with the gold reference. The result is a descriptive precision-recall tradeoff in this experiment—not proof that unanimity is universally best, not a causal result about model-family diversity, and not a portable production threshold (TRACE-CTI limitations).

That is why I treat thresholds as policy controls. It is also why confidence should not allocate an audit budget. The right question is not “Which number finds truth?” It is “Which error profile is acceptable for this use?”

A threat-intelligence research index, an internal suggestion queue, and an autonomous blocking action should not inherit the same threshold by accident.

Make disagreement first-class

Many systems resolve conflicts by overwriting an old value, choosing the most frequent answer, or keeping only the latest run. Those shortcuts make the cleanest record and the weakest audit trail.

Disagreement is part of the claim, not database noise. A useful claim record should preserve:

  1. the statement and its current state;
  2. retrievable evidence and source location;
  3. extraction and transformation provenance;
  4. supporting and conflicting evidence;
  5. validation grounds and the policy version;
  6. disagreement, revocation, and supersession links.

This record does not need a graph database. A relational model, document store, or event log can carry the same contract if it preserves the relationships and decision history.

The important constraint is append rather than erase. When a source, run, setup, or validation basis becomes unreliable, record the revocation, recompute affected support, and derive a new accepted view. Do not delete the observations that explain the earlier decision.

TRACE-CTI specifies this kind of non-destructive revocation architecture, but the reported experiment did not execute a revocation batch. It does not establish rollback speed, recovery quality, or live operational effectiveness. Those remain implementation and evaluation work (TRACE-CTI revocation contract and limits).

Route review by consequence

“Put a human in the loop” is not a complete control. It says nothing about which claims deserve review, what evidence the reviewer sees, or what decision they are authorized to make.

Universal manual review also turns the governance layer into a queue that eventually gets bypassed. The better operating rule is selective escalation.

Route a claim for review when one or more of these conditions apply:

  • the downstream consequence is high;
  • uncertainty crosses the active policy’s limit;
  • material evidence conflicts;
  • provenance is incomplete or degraded;
  • a source, model, or extraction run has been revoked;
  • the policy explicitly requires a person for that class of decision.
Consequence-routed promotion policy An evidence-bound claim passes through consequence, uncertainty, conflict, provenance, revocation, and human-review checks; routine cases enter automated validation, exceptions enter selective review, and both routes end in a recorded versioned decision before promotion or continued exclusion. Consequence-routed promotion policy Evidence-bound claim Route to selective review if any condition is true 1 High downstream consequence? 2 Uncertainty beyond policy limit? 3 Material evidence conflict? 4 Incomplete or degraded provenance? 5 Revoked source, model, or run? 6 Human decision required by policy? none true any true Automated policy validation with recorded grounds Selective review evidence + authority visible Versioned decision Promote for use • Remain unaccepted • Revoke or supersede
Human review is an escalation route. The policy and its recorded decision remain the control boundary.

Everything else can still move through automated validation—as long as the grounds and policy version are recorded.

This review-routing rule is my operating recommendation. TRACE-CTI permits analyst validation and provenance-aware review, but its experiment did not evaluate analyst behavior, staffing, usability, or review cost. Selective review should be tested as a system, not added as a comforting label (TRACE-CTI evaluation limits).

Build the acceptance boundary

A claim record used in production should answer a small set of hard questions without forcing an operator to reconstruct the pipeline from logs:

  • What exactly did the model predict?
  • Which evidence supports or conflicts with it?
  • Which run, configuration, and transformations produced it?
  • Which policy promoted it, and for what use?
  • Was disagreement present when the decision was made?
  • What downstream view currently includes it?
  • If trust changed, what was revoked or superseded—and why?

Then enforce the boundary with a few non-negotiable transition rules:

  • No prediction goes directly to accepted knowledge.
  • Valid schema output is never sufficient for promotion.
  • Confidence and frequency are evidence inputs, not undocumented acceptance policy.
  • Corroboration does not become trust without a recorded validation decision.
  • Conflicting evidence is retained, not silently overwritten.
  • Revoked claims return only through a new versioned decision.

This is the contract I want before extracted claims can steer another system.

The model’s answer is an input. Knowledge is a governed state: evidence-bound, policy-derived, contestable, and reversible. Once that distinction becomes explicit, the extraction pipeline stops pretending that parsing is the same thing as knowing.

Sources

#tags # ai-systems # knowledge-systems # ai-governance # llm-operations
m@berryhill in ~/posts$
$ cd ../ · back to posts/