m@berryhill: ~/berryhill.dev/posts/your-ai-security-checklist-has-a-version-problem.md
~ homeposts/about.md
m@berryhill in ~/posts$ cat your-ai-security-checklist-has-a-version-problem.md 
---
title:  Your AI Security Checklist Has a Version Problem
date:   2026-08-06
topic:  ai-security
read:   9 min
words:  2,054
slug:   your-ai-security-checklist-has-a-version-problem
views:  live post
tags:   [ai-security, ai-governance, llm-security, agent-operations]
---
essay · long read

Your AI Security Checklist Has a Version Problem

A practical control for pinning AI security guidance, preserving its source, and reopening reviews when that guidance changes.

by matthew berryhill
·
posted 2026-08-06
·
updated 2026-08-06
·
topic ai-security
table of contents
  1. The operating read
  2. A live link is not a review receipt
  3. Separate three references that do different jobs
  4. The Guidance Provenance Control
  5. What a usable review receipt looks like
  6. Provenance does not prove security
  7. The operator test

“We reviewed the system against the LLM Top 10.”

Good. Which one?

That question sounds fussy until the answer governs a production decision. If the review record contains only a checklist name and a link, can another engineer reproduce what the team actually evaluated? Can the security lead tell when that evidence became stale? Can anyone distinguish the guidance used at review time from whatever the URL serves today?

A security review that names a checklist but not its version is not reproducible. It is a memory of the evidence.

The fix is not more elaborate citation style. It is a small operating control for guidance provenance: pin the release, preserve the evidence used, and decide in advance which source changes reopen the review.

The operating read

  • A checklist name without a release cannot reproduce the evidence behind a review.
  • A canonical source tells you where guidance is maintained; an immutable reference tells you what the review used.
  • Every review needs a policy for recording no impact, running a scoped retest, or reopening affected work when guidance changes.
  • Provenance makes a decision traceable. It does not prove the system is secure or compliant.

A live link is not a review receipt

Teams often treat a reachable URL as sufficient evidence. It is not.

A URL can keep working while the material around it changes. A project can move active development while preserving old pages so existing citations do not break. A maintained branch can be the right place to find the latest source while still being the wrong reference for reconstructing a decision six months later.

OWASP’s LLM guidance gives us a clean, bounded example.

The legacy OWASP project page says that page and its repository are maintained as a historical archive so existing links and citations continue to work. It also says active development has moved to the OWASP GenAI Security Project. On the same page, OWASP identifies the OWASP GenAI LLM Top 10 2026 as the current release, published August 4, 2026, and points readers to the 2026/final directory as the canonical source. The dated release page independently identifies the 2026 release and its August 4 publication date.

None of that makes the archived material invalid or useless. Historical guidance can remain valuable, and stable old links are good for the web. The operational problem is narrower: “we used the LLM Top 10” no longer identifies the evidence that governed the review.

The release name matters. The publication date matters. The source location matters. If the source location is mutable, the exact retained evidence matters too.

Separate three references that do different jobs

A reproducible review should distinguish among:

  1. A reachable page. This helps a person find the project or release.
  2. The canonical maintained source. This is where the publisher says the governing material lives now.
  3. An immutable review reference. This identifies what the team actually evaluated at a point in time.

For the 2026 release, OWASP points to the main-branch 2026/final directory as the canonical source. That makes it useful for current guidance, but insufficient as a historical receipt.

For this article’s research, I also resolved that directory to the full commit 7e144e05142b2105faa1603178a1163a2a34df24 and verified the commit-pinned directory. The pinned source contained the ten numbered release files plus supporting release material.

Those two repository links serve different jobs. The main link answers, “Where does OWASP maintain this release source?” The commit-pinned link answers, “Which repository state did this research inspect?”

Do not choose between them. Record both.

Three references, three jobsA three-row operator list separates a reachable project page used for discovery, the canonical maintained source used to follow current guidance, and an immutable review reference used to reproduce the evidence behind a decision.THREE REFERENCES, THREE JOBS1Reachable pageFind the project or named release.Job: discovery2Canonical sourceFollow current publisher guidance.Job: current guidance3Immutable review referenceRecover the exact repository stateor retained object the team evaluated.Job: reproduction
A live link helps someone find guidance. Only a pinned reference can recover the evidence that governed the decision.

The Guidance Provenance Control

Here is the practical model I recommend for guidance that shapes an AI security decision. This is my operator synthesis, not an OWASP-prescribed process.

The Guidance Provenance ControlA three-stage vertical flow turns security guidance into a reproducible operating record: pin the named release, preserve the exact evidence used, and define which future changes reopen the review.GUIDANCE PROVENANCE CONTROL01Pin the releaseName + edition + publisher dateMakes the guidance identifiable.02Preserve the evidence usedCanonical URL + immutable reference+ exact retained object metadataMakes the review reproducible.03Define the reopen ruleSet impact, scope, and responsebefore the next source change.Makes staleness actionable.IDENTIFY → REPRODUCE → REASSESS
Provenance is not clerical metadata. It is the control that connects a source version to a review decision and its expiration conditions.

1. Pin the release

Record the human-readable guidance name, the named release or edition, and the publisher’s publication date.

“LLM Top 10” is not enough. “OWASP GenAI LLM Top 10 2026, published August 4, 2026” is specific enough to investigate and reproduce.

2. Preserve the evidence used

Record the canonical maintained URL, then add an immutable reference or retain an evidence artifact where your policy allows.

For repository-based guidance, preserve the full commit SHA and commit-pinned permalink. If you retain a file or bundle, identify the exact object: filename or object ID, media type, and byte length. If you digest it, hash the exact retained bytes with a named algorithm such as SHA-256.

Do not hash a mutable URL. Do not write “source bundle” without defining which bytes belonged to it. A digest is only useful when another person can identify the object that produced it.

3. Define what changes reopen the review

Version provenance is incomplete if it can tell you the evidence changed but not what to do next.

The response should depend on impact, system scope, and policy. A practical triage model is:

  • Record no impact. A typo or formatting correction changes no risk statement, mitigation, reference, or control mapping. Preserve the comparison and rationale; do not manufacture a retest.
  • Run a scoped retest. A mitigation or mapping changes for a risk that touches one reviewed component. Remap the affected evidence and retest that scope.
  • Reopen the affected review. A named release, taxonomy, or cross-cutting requirement changes enough that the prior scope or control mapping can no longer be relied on. Reopen the relevant review, up to a full rerun when the impact is broad.

These are triage examples, not universal rules and not OWASP mandates. The important move is deciding how impact will be assessed before the next release arrives.

Guidance change triageA decision tree starts with a detected guidance change and routes it by impact to one of three responses: record no impact, run a scoped retest, or reopen the affected review.GUIDANCE CHANGE TRIAGESource change detectedCompare versions, then assess impact.What actually changed?Impact + system scope + policyNO CONTROL IMPACTRecord no impactKeep the comparison and the rationale.BOUNDED IMPACTRun a scoped retestRemap and retest only affected scope.BROAD IMPACTReopen the affected review
“The source changed” is a detection event, not an automatic full rerun. The response should be proportional, explicit, and preserved.

What a usable review receipt looks like

Here is a compact, public-safe receipt format. It is an illustrative operator schema, not an OWASP format:

guidance: "OWASP GenAI LLM Top 10"
release: "2026"
published_at: "2026-08-04"
canonical_url: "https://github.com/GenAI-Security-Project/GenAI-LLM-Top10/tree/main/2026/final"
immutable_ref:
  commit: "7e144e05142b2105faa1603178a1163a2a34df24"
  permalink: "https://github.com/GenAI-Security-Project/GenAI-LLM-Top10/tree/7e144e05142b2105faa1603178a1163a2a34df24/2026/final"
evidence_object:
  object_id: "<retained artifact ID>"
  media_type: "<media type>"
  byte_length: "<exact byte length>"
digest_algorithm: "SHA-256"
evidence_digest: "<digest of the exact retained artifact bytes>"
reviewed_at: "<review date>"
system_scope: "<components and boundaries actually reviewed>"
reopen_policy: "<change triage and affected-review response>"

The placeholders are intentional. A receipt should never invent evidence precision. Populate the object metadata and digest only after retaining and hashing the exact artifact used by the review.

The same rule applies to system_scope. “The platform” is usually not a scope. Name the components, interfaces, data paths, agent permissions, or deployment boundaries that were actually evaluated. Otherwise a future team cannot tell whether changed guidance touches the prior decision.

Provenance does not prove security

This control has a hard boundary.

It can make the governing evidence reproducible. It can make staleness detectable. It can give a team a disciplined way to decide whether a source change needs a note, a scoped retest, or a reopened review.

It cannot prove that the original review was complete. It cannot prove that the system is secure. It cannot turn guidance into compliance evidence by itself. And it cannot compensate for a vague system scope or weak technical testing.

That is not a reason to skip provenance. It is a reason to describe the control accurately.

Security guidance is an input to judgment. Version provenance lets you identify the input instead of pretending the guidance was timeless.

The operator test

The next time a review says it used an industry checklist, ask two questions:

  1. Can the team reproduce exactly what governed the decision?
  2. Can the team detect when that evidence became stale and explain what happens next?

If the answer to either question is no, the review record is not finished.

Do not save only the checklist link.

Save the release. Save the source. Save the conditions that make the decision expire.

#tags # ai-security # ai-governance # llm-security # agent-operations
m@berryhill in ~/posts$
$ cd ../ · back to posts/