Why AI Governance Needs Evidence, Not Vibes
Most AI governance claims you'll read this year live somewhere between a marketing deck and a press release. This is the opening report for a publication that refuses to do that — starting with what "evidence-first" actually means when applied to a production AI system.
The vibes problem
Roughly speaking, three things tend to be called "AI governance" in 2026:
- A blog post asserting that the model is "aligned" without defining what was measured.
- A compliance checklist whose questions can be answered by any sufficiently confident PR team.
- A framework PDF that mentions "humans in the loop" twelve times and never specifies which humans.
None of these are governance. They are positioning. Governance only exists when an external party can independently verify a claim — and right now, almost no AI-governance claim can be independently verified.
If a governance claim cannot be reproduced from public artifacts, it is a marketing claim. It may still be true; it is not yet evidence.
What evidence-first governance looks like
For the rest of this publication, we'll use a working definition: a governance claim is evidence-first when an outside party can verify the claim using artifacts the claimant published. That requires at least four things:
-
A specific measurable claim. Not "the model is
safe." Something like: "Across the 412 prompts in
eval-set-v2, the model declined the harm class with a recall of 0.94 (95% CI: 0.91–0.96)." - A reproducible eval. The prompts, the rubric, the random seed, and the model version, posted to a public URL.
- A tamper-evident artifact. A signed record (HMAC, hash chain, content-addressed storage) so the claim cannot be silently edited after the fact.
- Stated limits. What this evidence does not cover, in plain language. Eval-set-v2 measured single-turn prompts; nothing in it speaks to multi-turn agent jailbreaks.
Any one of those four can be cheap to fake on its own. All four together are very hard to fake without a reproducible artifact someone could check.
What the newsletter will do with this
Every issue from here on will pick one specific governance, security, bias, or benchmark claim and audit it against those four criteria. Sometimes the claim is one we tested; sometimes it's a claim a public vendor is making and we ran the eval ourselves to check. Either way, the underlying report or benchmark dataset ships alongside the issue — there's no "trust us, we ran the numbers."
When a claim is also load-bearing in a research paper we're submitting, the paper appears on the Research Papers page with a status tag (working / in review / drafting), the PDF, and a BibTeX citation. The newsletter and the papers reference each other — readers should be able to walk from a headline claim to the data to the paper in two clicks.
What we won't do
We won't do:
- Hot takes on the latest model release without an eval to back them.
- Vendor scoring based on policy documents alone — only on tests we can re-run.
- Editorial framing dressed up as analysis. If we have an opinion we'll mark it as one.
- Sponsored deep dives. No paid placements. Ever.
If that's the kind of thing you want delivered to your inbox a couple of times a month — alongside the underlying data, methodology, and citation-ready papers — the subscribe box on the home page is the way in. The first benchmark report drops next week.