A Fact Label for AI Agents: Open-Source Checks on What Agents Say About Data
What happens when you build media literacy into the product
I've written twice about media literacy and AI.
In A Media Literacy Filter for AI Chatbots, I argued that AI models default to the "Voice of God," and offered a system prompt as a cognitive brake: cite sources, admit uncertainty, present a fair picture.
In Trust but Verify, I went further. Media literacy can't stay a burden on the reader. It has to be an architectural principle, built into the product around three pillars: Show Sources, Show Uncertainty, Show Humans.
This is the third piece: what happens when you actually build it.
The Problem: Agents That Write for Us
More and more of what we read is written by AI agents working on their own: the board update, the dashboard, the quarterly summary. Nobody is in the chat to push back. Here's an update an agent might write for a fundraising team:
"Q3 was a historic breakthrough for our campaign. We raised $1.25M, driven by an explosion of grassroots momentum from everyday supporters. Individual giving grew 40% over the quarter, and more than 2,000 new donors joined the movement."
The total is correct. Almost nothing else is. In the ledger, 90% of that $1.25M came from one corporate gift. Individual giving fell 30.6%. The whole quarter had 9 donors.
(The example is made up. The pattern isn't.)
In Trust but Verify I wrote that a calculator guarantees accuracy, while an LLM only guarantees plausibility. This draft is what that gap looks like in practice. The numbers come from a real system, so the story around them borrows their authority. Every sentence is plausible, and a reader has no way to check any of it.
factlabel: The Three Pillars, Built
factlabel is an open-source tool that sits between an AI agent and the people who read its work. It audits the agent's draft against the real data, blocks what doesn't hold up, and marks up the agent's own page so readers can see what was checked. Think of it as a nutrition label for AI-written content.
Each pillar from Trust but Verify became a feature.
1. Show Sources
If an LLM only guarantees plausibility, it shouldn't be the one deciding whether a number is right. So in factlabel, a model never does. Code recalculates every figure from the source records and does the comparison, reading "over 90%" as literally over 90%. Every quote has to appear word for word in the document it cites.
Every flag shows its lineage: the source records it came from, and the formula used, in plain words like sum of donations.amount where quarter = Q3. That's the traceable lineage I argued for: back to the raw data, not to the model's summary of it.
2. Show Uncertainty
The judgment calls, like "does this sentence credit the right cause?", need a model. factlabel uses one that has to show its uncertainty (more on it below). Every judgment comes with a confidence, and the interface makes the difference visible: red for problems the checks are sure of, amber for ones that need a person, and "unclear" when the evidence could go either way, rather than a guess dressed up as a verdict.
The summary shows uncertainty in the draft itself, too: whether it states its limits and whether it buries the bad news.
3. Show Humans
Every flag opens a "why this was flagged" log: each step, marked as decided by code or by the model, the exact question the model was asked, its answer and confidence, the evidence used, and the rule that applied. Anyone can follow the logic and disagree with it.
And the human stays in charge. Anything the checker isn't sure about goes to a person instead of blocking on its own. When factlabel corrects a number on the page, it shows the old value struck through next to the new one rather than swapping it silently. When an automated fix can't get a draft to pass, the loop stops and hands it to a person.
Isn't This Just AI Checking AI?
In Trust but Verify, I warned that relying blindly on "AI-on-AI" validation compounds error. factlabel uses one model to check another, and optionally a third to fix the text. So it's a fair question.
The answer is the word blindly. Three things keep factlabel from compounding errors:
- The numbers never depend on a model. Code checks them against the source.
- The checker can't ramble or be talked into agreeing. It answers fixed questions with fixed options and a confidence, so there's no persuasive essay to be swayed by.
- Nothing is hidden. Every judgment is shown to the reader, and every uncertain one goes to a person.
"Verification you can inspect is the opposite of blind trust."On AI checking AI
Why Jev Is Different
The checker is Jev, a new model from TypeSafe AI, and it's built for exactly this job. Jev isn't a chatbot. It doesn't write text. It answers typed questions about the data you give it:
- Choice: pick one option from a list you define.
- Score: place something on a scale you describe.
- Noul: how likely a yes/no statement is to be true.
It answers them all at once, each with a confidence. Every answer is one of the options I defined, and every answer says how sure it is: Show Uncertainty, built into the model itself. It's also fast and cheap enough to run on every draft. A full audit of the example above asks about 35 questions, takes around two seconds, and costs about $0.0003.
Jev can still be wrong. That's why the design never trusts it with the math, and never hides what it decided.
What the First Real Runs Taught Me
The first time I ran factlabel against the live model, it caught everything in the example. It also caught me out on three of my own questions:
- I'd mixed up spin and accuracy. My rubric for spin mentioned whether the wording went "beyond what the facts show," so plainly worded but false sentences were rated as sensational. Spin should be about word choice alone.
- One question wasn't enough for attribution. On its own, the false-cause check flagged four sentences when only one claimed a cause. Now two separate answers have to agree.
- Wrong numbers can bias the checker. When the model could see a dashboard tile's wrong value, it became unsure which metric the tile was even showing. Hiding the value took its confidence from 54% to 100%.
One lesson runs straight back to my first essay: the instructions are the product. Whether it's a system prompt or an audit question, treat the wording like code and test it against real output.
After those fixes, the misleading draft is blocked, and an honest version of the same update passes cleanly, three runs out of three. When factlabel sent the findings back to Claude to revise, the second draft passed. It named the corporate gift as the real source of the growth, stated the 30.6% decline plainly, and quoted the board memo word for word.
Try It
factlabel is open source under the Apache-2.0 license. It installs with one pip command, and the example runs in four: audit the misleading draft, annotate its dashboard, have Claude fix it, and rebuild the page. You bring your own keys: TypeSafe for Jev, and optionally Anthropic for the fix step.
It's early. The questions have been tuned on one example, and I'd love help with more: real agent drafts that went wrong, the data behind them, and honest versions to compare against.
factlabel on GitHub →The Takeaway
A prompt asks an AI to be media literate. A design principle says our products should be. factlabel is my attempt at the next step: showing the sources, showing the uncertainty, and keeping the humans in charge, on every draft, before anyone reads it.
Trust, but verify. And show your work.