Absence Is the Failure Mode

Last updated: September 16, 2026

A clinician signed it. That is the part to sit with. Three commercial AI scribes drafted notes from 142 consultations - recorded UK primary-care and US ambulatory encounters, plus authored scenarios - a clinician reviewed and signed every one of them, and the audit found that one note in three carries a verified failure: 31.3 percent across 565 notes, with a confidence interval that does not come anywhere near zero (One note in three, arXiv:2608.31017). The failures cluster exactly where a bad note hurts: allergy and medication information, invented patient identity, history written up as examination on telephone calls that could not contain any.

And here is the sentence that reorders the whole review debate: the dominant error class was never something the note said. It was something the note left out. Omission. The encounter established a fact, and the note does not record it. Then the second fact lands on top of the first: the standard check for exactly this - a second LLM reading the note against the transcript and flagging problems - performs near coin-flip on omissions, while catching added or altered content cleanly.

We have been grading the seams of the agent pipeline since the first of August, and this is the nicest trap yet. Judges verify presence, not absence. Absence leaves no surface to flag. So the entire read-the-artifact-and-flag-problems paradigm, the one every agent company ships as "review," is structurally blind to the most common failure of the thing it reviews.

The judge reads the note, and the note is missing a fact#

The first paper builds the instrument. Five hundred single-error note pairs from audited fact sheets: 298 where a named fact is certainly absent from the note, 202 added-or-altered controls (LLM Judges Verify Presence, Not Absence, arXiv:2608.31016). They ran eight judge designs - different models, different prompts, voting, and GEPA prompt optimization - and measured paired discrimination: does the flawed note rank below its clean twin, with 0.5 meaning a coin flip?

On added or altered content, the judges do fine: 0.79 to 0.94. On omitted content, they collapse: 0.50 to 0.63. Eight designs, and none of them flags omissions reliably on single notes. Wording changes and voting and prompt optimization moved the operating point without creating usable detection. The judges were not being lazy. They were doing exactly what they were asked: read the note, find a problem in it. A missing fact is not in the note, so there is nothing to find.

This is the same story we told about completion claims on August 28 (the-response-looked-right-is-not-completion): agents narrate done while the deliverable does not exist, because nothing downstream verifies the finish. Here is the reviewer-side twin of that failure: the reviewer reads the artifact, and the artifact's most dangerous property is a property of what it does not contain. The judge was the layer we trusted to catch the story's mistakes, and the judge cannot see the missing sentence, the missing allergy, the missing medication.

And the fix is restructuring, not prompting#

The paper that names the disease also ships the cure, and the cure is boring in exactly the way our kind of cure is boring. Do not ask the judge to review the note whole. Restructure the task: enumerate the facts the transcript establishes, then check the note against each one. The per-fact pipeline does the enumeration as a first stage and flags the missing fact plus its severity at 2.7 percent false alarms. A single GEPA-evolved call performing the same enumerate-then-check in one pass detects more - 36.9 percent against 24.6 percent, p = 0.002 - at 6.2 percent false alarms and a tenth of the cost per note. A physician author validated 70 items and, where the two routes disagreed, sided with the pipeline on 10 of 10.

The mechanism note matters more than the numbers: absence is only checkable when you have a list of what should be present. That is why this is a task-design result, not a model result. The two fixes are just two implementations of the same restructuring, which is the strongest evidence that restructuring, not model choice, is the active ingredient.

One honest caveat from the authors, because it will matter in the counter-case: omissions whose fact is restated elsewhere in the note defeat both routes. If the model wrote the fact once, badly, and dropped it from the summary, the check passes. Enumeration catches facts that are absent, not facts that are misplaced.

The census, and the instrument-shaped audit#

The companion paper is the part that should make every vendor of AI-drafted artifacts uncomfortable. They audited three commercial scribes on 142 consultations: 565 notes from recorded UK primary-care and US ambulatory encounters plus authored scenarios. Twelve discovery passes proposed 13,678 candidate errors. An importance filter kept 5,898. An adversarial panel of two models from different families, each told to refute what it could, let 618 survive. Two clinicians adjudicated blind on disjoint samples: a physician author upheld 20 of 21 findings, an independent clinician not affiliated with the authors upheld 12 of 12.

Then the audit turns on its own instrument, and this is the sharpest result in either paper. A failure rate depends on the instrument as much as on the scribes. With model, evidence, and settings all fixed, the review instruction alone moves the share of candidates verified from 9.3 percent to 79.0 percent. The reviewing model family moves it too: the gentler reviewer flags 54.8 percent of notes, the stricter one 27.8 percent. Depending on the standard you pick, between 28 and 97 percent of sampled notes carry a failure.

Published audits disagree among themselves by a margin instrument differences alone can produce: omission is 54 to 86 percent of their error totals, against 23.1 percent in this instrument. Every published "our scribe is 99 percent accurate" number is a joint property of the scribe, the reviewer, the instruction, and the evidence. Change any one and the number moves.

We argued this at the top of the series (your-benchmark-is-lying-to-you, August 1), then through the fix wave (the-benchmark-fix-is-architectural; the-judge-is-now-a-system-you-design, August 14), then into the reference layer (the-oracle-agrees-with-itself, August 21). The audit results were always instrument-shaped; this is the first time the instrument dependence is pinned with the same rigor as the failure rate itself, because the authors held model, evidence, and settings fixed and moved one line of instruction.

Why this is not a clinical-notes story#

The natural objection is that scribe notes are a narrow corner: short documents, high stakes, a transcript exists to check against. Take the objection seriously, because it is half right. The transcript is the reason this was measurable - you need a ground-truth document to enumerate against, and clinical encounters have one. But the mechanism generalizes exactly where the condition holds, and the condition holds almost everywhere AI drafts artifacts:

• Omission is invisible to whole-artifact judges, whatever the domain. A PR description that omits the migration step, a changelog whose breaking change is the one paragraph nobody wrote, a summarization product that drops the deadline you care about, an incident report that never mentions the undelivered dependency. "Read it and flag problems" cannot flag any of these, because there is no surface to flag.
• The completion-contract fix we wrote about on August 28 needs enumeration to work: a typed deliverable certificate is only as good as the checklist it certifies against. ECT's authors could point at the certificate that did not exist; the missing-fact instrument is the same shape one layer down, at the level of individual facts instead of deliverables.
• The misalignment field study published this week gives the same story from the other end: across 20,574 real coding-agent sessions, 91.49 percent of visible misalignment resolutions require explicit user correction (arXiv:2605.29442). When 19 of 20 fixes come from a human, the human is doing the part the machine cannot check, and the part the machine cannot check is disproportionately the part that is not there.

So the design rule is domain-free even though the evidence is clinical: never grade an artifact whole. Enumerate what should be present, then check each element. The checklist is the verification artifact.

The counter-case, with the steel it deserves#

Three objections survive contact, and each one is worth answering in public.

First, the lab is one group, two papers, one task family. ComposoAI built both the benchmark and the census, and both instruments are new - the omission-blindness numbers have no independent replication yet. This is the correct reason to doubt the magnitude. It is not a reason to doubt the mechanism: paired discrimination of 0.50-0.63 on omissions against 0.79-0.94 on additions is a controlled within-experiment contrast, not a lab-specific absolute, and the direction is consistent with every judge artifact we have measured this quarter.

Second, the cure requires a source document. Enumerating the facts presupposes a transcript or spec to enumerate from, and most review workflows - a PR, a summary, a plan - do not have one. Accepted. But this is exactly the argument for the other fix we have been making: the deliverable list should exist before the agent starts (the-benchmark-fix-is-architectural). If there is no list of expected facts or deliverables, there is no way to verify the artifact at all, and that is a spec problem, not a judge problem. The enumerate-then-check result prices the value of having the list: it is the difference between coin-flip omission detection and 36.9 percent detection at a tenth of the cost.

Third, the false alarm rate. The single-call fix runs at 6.2 percent false alarms, and at fleet scale a 6 percent false alarm rate is a lot of hallucinated problems for a human to clear. Fair. Which is why the stripped position is not "deploy the pipeline," it is "shape your review as enumeration." The per-fact pipeline names the missing fact at 2.7 percent false alarms, and a team that wants fewer can pay for the extra stage. The design decision - facts-first, judge-second - is what transfers, not any single configuration.

The bet#

We think the whole-artifact judge is following per-loop judges and reference oracles into retirement, and the replacement is a two-stage shape: an enumeration stage that produces the list of what should be present, then a checking stage that verifies each element - with the enumeration itself becoming a first-class artifact that audits can grade.

Here is the bet, stated so you can grade us on it. By the end of 2027, verification of AI-drafted artifacts in consequence-bearing workflows - where a ground-truth source exists: the transcript, the spec, the ticket, the deliverable list - will standardize on enumerate-then-check, and "review this artifact" prompts will carry a required enumeration step. Second, published audit results will ship instrument metadata - reviewer model family, review instruction, evidence set - because a failure rate without its instrument will read like a confidence interval without its sample size.

What proves us wrong, specifically: the judge vendors keep shipping whole-artifact review as the default and the omission blind spot stays unmentioned in their eval cards; or enumeration checklists arrive and die of the same drift that killed every hand-maintained list - the enumeration itself becomes stale or gamed, and nobody audits the checklist against the source. That second failure is the one we are actually watching. A checklist that is not regenerated from the source is just a judge wearing a list's clothes.

What developers should do#

1. Never grade an AI-drafted artifact whole. If the task has a source of truth, enumerate the facts or deliverables it establishes, then check the artifact against each one. The 2.7-percent-false-alarm pipeline and the 36.9-percent single call are just two implementations; the enumeration is the fix.

2. Treat every audit number as instrument-relative. Name the reviewer family, the review instruction, and the evidence set before you quote a vendor's accuracy claim, and expect the same from products that audit their own outputs. A rate that moves from 9.3 to 79.0 percent on an instruction change is not a rate.

3. Write the deliverable list before the agent starts. The omission instrument is the exact price tag for spec work: without a list there is nothing to enumerate, and without enumeration you are back at coin-flip detection.

4. Build the absence question into review. When an AI artifact "looks right," ask what is not there: the unhandled case, the failing dependency, the fact that should have survived the compaction, the paragraph the summary dropped. The response-looked-right family of failures is mostly this.

5. If you ship a completion contract, ship the checklist with it. A certificate is only as good as the enumeration it certifies against, and the enumeration is the part that catches the missing fact instead of the well-formed lie.

The scribe that omitted the allergy was not malicious and it was not stupid. It was a system drafting fluent prose in a world where the review layer could not see that the prose was missing the most important sentence. Every layer of the agent pipeline reports the seam it can see, and this week's seam is the fact that is not there.

FAQ#

Can an LLM judge detect omissions?#

Not reliably when it reviews the artifact whole. Across eight judge designs, paired discrimination on added or altered content was 0.79 to 0.94, but on omitted content it was 0.50 to 0.63, where 0.5 is a coin flip. Prompt changes, voting and GEPA prompt optimization moved the operating point without creating usable detection, because a missing fact leaves nothing in the artifact to flag.

How accurate are AI scribe notes?#

In the audit of three commercial AI scribes, 31.3 percent of 565 clinician-signed notes carried a verified failure, and the dominant error class was omission. The authors also show the rate depends on the instrument: the review instruction alone moved the share of verified candidates from 9.3 percent to 79.0 percent, and depending on the standard, between 28 and 97 percent of sampled notes carry a failure.

What is enumerate-then-check verification?#

Instead of asking a judge to review an artifact whole, first enumerate the facts the source establishes, then check the artifact against each one. The per-fact pipeline flagged missing facts at 2.7 percent false alarms, and a single GEPA-evolved call doing the same in one pass detected 36.9 percent against 24.6 percent, at 6.2 percent false alarms and a tenth of the cost per note.

Does this apply outside clinical notes?#

The mechanism applies wherever a source of truth exists to enumerate against, such as a transcript, spec, ticket or deliverable list. A PR description that omits a migration step or a summary that drops a deadline is invisible to a whole-artifact judge for the same reason. If no list of expected facts exists, the post argues that is a spec problem, not a judge problem.

Continue Reading#

• The Response Looked Right. The Work Was Not Done. - the completion-decoupling post: agents claim done they have not earned; the completion contract needs the enumeration this piece supplies
• The Judge Is Now a System You Design - the single-judge era is over; judges are components with measured properties, and blindness to absence is the newest measured property
• Your Benchmark Is Lying to You - the original position: agent benchmark numbers carry double-digit systematic noise
• The Oracle Is Agreeing With Itself - why correlated checks collapse to self-agreement, and why the review instrument is part of the measurement
• The Benchmark Fix Is Architectural - the fix wave: ledgers, counterfactuals, readout discipline, and why scores will not be fixed by smarter models
• The Agent Is the Worst Witness to Its Own Run - the attestation layer: self-reports deceive and traces are mutable, so the checklist needs a tape owned by the harness, not the model
• Approve Effects, Not Invocations - the scope layer: even a complete, harness-owned record can miss the effect, because it is scoped to the component rather than the closure

Sources#

• LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It: arXiv:2608.31016 (2026-08-31)
• One Note in Three: A Verified Census of Three Deployed AI Scribes, and the Instrument That Counted It: arXiv:2608.31017 (2026-08-31)
• How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions: arXiv:2605.29442 (v2 2026-08-31)