10 min read
The Agent Is the Worst Witness to Its Own Run
An agent's report on its own run is unreliable, and eight studies show why. The fix: append-only traces, obligation ledgers, and records the agent cannot write.
EVALUATION
13 articles
LATEST
Approve Effects, Not Invocations
Approve effects, not invocations: why agent approval records miss harm in install hooks and skill combinations, and why pipelines need closure metrics.
•11 min read
Read Article
8 min read
Coding Agents Are Learning to Please the Grader
Handshake's DeepSWE audit found frontier coding agents reasoning about imagined graders. Stop rewarding patches that pass tests but drift from the user's spec.
Read more
11 min read
Absence Is the Failure Mode
AI scribe audits show omission is the dominant error, and LLM judges are near coin-flip at catching it. The fix: enumerate the facts, then check each one.
10 min read
The Response Looked Right. The Work Was Not Done.
AI agents often claim completion they did not earn. New benchmarks measure the gap, and evidence-carrying termination makes done a checked artifact.
7 min read
6 of 11 ASR Models Transcribe the Benchmark, Not the Audio: Benchmark Optimization in Speech, Quantified
ASR benchmark optimization, quantified: a Hume AI and Hugging Face study finds 6 of 11 speech models reproduce benchmark transcripts instead of the audio.
Read more
10 min read
The Oracle Is Agreeing With Itself
Self-improving test generation loops can grade against an oracle that agrees with itself. An audit shows the inflated gain and why independent checks count.
11 min read
The Judge Is Now a System You Design
LLM judges flip verdicts under pushback and rankings shift with token budget. The design spec for a judge system, including cheap deliberating juries.
10 min read
The Plateau Was the Instrument
A benchmark plateau can be a broken yardstick: a SciCode audit found 263 defects, and corrected scores jump to 84-98 percent. How to read flat leaderboards.
10 min read
The Judge Is Leaving the Agent Loop
The LLM judge is leaving the agent loop: evidence gates, verifiable rewards and deploy-time certificates replace verdicts with structure. Here is our bet.
11 min read
The Fix for Broken Benchmarks Is Architecture, Not Smarter Models
Agent benchmark noise is being fixed by architecture - ledgers, counterfactuals, decomposition, personas, readout discipline - not by smarter judges.
10 min read
Your Benchmark Is Lying to You
Agent benchmarks are noisier than their scores admit: audits found misaligned ground truth, lenient model judges and scalars that hide failures. What to trust.
8 min read
Deep Research Agents Need Constraint Ledgers
Deep research agents need constraint ledgers: a persistent record of constraints, evidence obligations, failed paths and open questions, drawn from AREX.
Read more
Showing 12 of 12 articles
Keep exploring Evaluation
- - Evaluation Topic Hub - tools and guides for Evaluation from the Developers Digest directory
- - Glossary - dive deeper across the Developers Digest knowledge base
- - Developers Digest on YouTube - video tutorials covering Evaluation and more
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever
Explore 842 topics
Browse All Topics