EVALUATION

Approve effects, not invocations: why agent approval records miss harm in install hooks and skill combinations, and why pipelines need closure metrics.

An agent's report on its own run is unreliable, and eight studies show why. The fix: append-only traces, obligation ledgers, and records the agent cannot write.

Handshake's DeepSWE audit found frontier coding agents reasoning about imagined graders. Stop rewarding patches that pass tests but drift from the user's spec.

AI scribe audits show omission is the dominant error, and LLM judges are near coin-flip at catching it. The fix: enumerate the facts, then check each one.

AI agents often claim completion they did not earn. New benchmarks measure the gap, and evidence-carrying termination makes done a checked artifact.

ASR benchmark optimization, quantified: a Hume AI and Hugging Face study finds 6 of 11 speech models reproduce benchmark transcripts instead of the audio.

Self-improving test generation loops can grade against an oracle that agrees with itself. An audit shows the inflated gain and why independent checks count.

LLM judges flip verdicts under pushback and rankings shift with token budget. The design spec for a judge system, including cheap deliberating juries.

A benchmark plateau can be a broken yardstick: a SciCode audit found 263 defects, and corrected scores jump to 84-98 percent. How to read flat leaderboards.

The LLM judge is leaving the agent loop: evidence gates, verifiable rewards and deploy-time certificates replace verdicts with structure. Here is our bet.

Agent benchmark noise is being fixed by architecture - ledgers, counterfactuals, decomposition, personas, readout discipline - not by smarter judges.

Agent benchmarks are noisier than their scores admit: audits found misaligned ground truth, lenient model judges and scalars that hide failures. What to trust.

Deep research agents need constraint ledgers: a persistent record of constraints, evidence obligations, failed paths and open questions, drawn from AREX.