BENCHMARKS

LLM judges flip 25 to 71 percent of their verdicts under pushback and 62 to 91 percent under a trainable persuader, model rankings reverse across token budgets, and a deliberating jury of cheap open-weight models beats frontier single judges at 8 to 15 percent of the cost. The single-judge era is over. Here is the design spec that replaces it.

A new August 2026 paper argues that coding agents improve faster when they compare attempts across tasks and lineages, not just retry one failed trajectory.

AI2 released TutorMoments, a replay-based benchmark that drops seven LLMs into real math tutoring transcripts and scores whether they scaffold when help is needed or push for rigor when the student can do more. The default finding: models over-help, and spelling out the trade-off in the prompt lifts every score but does not close the gap to a consistent human call.

Twelve frontier models sat at 60 percent on scientific coding, successors tying predecessors - a textbook saturation curve. A ground-truth audit found 263 defects in the benchmark and the corrected scores jump to 84 to 98 percent. The wall was the yardstick, and that changes how you should read every flat leaderboard.

Evidence gates, verifiable reward games, deploy-time certificates: the fixes that moved agent quality this week did not make judges better, they removed the judge. We think the LLM verdict inside the agent loop is a transitional technology, and here is the bet you can grade us on.

Since we published your-benchmark-is-lying-to-you, roughly 25 new results have landed on the eval-integrity question. The surprise: every fix that works is structural - ledgers, counterfactuals, decompositions, personas, readout discipline - and none of them asks the model to be smarter.

A wave of audits in the last two days measured the noise floor of agent benchmarks: misaligned ground truth, lenient model judges, and aggregate scalars that hide real failures. Here is what the numbers actually mean, what to trust, and how to buy agents without being played.

A new arXiv benchmark ran 6,560 sandboxed runs across Claude Code, Codex, OpenClaw, and Hermes with five LLMs. 68% of runs triggered unsafe signals, and 66% of all runs were unsafe yet still passed completion checks. Task completion does not prove an agent ran safely.

DeepSeek re-post-trained V4 Flash into an agent workhorse: Terminal Bench 82.7, DeepSWE 54.4, native Responses API, and first-party Codex support - all at $0.14/$0.28 per million tokens. What changed, what the numbers actually mean, and how to wire it up today.

Running Opus 5 through SlopCodeBench's multi-checkpoint gauntlet reveals that frontier models still degrade codebases over time. 24% strict pass rate, 5x more functions than Opus 4.8, and 93% of code lines trigger slop detectors.

Long-Horizon-Terminal-Bench tests coding agents on 46 terminal tasks that can run for 90 minutes. The takeaway is not that agents are useless. It is that evals need to measure endurance, recovery, and partial progress.

New benchmarks on 5,559 test utterances show Apple's iOS 26 SpeechAnalyzer API achieving 2.12% word error rate - beating all Whisper model sizes while running 3x faster.

Cursor shipped Composer 2.5 in May 2026 - a 1T parameter agentic coding model that matches Opus 4.7 and GPT-5.5 on benchmarks at roughly one tenth the cost. Here is everything you need to know to use it effectively.

New benchmark data shows GPT-5.5 hallucinates 86% of the time when it does not know the answer - versus 28% for the open-weights GLM-5.2. The numbers challenge the assumption that bigger models equal more reliable output.

Fable 5 is API-only at 2x GPT-5.6 Sol's price with a 15-point SWE-Bench Pro gap. Here is the decision framework for choosing between them in August 2026.

SWE-Bench has an 81% false-positive problem. FrontierCode replaces it with mergeability as the metric - and the scores are sobering for every AI coding tool on the market.

GPT-5.5 and 5.5 Pro hit the API on April 24. Here is what changes for builders: pricing, agentic tasks, tool-use, and the real benchmarks I ran the day it dropped.

Benchmarks are useful, but frontend work fails in places leaderboards barely measure. Here is how Web Dev Arena turns AI model comparison into a practical UI evaluation workflow.

Anthropic's Sonnet 4.6 narrows the gap to Opus on agentic tasks, leads computer use benchmarks, and ships with a beta million-token context window. Here's what actually changed.

xAI has launched Grok 4, claiming the title of the world's most powerful AI model. With a $300/month Super Grok tier, saturated AMI benchmarks, and a coding model on the horizon, this is xAI's bigge...