Topic
BENCHMARK
All blog posts, tools, and guides about Benchmark from Developers Digest.
5 resources - 5 posts
Blog Posts
View in blog →
The ICML 2026 Agent Reproduction Audit: 23% of Examined Papers Had Falsified or Contested Claims
Hugging Face's open challenge used 1,200+ participants and their coding agents to attempt 2,226 ICML 2026 papers claim by claim. 51% had claims independently verified, 23% had a falsified or contested claim, and four documented falsifications include a spotlight theorem that fails after step 224.
7 min read
Change2Task: The Assembly Line for Coding Agent Training Data
Microsoft's Change2Task turns merged pull requests into verified, executable coding agent tasks: 79.6% construction success across 1,130 repo changes, 29.2% more verified tasks than PR baselines, and tasks that stay current with the codebase.
6 min read
ORCA-bench: Frontier Agents Score 10% on Hard Oncall RCA
A new benchmark drops five frontier coding agents into a live OpenTelemetry microservice system with real Prometheus, Jaeger, and OpenSearch telemetry. Best RCA accuracy: 25.3% on Medium, 10.0% on Hard. Even Claude Fable 5 is far from oncall-ready.
6 min read
PAIChecker: 13.6% of SWE-bench Verified Instances Have Misaligned PR-Issue Pairs
A systematic audit of SWE-bench Verified finds 68 of 500 instances (13.6%) pair a pull request with an issue it does not actually resolve, penalizing agents that correctly solve the stated problem. PAIChecker, a three-phase multi-agent checker, flags them with up to 92.12% binary accuracy.
7 min read
SWE-NFI: The Benchmark That Catches What Coding Agents Miss
A new 188-task benchmark for non-functional improvements finds coding agents hit 70% on functional correctness but lag humans on refactors and structural changes - the quality gap that becomes tech debt.
6 min read
Keep exploring
More on Benchmark
- - Compare Tools - dive deeper across the Developers Digest knowledge base
- - All Benchmark articles in the blog archive
- - Developers Digest on YouTube - video tutorials covering Benchmark and more
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever
Explore 862 topics
Browse All Topics