5 articles
Hugging Face's open challenge used 1,200+ participants and their coding agents to attempt 2,226 ICML 2026 papers claim by claim. 51% had claims independently verified, 23% had a falsified or contested claim, and four documented falsifications include a spotlight theorem that fails after step 224.
Microsoft's Change2Task turns merged pull requests into verified, executable coding agent tasks: 79.6% construction success across 1,130 repo changes, 29.2% more verified tasks than PR baselines, and tasks that stay current with the codebase.
A new benchmark drops five frontier coding agents into a live OpenTelemetry microservice system with real Prometheus, Jaeger, and OpenSearch telemetry. Best RCA accuracy: 25.3% on Medium, 10.0% on Hard. Even Claude Fable 5 is far from oncall-ready.
A systematic audit of SWE-bench Verified finds 68 of 500 instances (13.6%) pair a pull request with an issue it does not actually resolve, penalizing agents that correctly solve the stated problem. PAIChecker, a three-phase multi-agent checker, flags them with up to 92.12% binary accuracy.
A new 188-task benchmark for non-functional improvements finds coding agents hit 70% on functional correctness but lag humans on refactors and structural changes - the quality gap that becomes tech debt.
Showing 4 of 4 articles
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
Explore 862 topics
Browse All Topics