BENCHMARK

ICML 2026 reproduction audit: coding agents checked 2,226 papers claim by claim, and 23% had a falsified or contested claim. What the challenge found.

Change2Task turns merged pull requests into verified, executable coding agent tasks that stay current with the codebase, across five task families.

ORCA-bench puts five frontier coding agents on live oncall root cause analysis with real telemetry. The best scored 10% on hard tasks, far from oncall-ready.

PAIChecker finds 13.6% of SWE-bench Verified instances pair a PR with an issue it does not resolve, penalizing agents that solve the stated problem.

SWE-NFI benchmarks coding agents on 188 behavior-preserving improvement tasks. The best hits 70% correctness, but all lag humans on structural refactors.