Benchmark leaderboard - 43 entries on 2 boards
DeepSWE, NL2Repo and ProgramBench leaderboard
DeepSWE measures frontier coding agents on original, long-horizon software engineering tasks, and ProgramBench gives agents only a compiled binary and its documentation and asks them to build a complete codebase that reproduces the program's behavior. As of October 17, 2026, gpt-6-astra ranks first on the DeepSWE v1.1 board with 74%±3% and Claude Opus 5 (xhigh) ranks first on the ProgramBench board with 4.5% resolved (Almost: 37.0%).
Last verified from deepswe.datacurve.ai, programbench.com. Read the caveats
This page covers 2 separate boards: DeepSWE v1.1 and ProgramBench. Each is scored on its own terms, so compare rows only within the same board.
DeepSWE v1.1
21 entries, in the board's order
| 1 | gpt-6-astra mini-swe-agent, xhigh effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $4.43/task Official leaderboard | 74%±3% |
| 2 | gemini-3.8-flash mini-swe-agent, high effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $2.36/task Official leaderboard | 74%±1% |
| 3 | claude-opus-5 mini-swe-agent, max effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $11.84/task Official leaderboard | 74%±4% |
| 4 | gpt-5.6-sol mini-swe-agent, max effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $6.46/task Official leaderboard | 73%±3% |
| 5 | claude-fable-5 mini-swe-agent, xhigh effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $13.41/task Official leaderboard | 70%±3% |
| 6 | glm-5.3 mini-swe-agent, max effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $3.99/task Official leaderboard | 69%±3% |
| 7 | kimi-k3 mini-swe-agent, max effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $4.65/task Official leaderboard | 69%±5% |
| 8 | grok-4.6 mini-swe-agent, medium effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $3.45/task Official leaderboard | 67%±2% |
| 9 | gpt-5.6-luna mini-swe-agent, max effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $0.61/task Official leaderboard | 67%±4% |
| 10 | gpt-5.5 mini-swe-agent, xhigh effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $7.23/task Official leaderboard | 67%±6% |
| 11 | gemini-3.7-flash mini-swe-agent, medium effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $2.03/task Official leaderboard | 65%±3% |
| 12 | glm-5.3-flash mini-swe-agent, max effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $0.24/task Official leaderboard | 63%±4% |
| 13 | deepseek-v4-pro mini-swe-agent, max effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $1.67/task Official leaderboard | 63%±6% |
| 14 | claude-opus-4.8 mini-swe-agent, max effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $13.22/task Official leaderboard | 59%±2% |
| 15 | qwen3.8-max mini-swe-agent, xhigh effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $3.73/task Official leaderboard | 57%±3% |
| 16 | muse-spark-1.2 mini-swe-agent, xhigh effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $3.70/task Official leaderboard | 55%±2% |
| 17 | claude-sonnet-5 mini-swe-agent, max effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $26.40/task Official leaderboard | 54%±4% |
| 18 | deepseek-v4-flash mini-swe-agent, max effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $0.46/task Official leaderboard | 53%±4% |
| 19 | gemini-3.6-flash mini-swe-agent, high effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $2.21/task Official leaderboard | 47%±4% |
| 20 | glm-5.2 mini-swe-agent, max effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $3.92/task Official leaderboard | 44%±2% |
| 21 | gemini-3.5-flash mini-swe-agent, high effort, Pass@1, v1.1 (113 tasks), best effort level shown; avg cost $3.45/task Official leaderboard | 36%±4% |
ProgramBench
22 entries, in the board's order
| 1 | Claude Opus 5 (xhigh) mini-SWE-agent, 200 tasks, rank 1; avg cost $50.53/task Official leaderboard | 4.5% resolved (Almost: 37.0%) |
| 2 | Muse Spark 1.3 (max) mini-SWE-agent, 200 tasks, rank 2; avg cost $6.46/task Official leaderboard | 2.5% resolved (Almost: 25.0%) |
| 3 | Muse Spark 1.3 (xhigh) mini-SWE-agent, 200 tasks, rank 3; avg cost $2.02/task Official leaderboard | 1.0% resolved (Almost: 16.5%) |
| 4 | GPT-5.6 Sol (xhigh) mini-SWE-agent, 200 tasks, rank 4; avg cost $6.08/task Official leaderboard | 1.0% resolved (Almost: 15.5%) |
| 5 | GPT 5.5 (xhigh) mini-SWE-agent, 200 tasks, rank 5; avg cost $8.85/task Official leaderboard | 0.5% resolved (Almost: 13.5%) |
| 6 | GPT 5.5 (high) mini-SWE-agent, 200 tasks, rank 6; avg cost $3.65/task Official leaderboard | 0.5% resolved (Almost: 5.0%) |
| 7 | Gemini 3.6 Flash mini-SWE-agent, 200 tasks, rank 7; avg cost $4.83/task Official leaderboard | 0.5% resolved (Almost: 4.0%) |
| 8 | Claude Opus 4.8 (xhigh) mini-SWE-agent, 200 tasks, rank 9; avg cost $21.02/task Official leaderboard | 0% resolved (Almost: 16.5%) |
| 9 | GLM-5.2 mini-SWE-agent, 200 tasks, rank 10; avg cost $25.36/task Official leaderboard | 0% resolved (Almost: 8.5%) |
| 10 | Gemini 3.7 Flash mini-SWE-agent, 200 tasks, rank 11; avg cost $2.04/task Official leaderboard | 0% resolved (Almost: 5.5%) |
| 11 | Muse Spark 1.2 (xhigh) mini-SWE-agent, 200 tasks, rank 12; avg cost $1.48/task Official leaderboard | 0% resolved (Almost: 4.5%) |
| 12 | Claude Opus 4.7 (xhigh) mini-SWE-agent, 200 tasks, rank 13; avg cost $10.96/task Official leaderboard | 0% resolved (Almost: 4.5%) |
| 13 | Muse Spark 1.1 (xhigh) mini-SWE-agent, 200 tasks, rank 14; avg cost $0.73/task Official leaderboard | 0% resolved (Almost: 4.0%) |
| 14 | Gemini 3.5 Flash mini-SWE-agent, 200 tasks, rank 15; avg cost $5.60/task Official leaderboard | 0% resolved (Almost: 3.0%) |
| 15 | Claude Opus 4.6 mini-SWE-agent, 200 tasks, rank 17; avg cost $11.38/task Official leaderboard | 0% resolved (Almost: 2.5%) |
| 16 | Claude Sonnet 4.6 mini-SWE-agent, 200 tasks, rank 19; avg cost $26.73/task Official leaderboard | 0% resolved (Almost: 1.0%) |
| 17 | GPT 5.4 mini-SWE-agent, 200 tasks, rank 20; avg cost $0.33/task Official leaderboard | 0% resolved (Almost: 0.0%) |
| 18 | Gemini 3.1 Pro mini-SWE-agent, 200 tasks, rank 21; avg cost $1.51/task Official leaderboard | 0% resolved (Almost: 0.0%) |
| 19 | Gemini 3 Flash mini-SWE-agent, 200 tasks, rank 22; avg cost $0.30/task Official leaderboard | 0% resolved (Almost: 0.0%) |
| 20 | Claude Haiku 4.5 mini-SWE-agent, 200 tasks, rank 23; avg cost $0.80/task Official leaderboard | 0% resolved (Almost: 0.0%) |
| 21 | GPT 5.4 mini mini-SWE-agent, 200 tasks, rank 24; avg cost $0.04/task Official leaderboard | 0% resolved (Almost: 0.0%) |
| 22 | GPT 5 mini mini-SWE-agent, 200 tasks, rank 25; avg cost $0.03/task Official leaderboard | 0% resolved (Almost: 0.0%) |
About this board
- NL2Repo / NL2Repo-Bench: no official live leaderboard found. The official repo (github.com/multimodal-art-projection/NL2RepoBench) and paper (arxiv.org/abs/2512.12730) only report results for older models (Claude Sonnet 4.5, GPT-5, etc.), which do not count as current. Current numbers exist only on aggregators (benchlm.ai, llm-stats.com), which are excluded as sources. Vendor model cards (DeepSeek, Z.AI GLM, Qwen, MiniMax, Tencent Hy) would have to be checked one by one to build a vendor-reported table; that was not done here.
- ProgramBench: only the top 25 of the official leaderboard (programbench.com, updated Sep. 28, 2026, 200 tasks) were recorded; the rows behind 'Load more' were not read. Each score is Resolved %, with Almost (95% or more of tests passed) shown in brackets.
FAQ
What does DeepSWE, NL2Repo and ProgramBench measure?
DeepSWE measures frontier coding agents on original, long-horizon software engineering tasks, and ProgramBench gives agents only a compiled binary and its documentation and asks them to build a complete codebase that reproduces the program's behavior. This page covers 2 separate boards: DeepSWE v1.1 and ProgramBench. Each is scored on its own terms, so compare rows only within the same board.
Which model leads DeepSWE, NL2Repo and ProgramBench?
As of October 17, 2026, gpt-6-astra ranks first on the DeepSWE v1.1 board with 74%±3% and Claude Opus 5 (xhigh) ranks first on the ProgramBench board with 4.5% resolved (Almost: 37.0%). That is the board's own order, not a Developers Digest ranking.
Where do these DeepSWE, NL2Repo and ProgramBench scores come from?
All 43 entries come from the official leaderboards at deepswe.datacurve.ai and programbench.com. Every entry links to its source, which was read on October 17, 2026, and lists the setting it was run under.
Related reading
- Coding Agents Are Learning to Please the GraderHandshake's DeepSWE audit found frontier coding agents reasoning about imagined graders. Stop rewarding patches that pass tests but drift from the user's spec.
- DeepSeek V4.1-Flash-1017: Benchmarks vs Opus 5, Pricing, Switch?DeepSeek V4.1-Flash-1017 jumps 11 points on Terminal-Bench 4.0 at the same deepseek-flash price. Benchmarks vs Opus 5, K3, GLM-5.3 and when to switch.
- Muse Spark Open Weights: 816B Model, Benchmarks, and How to Run ItMeta released Muse Spark Open weights on October 15, 2026: 816B total, 42B active, 1M context. Run recipes, hardware floor, license catch, and where it trails.
- Agent Harness Lessons From a 500B-Token Decompile: Give Agents a PASS/FAIL OracleAgent harness lessons from a 500B+ token decompile: a byte-matching PASS/FAIL oracle beat the reviewer agent. How agents cheated it and what to copy.
- Claude Advisor Tool vs a Bigger Model: When the Pairing PaysAnthropic's own runs: a Fable 5.1 advisor added 1.7 points to Opus 5.5 for 2.1x the cost. When the Claude advisor tool pays, and when to raise effort instead.
- Step 5 Preview: Pricing, Benchmarks and the October 15 Open WeightsStep 5 Preview is StepFun's 600B MoE with 27B active and a 1M window. What it scores, what a session costs, and what to check before the October 15 weights.
Sources
Last verified . Each row links to the page its score was copied from.
Newsletter
Get new benchmark results as they land
One email per week. New model releases, the leaderboard moves that matter and what they mean for what you build.