Evals & Safety
SWE-bench Pro
A software engineering benchmark from Scale AI with 1,865 long-horizon tasks drawn from 41 actively maintained repositories, built as a harder successor to SWE-bench.
In depth
A software engineering benchmark from Scale AI with 1,865 long-horizon tasks drawn from 41 actively maintained repositories, built as a harder successor to SWE-bench. Tasks often require multi-file patches that would take a professional engineer hours to days, and every task is human-verified. To resist training-data contamination, only 11 repositories are public, with the rest split into a held-out set and a commercial set of proprietary startup codebases.
Primary source: arxiv.org
Example
In practice, developers reach for SWE-bench Pro when they need the capability described above as part of an AI feature or workflow.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover Evals & Safety.
FAQ
Common questions
What is SWE-bench Pro?
A software engineering benchmark from Scale AI with 1,865 long-horizon tasks drawn from 41 actively maintained repositories, built as a harder successor to SWE-bench.
Why does SWE-bench Pro matter for AI developers?
SWE-bench Pro sits in the Evals & Safety part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about SWE-bench Pro?
Developers Digest publishes tutorials and videos that cover Evals & Safety topics including SWE-bench Pro. Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
Evals & Safety
SWE-Lancer
An OpenAI benchmark of over 1,400 real freelance software engineering tasks from Upwork, together worth $1 million in actual payouts.
Evals & Safety
System Card
A system card is a detailed safety and capability report that a frontier lab publishes with a model release, extending the model card idea from the model to the whole deployed system.
Evals & Safety
Benchmark Contamination
Benchmark contamination happens when a model's training data contains the questions or answers from an evaluation benchmark.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever