Agents
SWE-bench Verified
A 500-task subset of SWE-bench that OpenAI released in August 2024 after 93 professional software developers screened the original test set.
In depth
A 500-task subset of SWE-bench that OpenAI released in August 2024 after 93 professional software developers screened the original test set. Annotators filtered out tasks with underspecified issue descriptions or unit tests that could reject valid fixes, so a failure is more likely to reflect the model than a broken task. It became the headline coding score in model launches, but because its tasks are public and Python-only, harder contamination-resistant successors like SWE-bench Pro now sit alongside it.
Primary source: openai.com
Example
It became the headline coding score in model launches, but because its tasks are public and Python-only, harder contamination-resistant successors like SWE-bench Pro now sit alongside it.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover Agents.
FAQ
Common questions
What is SWE-bench Verified?
A 500-task subset of SWE-bench that OpenAI released in August 2024 after 93 professional software developers screened the original test set.
Why does SWE-bench Verified matter for AI developers?
SWE-bench Verified sits in the Agents part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about SWE-bench Verified?
Developers Digest publishes tutorials and videos that cover Agents topics including SWE-bench Verified. Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
Agents
SWE-bench
A benchmark that evaluates AI coding agents on real-world software engineering tasks pulled from GitHub issues.
Agents
OSWorld
A benchmark that runs multimodal agents inside real virtual machines and asks them to complete open-ended computer tasks across desktop apps, file operations, and multi-app workflows.
Agents
Pass-Hat-k (pass^k)
A reliability metric that counts a task as solved only if all k independent attempts succeed, the strict counterpart to pass@k.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever