HumanEval

In depth

A code generation benchmark from OpenAI made of 164 hand-written Python programming problems, each with a function signature, a docstring, and unit tests. A model passes a problem only if the code it writes passes the hidden tests, and results are reported as pass@k: the chance that at least one of k samples is correct. It was released in 2021 alongside the original Codex paper and became the standard coding score for years. Frontier models now score near the ceiling, so most evaluators have moved on to harder agentic benchmarks like SWE-bench.

Primary source: arxiv.org