Inference
pass@k
An evaluation metric that counts a problem as solved if at least one of k sampled attempts passes, usually checked by unit tests.
In depth
An evaluation metric that counts a problem as solved if at least one of k sampled attempts passes, usually checked by unit tests. It was popularized by OpenAI's 2021 Codex paper, which introduced HumanEval and an unbiased estimator: generate n samples, count the c that pass, and compute 1 - C(n-c, k) / C(n, k). pass@1 approximates single-try accuracy, while a larger k shows what repeated sampling can reach if you have a reliable way to pick the right answer.
Primary source: arxiv.org
Example
In practice, developers reach for pass@k when they need the capability described above as part of an AI feature or workflow.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover Inference.
FAQ
Common questions
What is pass@k?
An evaluation metric that counts a problem as solved if at least one of k sampled attempts passes, usually checked by unit tests.
Why does pass@k matter for AI developers?
pass@k sits in the Inference part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about pass@k?
Developers Digest publishes tutorials and videos that cover Inference topics including pass@k. Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
Inference
Best-of-N Sampling
A technique where you generate N candidate answers to the same prompt and keep the one a scorer ranks highest.
Inference
Test-Time Compute
The computation a model spends while answering a prompt, as opposed to the compute spent training it.
Inference
Prefill
The first phase of LLM inference, where the model processes the entire input prompt in parallel and builds the KV cache before producing any output.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever