Pass-Hat-k (pass^k)

In depth

A reliability metric that counts a task as solved only if all k independent attempts succeed, the strict counterpart to pass@k. It was introduced in the tau-bench paper and is estimated from n trials with c successes as C(c, k) / C(n, k). Because a single failure sinks the score, pass^k drops as k grows for any agent that is only right some of the time: in tau-bench, strong agents solved under 50% of tasks and fell below 25% at pass^8 in retail. It is the better metric when an agent will repeat the same kind of task in production.

Primary source: arxiv.org