Agents
tau-bench
A benchmark from Sierra that tests agents in multi-turn customer-service conversations, where a language-model-simulated user talks to an agent that has domain API tools and a written policy to follow.
In depth
A benchmark from Sierra that tests agents in multi-turn customer-service conversations, where a language-model-simulated user talks to an agent that has domain API tools and a written policy to follow. The original domains are retail and airline, and each run is scored by comparing the database state at the end of the conversation with the annotated goal state. The paper also introduced the pass^k reliability metric. A follow-up, tau2-bench, adds a telecom domain where the user also has tools and the agent must guide them.
Primary source: arxiv.org
Example
In practice, developers reach for tau-bench when they need the capability described above as part of an AI feature or workflow.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover Agents.
FAQ
Common questions
What is tau-bench?
A benchmark from Sierra that tests agents in multi-turn customer-service conversations, where a language-model-simulated user talks to an agent that has domain API tools and a written policy to follow.
Why does tau-bench matter for AI developers?
tau-bench sits in the Agents part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about tau-bench?
Developers Digest publishes tutorials and videos that cover Agents topics including tau-bench. Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
Agents
Agent Handoffs
A multi-agent mechanism where one agent passes control of the conversation to a more specialized agent, rather than calling it as a subroutine.
Agents
Tool Chaining
A pattern where an AI agent uses the output of one tool call as the input for the next, building a multi-step pipeline of actions.
Agents
Pass-Hat-k (pass^k)
A reliability metric that counts a task as solved only if all k independent attempts succeed, the strict counterpart to pass@k.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever