Many agent evaluations report a single number: "the agent completes 80% of tasks." That number usually comes from running each task once, or from the best of several runs. For a chatbot answering questions, it might be enough. For an agent that changes bookings, issues refunds or updates records, it hides the question that matters: if a customer asks for the same thing ten times, how often does the agent get it right every time?
What τ-bench measures
Sierra's τ-bench put agents in realistic customer service domains. The agent talks to a simulated user, calls domain APIs and must follow a written policy. Success is judged objectively: does the final state of the database match the expected outcome12?
It also introduced pass^k: the probability that an agent succeeds on all k independent attempts at the same task1. That is a reliability measure, unlike pass@k, which asks whether any attempt succeeds. The authors reported that even state-of-the-art function-calling agents at the time succeeded on fewer than half the tasks, and that consistency was worse, with pass^8 below 25% in the retail domain1.
A 60% success rate on one attempt can mean a much lower chance of handling the same request correctly every time. Customers experience the second number.
When users act too
Real support often involves the customer doing things: restarting a device, checking a setting, confirming a code. τ²-bench extended the idea to a dual-control setting where both agent and user use tools to change a shared environment, and reported that performance dropped noticeably when agents moved from no-user settings to dual control34. Guiding a person through a task is a skill of its own, and it needs its own tests.
How I build agent evaluations
- Judge outcomes, not transcripts. Wherever possible, check the end state in the system of record, as τ-bench does, rather than asking a model whether the conversation looked right.
- Write policies down, then test them. Refund limits, identity checks and escalation rules become explicit test cases, including ones where the right answer is "no".
- Simulate users with variety: impatient, confused, adversarial, and users who change their minds. Include prompt injection attempts, as in prompt injection defence.
- Run every task several times and report pass^k alongside the single-run rate. Release gates should use the reliability number.
- Grow the suite from production. Every incident and escalation becomes a test case, per AI incident postmortems.
- Trace everything. Tool calls, arguments and results, so a failing run can be diagnosed; see LLM observability.
This sits on top of the general method in evaluation sets that predict production, with two additions specific to agents: state-based judging and repeated trials.
What to do with the results
Low reliability isn't always a reason to stop. It's a reason to change the design: narrower tools, deterministic code for policy checks, human approval for irreversible actions, and the containment patterns in MCP Apps and tool annotations. Often the best improvement isn't a better model but less for the model to decide.
How I can help
I design agent architectures and the evaluation suites that decide whether they're ready for production. See my AI architecture work, or analyse your challenge and bring the result to a call.
- 1τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World DomainsYao et al., Sierra (arXiv) · 2024
- 2
- 3τ²-Bench: Evaluating Conversational Agents in a Dual-Control EnvironmentBarres et al. (arXiv; ICML 2026) · 2025
- 4τ²-Bench (PMLR volume 306)Proceedings of Machine Learning Research · 2026