τ-Bench

τ-Bench is a testing procedure for computer programs that independently perform tasks for people. Such programs are called assistants or agents: they hold a conversation while solving a task on the side. In τ-Bench, the program plays a customer service representative, for example at an airline or an online shop. A second program plays the customer and asks questions or expresses requests. The assistant must understand what is wanted, look it up in a simulated company database, and then actually make a change, for example rebooking a flight. At the end, it is checked whether the database afterward has exactly the state it should have. The name comes from the Greek letter Tau, which stands for “tool-agent-user”, meaning tool, agent, and user.

Why customer service is the more honest stress test

Most well-known tests for AI models consist of individual questions with one correct answer. Math problems, multiple-choice questions from medical school, coding puzzles. Such tests measure knowledge and reasoning ability, but not reliability over time. A model can excel at math and still forget in the third conversation turn that the customer wanted a window seat.

This is exactly where τ-Bench comes in. A process stretches across many messages, and the customer often only reveals their information piece by piece. The assistant must ask follow-up questions when something is missing. It must also follow company rules that are given to it beforehand as text. A typical rule states: for certain fares, nothing may be canceled without the customer’s consent.

The results were sobering upon release in 2024. Even strong models correctly solved less than half of the tasks in the airline environment. An even more telling figure is a second metric: if you run the same task eight times, good models often only manage to get it right every single time in a fraction of the attempts. For companies, this is crucial, because an assistant that books incorrectly one in four times is not usable.

Simulated customer, real database

The test setup consists of three parts. First, the environment: a small database with fictitious orders, flights, and customer accounts. Second, a set of tools, meaning commands like “search flight” or “cancel order”, that the assistant is allowed to call. Third, a language model that plays the customer and is given a predefined role along with goals.

The evaluation is deliberately strict and fully automatic. It doesn’t matter whether the answer sounds polite or plausible. What is compared is the state of the database after the conversation against a stored target state. If a single field deviates, the task is considered failed. Additionally, it is checked whether the assistant told the customer all the required information.

From multiple runs, the developers calculate the metric pass^k. It indicates how often a model succeeded every single time across k attempts. A common mistake is confusing this with pass@k from code evaluation, where a single hit out of several attempts is enough. With τ-Bench, it’s the reverse: one failed attempt ruins the result, because in real customer service, nobody gets eight attempts.

τ-Bench in model announcements

Anyone reading press releases about new AI models regularly comes across τ-Bench. Anthropic, OpenAI, and other providers cite the figures when they want to show that a model doesn’t just talk, but acts reliably. Usually two numbers are given, one for the retail environment and one for the airline environment. The airline figures almost always come out worse, because more rules apply there simultaneously.

The test is practically relevant everywhere AI is built into company workflows. Booking systems, refunds, appointment scheduling, order management. There is now an extension called τ²-Bench, in which the customer themselves must also perform actions, such as restarting their phone. For you as a reader, the figure is above all a warning light: as long as the numbers remain clearly below one hundred percent, a human should still keep an eye on the end of such processes.