Direct answer: before an AI agent is allowed to change bookings, issue refunds or answer customers for a UK tour operator, hotel group or airline, it should pass a test set you own — typically 20 to 50 realistic tasks to start, each written against your own booking rules, each with a known correct end state, and each run several times rather than once. Score whether the booking record ended up right, not whether the reply sounded right. The best public evidence for why comes from the airline domain of τ-bench, a customer-service agent benchmark its maintainer Sierra describes as the de facto standard (Sierra): when it launched, the strongest model it tested completed about 35% of airline tasks, and consistency fell sharply when the same task was repeated (Yao et al., 2024). Less than two years later, its maintainers found that 27 of the 50 airline tasks had themselves needed fixing (τ-bench, February 2026). Your test set will have the same two problems: the agent will be less consistent than one demo suggests, and some of your expected answers will be wrong.
Kaize is a UK AI product studio for travel and hospitality. This article is based on the published τ-bench, τ²-bench and τ³-bench papers and posts from Sierra (the company that maintains the benchmark), evaluation guidance published by Anthropic and OpenAI, OpenAI’s deprecations page and the UK AI Security Institute’s Inspect documentation, all read on 10 October 2026. Sierra, Anthropic and OpenAI are AI vendors; we treat their material as primary sources for what they published, not as neutral verdicts. This is not a case study: we report no client results, and every threshold below is labelled as our recommendation.
Why this question is live in October 2026
Three things make the test set more urgent than it was a year ago. First, the models under travel agents keep changing underneath them. OpenAI’s deprecations page lists dated shutdowns for older model snapshots — for example, a batch of legacy snapshots on 23 October 2026 and the original GPT-5 snapshots on 11 December 2026 (OpenAI deprecations). Every forced model swap is a regression risk on a live booking workflow, and the only cheap way to see it is to rerun a fixed set of tasks.
Second, some teams built their tests inside a vendor dashboard that is going away. OpenAI says existing evals on its Evals platform become read-only on 31 October 2026 and the Evals dashboard and API are scheduled to shut down on 30 November 2026; Agent Builder is scheduled to shut down on the same date (OpenAI deprecations). If your travel agent’s test cases live only there, export them now.
Third, the benchmark everyone quotes for airline customer service was itself corrected. In February 2026 Sierra released τ³-bench with fixes to 27 airline and 26 retail tasks, plus new voice and knowledge-base domains (τ-bench; Sierra, 13 February 2026). The corrections are the most useful part for an operator, because they show exactly how a well-resourced test set goes wrong.
What the airline benchmark actually tests
τ-bench puts an agent in a simulated airline with 500 users, 300 flights and 2,000 reservations, a set of booking tools, and a written policy covering things a UK operations team would recognise: which fares can be changed, when a cancellation is allowed, how many free checked bags each tier gets, when compensation may be offered and when the agent must hand over to a human. A second model plays the customer. At the end of the conversation the benchmark compares the reservation database with the expected end state and checks the agent told the customer the information it had to (Yao et al., 2024).
Two design choices matter more than the headline scores. The grade is the state of the booking, not the tone of the chat. And the benchmark introduced pass^k: the chance that the agent gets the same task right in all of k separate attempts, rather than at least once (Yao et al., 2024). Anthropic’s 2026 guidance explains the difference plainly: an agent with a 75% per-attempt success rate passes three attempts in a row only about 42% of the time, and pass^k is the metric that matters for customer-facing agents where users expect reliable behaviour every time (Anthropic, 9 January 2026).
Six lessons a UK travel test set should copy
1. Score the booking record, not the reply
Anthropic’s definitions put it in travel terms: a flight-booking agent might say “your flight has been booked”, but the outcome is whether a reservation exists in the database (Anthropic). For a tour operator that means checking the reservation, the supplier request, the payment or refund record and the customer note after each test — not reading the message and nodding. The original τ-bench paper also warns that a correct final database is necessary but not sufficient: an agent can reach the right state while skipping an explicit customer confirmation the policy requires (Yao et al., 2024). So add a separate check for the confirmations your business requires before any change, cancellation or refund.
2. Run every case more than once
In the 2024 results, the best agent’s chance of solving the same retail task in all eight attempts fell below 25%, even though it succeeded on more than 60% of single attempts (Yao et al., 2024). That is not a historic curiosity; it is how these systems behave. The arithmetic is simple: if an agent gets a given refund case right 90% of the time and attempts are independent, it gets four in a row right about 66% of the time (0.9 × 0.9 × 0.9 × 0.9). That figure is an arithmetic illustration, not a measurement of any product. The practical rule: one green run in a demo tells you the agent can do the task, not that it will.
3. Put your policy in the test, and test the refusals
When the researchers removed the airline policy from the agent’s instructions, the strongest model’s airline score fell from 33.2% to 10.8% — the airline rules are ad hoc enough that common sense does not cover them (Yao et al., 2024). UK travel rules are the same: fare conditions, package alteration rules, non-refundable hotel rates, compensation eligibility. A good test set includes the cases where the right answer is no — a change to a non-changeable fare, a refund outside policy, a goodwill gesture the customer is not eligible for — because Anthropic’s guidance is explicit that one-sided test sets produce one-sided agents (Anthropic). Our pages on UK261 disruption notices and alternative arrangements under the Package Travel Regulations are a ready source of policy edge cases to turn into tasks.
4. Assume some of your expected answers are wrong
This is the lesson operators most often miss. Sierra’s February 2026 audit found airline tasks that expected the agent to offer delayed-flight compensation to passengers the policy said were not eligible, and tasks that expected cancellations the policy did not allow. Others asked for “economy” when the expected answer assumed basic economy, required a payment card the customer did not have, or let an agent dodge the no-change rule on basic economy by cancelling and rebooking (τ-bench). After the fixes, airline pass^1 rose by 14 to 20 points depending on the model, “largely because many of the original airline tasks penalized agents for correctly following the policy” (τ-bench). Anthropic adds that, with frontier models, a 0% pass rate across many attempts is most often a sign of a broken task, not an incapable agent (Anthropic). Build in a review step: every expected answer is checked against the written policy by someone from operations, and every failure is read before it is counted.
5. The simulated customer is part of the test
Most teams test with a second model playing the customer. τ²-bench measured how often that simulated customer itself went wrong: in the airline domain, annotators found errors in 47% of 100 conversations, 13% of them serious enough to stop the task being completed (Barres et al., 2025). In its telecom domain, the same paper found agents did markedly worse when they had to guide an active customer rather than act alone, and that performance varied by customer persona (Barres et al., 2025). For travel, write personas you actually serve — the anxious first-time package customer, the travel-agent booker with three reservations, the guest whose first language is not English — and keep a sample of human-played tests alongside the simulated ones.
6. Voice is a separate test, not a channel switch
If the agent will answer the reservations line, test it by voice. Sierra’s τ-Voice results, published by Sierra in February 2026 and comparing its own benchmark across three voice providers, put voice agents at roughly 26–38% under realistic audio and turn-taking, against about 54% for comparable non-reasoning text agents, and named authentication — mishearing a name or email — as the bottleneck (Sierra). A travel caller spelling a booking reference over a station announcement is exactly that case.
How to build the first test set
Start small and real. Anthropic recommends 20 to 50 tasks drawn from real failures as a starting point, beginning with the checks you already run by hand and the issues in your support queue (Anthropic). OpenAI’s guidance lists “vibe-based evals” — “it seems like it’s working” — and test data that does not reflect real traffic as anti-patterns (OpenAI). For a UK tour operator, hotel group or airline, a first set might be built like this:
- Pull 30 to 50 recent cases from the inbox and contact centre: date changes, name corrections, cancellations, refund requests, missed connections, special assistance, complaints. Cover the work the agent will actually be given.
- Rewrite each as a task with a starting booking state, a customer goal and persona, and a single correct end state under your policy. If two of your senior operations staff would not independently reach the same verdict, the task is ambiguous — fix it (this is Anthropic’s test for a good task).
- Include the no cases: requests the policy refuses, requests that must go to a human, and requests where the right move is to ask a question first.
- Include messy states: a supplier component still on request, a CRM name that does not match the booking, a customer who has already phoned. Our note on what breaks in production lists the joins that usually fail.
- Include hostile inputs: a guest message or supplier email that tries to instruct the agent. See prompt injection in travel AI agents.
- Write a reference solution for each task — the exact booking changes and messages a correct agent would produce — so you can prove the task is solvable and the grader is configured correctly.
- Use synthetic or minimised customer data. A test set built from copied live records is still personal data processing; see booking changes, UK GDPR and ATOL controls.
Choosing graders
Use code to check facts and models to check judgement, and calibrate both. Anthropic describes three grader types — code-based (fast, objective, checks state and tool calls), model-based (flexible but non-deterministic, needs calibration against human graders) and human (the gold standard, slow and expensive) — and recommends grading what the agent produced rather than the exact path it took, because agents often find valid routes the designer did not anticipate (Anthropic). It gives a travel example of its own: a model found a loophole in a τ²-bench flight-booking policy that “failed” the test as written but was better for the customer (Anthropic). OpenAI warns that model judges show position and verbosity bias and should be validated against human labels before you rely on them (OpenAI).
In practice for travel: a code check that the reservation, refund and supplier records match the reference; a code check that the required confirmation was asked before the write; a model rubric for clarity and tone, with an “unknown” option and regular human spot checks; and partial credit, so that an agent that correctly identifies the fare rule but fails to process the refund scores better than one that does nothing.
Two suites, not one
Anthropic separates capability evals, which should start with a low pass rate and show what the agent cannot yet do, from regression evals, which should pass at close to 100% and exist to catch backsliding (Anthropic). A travel business needs both. The capability suite tells you whether the agent is ready to take on, say, involuntary rebooking. The regression suite is what you rerun every time a prompt, a tool, a supplier integration or the underlying model changes — and, given the dated model retirements on OpenAI’s deprecations page, the model will change on a timetable you do not set. Our page on AI rebooking for UK tour operators is a good example of a workflow that belongs in the capability suite long before it belongs in production.
Tooling without lock-in
You do not need to build a harness from scratch, and you should not let your test cases live only in someone else’s product. The UK AI Security Institute’s Inspect is an open-source evaluation framework, developed with Meridian Labs, that structures each evaluation as a dataset, a solver (which can be a full tool-using agent) and a scorer, supports more than 20 model providers, and ships more than 200 pre-built evaluations (Inspect). Sierra publishes τ³-bench’s code and tasks openly (τ-bench), and its airline domain is a useful template for how to write a travel policy, tools and tasks. OpenAI’s deprecation notice points users of its closing Evals platform to a migration path to Promptfoo, a third-party tool (OpenAI deprecations). Whatever you use, keep the tasks, policies, reference solutions and results as plain files in your own repository, so you can change vendor, model or framework without losing the asset. This is the same build-or-buy logic as the rest of the stack; see build vs buy for travel software.
A pre-go-live gate (Kaize recommendation)
This is our recommended gate for a travel agent that can write to bookings, payments or customer records. It is not a standard or a certification, and the numbers are starting points to agree with your operations and compliance owners.
- Owned test set: at least 30 tasks covering the real work, with reference solutions, stored in your own repository.
- Policy coverage: every rule the agent is allowed to apply has at least one yes case and one no case.
- Outcome grading: every task is graded on the final booking, payment and supplier state, plus a separate check that required confirmations were obtained.
- Repeated runs: every task run at least four times; for irreversible actions (cancel, refund, issue, pay), the gate requires all four runs correct, not an average.
- Audited answers: every failure read by a person before it is counted, and every expected answer checked against the written policy by operations.
- Human-played sample: a share of tasks played by real staff using your actual customer personas, and by voice if the agent will take calls.
- Regression on change: the full suite rerun before any change of model, prompt, tool or integration, and before each scheduled model retirement.
- A named owner: someone in operations, not only engineering, who owns the test set and adds a task every time a real customer case goes wrong.
An agent that fails the gate can still be useful as a drafting assistant with a human sending every message. How that human sits in the loop, and what the agent may write without them, is covered in how travel companies should control AI agents.
Where this stops and production monitoring starts
A pre-go-live test set answers “is this agent ready to be near customers?” It does not replace watching the live system. Anthropic lists production monitoring, A/B testing, user feedback and manual transcript review as complementary layers, and notes that automated evals can create false confidence if they do not match real usage (Anthropic). The bridge is simple: every live failure becomes a new task in the suite. Our earlier piece on why travel AI breaks in production covers what to log when a live draft is wrong.
What this is not
- Not a ranking of AI models for travel. Public leaderboard scores change often, were materially revised in February 2026, and do not tell you how an agent will handle your fares, suppliers and customers.
- Not a claim that any benchmark score makes an agent safe to deploy; τ-bench’s domains are simplified, US-based simulations.
- Not legal, regulatory or data-protection advice; agree thresholds and data handling with your own advisers.
- Not a case study; Kaize reports no client results here.
Questions
How many test cases does a travel AI agent need before go-live?
Enough to cover the real work and every policy rule it applies, with yes and no cases. Anthropic suggests 20 to 50 tasks drawn from real failures as a good start (Anthropic); we recommend at least 30 for an agent that can change bookings, growing every time a live case goes wrong.
What is pass^k and why does it matter for travel?
It is the chance an agent gets the same task right in all k attempts (Yao et al., 2024). Customers and regulators judge you on every booking, not your best attempt, so consistency on irreversible actions matters more than an average score.
Can we use a public benchmark score instead of our own tests?
Use it to shortlist models, not to approve a deployment. The airline benchmark’s own expected answers needed 27 fixes in 2026 (τ-bench), and none of its policies are yours.
Should the test set use real customer bookings?
Prefer synthetic or minimised records built from real patterns. If you do use real data, treat the test set as personal data processing with the same controls as production.
Is this a Kaize evaluation case study?
No. It is a guide based on published research and vendor guidance, with recommendations labelled as ours.
Kaize point of view
The test set is the product specification for a travel AI agent. It is where your fare rules, refund rules, supplier realities and customer tone stop being opinions and become checks. Our view is that UK travel businesses should own that asset outright, grade it on the booking record rather than the reply, run it more than once, and expect to find mistakes in their own expected answers — because the most-cited airline benchmark did. Models will keep being retired and replaced on other people’s timetables; a good test set is how you change models without changing what your customers can rely on.
If you run a tour operator, DMC or hotel group and want help turning your real booking cases into a test set and go-live gate, start with an AI opportunity review or discuss a project with Kaize.