I plan on going to Tiradentes Minas GeraisTravel and Tourism
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5.5
100%
GPT-6 sol
100%
GPT-5.6 sol
17%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
21m
GPT-5.6 sol
20m
Claude Opus 5
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
119
Claude Opus 5
100
Claude Opus 5.5
53
GPT-6 sol
49
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (6) | ||||
| 100% | 100% | 100% | 100% | |
How to passSearch the Booking website for hotels in Tiradentes, Minas Gerais, for 2 nights next weekend that have a jacuzzi available. Fetch information about them. https://www.booking.com/ | ||||
| 100% | 100% | 100% | 0% | |
How to passFor all of the hotels found in Step 1, check the Booking.com listing and publicly available Genius information to determine whether a Genius discount is visibly offered and, where the discount percentage or discounted price is publicly displayed, record the amount. If Booking requires sign-in to reveal a personalized Genius offer, record “not publicly verifiable” rather than assuming a discount. | ||||
| 100% | 100% | 100% | 0% | |
How to passSearch reputable tourism, attraction and mapping sources for activities involving waterfalls near Tiradentes. For each suitable location, record the waterfall/activity name, approximate distance from Tiradentes city center, practical directions or route information, and a source link. If a public map directions link is available, include it. | ||||
| 100% | 100% | 100% | 0% | |
How to passSearch TripAdvisor for restaurants that serve international cuisine and have high ratings on TripAdvisor based on their food quality and service. Fetch their information. Pick the 4 highest-rated restaurants based on their food quality and service https://www.tripadvisor.com/ | ||||
| 100% | 100% | 100% | 0% | |
How to passSearch reputable tourism, museum and historical sources for museums in Tiradentes that help explain the city's development and important historical periods. Select 3 museums and record their names, historical relevance, location and source information. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a spreadsheet with the hotel information ordered by price. Include columns for Genius discount status and discount amount, using “not publicly verifiable” where Booking does not expose a personalized discount without sign-in. List the available waterfall activities with their distance from the city center and source/directions link. Include 4 international-cuisine restaurants with an explanation of why each is worth visiting based on food quality and service, and list the 3 selected museums with their historical relevance. | ||||
Failure modes on this task
5 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Standing up the evidence1 mode | — | — | — | 1 |
| Delivering the result1 mode | — | — | — | 1 |
| No signal in the judge’s words1 mode · the grader | — | — | — | 3 |