Eight cities one country one tournament The 2027Sports
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
80%
Claude Opus 5.5
40%
GPT-5.6 sol
40%
Claude Opus 5
20%
GPT-6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
39m
Claude Opus 5
28m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
115
Claude Opus 5
74
GPT-6 sol
67
GPT-5.6 sol
43
Claude Opus 5.5
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (5) | ||||
| 0% | 0% | 100% | 100% | |
How to passNavigate to fifa.com/en/tournaments/womens/womensworldcup/brazil-2027/host-cities and list all eight host cities with the official FIFA stadium name and the listed capacity for each. Record the URL so the source is traceable. | ||||
| 100% | 100% | 100% | 100% | |
How to passChoose three host cities that form a compact cluster for a visitor arriving through São Paulo or Rio de Janeiro as the international gateway. On rome2rio.com, look up the travel time, cheapest transport mode, and estimated fare between each of the three city pairs, producing three route lookups total. Record mode, journey time, and fare for each leg. | ||||
| 100% | 0% | 0% | 0% | |
How to passFor each of the three chosen cities, navigate to numbeo.com/cost-of-living/in/[city-name] and record the cost of living index and the average price of a meal at an inexpensive restaurant shown on the page at task-start time. Note the date the Numbeo data was last updated for each city, since Brazilian cities are not always updated at the same frequency, and flag any city where the data appears older than 12 months. | ||||
| 100% | 0% | 0% | 0% | |
How to passOn booking.com, search for hotels within 3 km of the stadium in each of the three cities for a single night during the tournament window. Record the cheapest available option with the property name, price per night, and the distance to the stadium as shown on the listing. For the same city, run a second search for a comparable night in the same month that falls outside the tournament window. Record the cheapest option and calculate the tournament-night price premium as a percentage over the non-tournament night, showing the calculation explicitly. | ||||
| 100% | 0% | 0% | 0% | |
How to passRank the three cities from most to least practical for the first-time South America visitor. Each ranking position must cite at least two specific figures from Phases 2 to 4, with no reasoning drawn from general Brazil travel knowledge or training recall about those cities. For the top-ranked city, name one specific logistical risk that the data found during the task suggests could still go wrong for this visitor, tied to a figure rather than a general concern about travelling in Brazil. | ||||
Failure modes on this task
11 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Standing up the evidence1 mode | 1 | 1 | 1 | 1 |
| Delivering the result1 mode | — | 1 | 1 | 2 |
| Reaching the page1 mode | — | — | 1 | — |
| No signal in the judge’s words1 mode · the grader | — | 2 | — | — |