Plan a realistic 9-day family trip through FranceTravel and Tourism
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
57%
Claude Opus 5
14%
Claude Opus 5.5
0%
GPT-6 sol
0%
GPT-5.6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
34m
Claude Opus 5
8m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
58
Claude Opus 5
11
GPT-6 sol
5
GPT-5.6 sol
0
Claude Opus 5.5
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (7) | ||||
| 0% | 0% | 0% | 0% | |
How to passUse reliable public travel sources to identify at least five candidate destinations outside Paris and compare their family-friendly attractions, rail accessibility, travel time, and suitability for a short stay. | ||||
| 0% | 0% | 0% | 0% | |
How to passCompare at least two possible three-city routes using reliable public rail journey and route-order information. Evaluate total travel time, transfers, and whether the route creates unnecessary backtracking. | ||||
| 0% | 0% | 0% | 0% | |
How to passFor each selected destination, compare at least three neighborhoods or lodging areas for family convenience based on transport access, attraction proximity, and food options. Identify representative public accommodation listings. | ||||
| 0% | 0% | 0% | 100% | |
How to passFind at least four suitable attractions in each destination. Record official visitor information, expected visit duration, child suitability, reservation requirements, and any recurring access restrictions. | ||||
| 0% | 0% | 0% | 100% | |
How to passIdentify family-appropriate restaurants, bakeries, markets, or food experiences in each destination. Compare menus, location, practical access, and review themes. | ||||
| 100% | 0% | 0% | 100% | |
How to passCreate a nine-day itinerary with transportation, attractions, meals, free time, and backup indoor options. Include realistic transfer time and avoid scheduling multiple major attractions that require cross-city travel on the same day. | ||||
| 0% | 0% | 0% | 100% | |
How to passProduce a complete itinerary with accommodation areas, transport, attractions, meals, approximate budget categories, booking or reservation requirements, and source links. End with a checklist of information that should be rechecked before booking. | ||||
Failure modes on this task
23 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Standing up the evidence1 mode | 2 | 2 | 1 | 1 |
| Reaching the page1 mode | 1 | 1 | 1 | — |
| Delivering the result1 mode | — | 1 | 2 | — |
| No signal in the judge’s words1 mode · the grader | 3 | 3 | 3 | 2 |