I will be studying Advanced Software EngineeringTravel and Tourism
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5.5
100%
Claude Opus 5
20%
GPT-6 sol
0%
GPT-5.6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
41m
Claude Opus 5
15m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
98
Claude Opus 5
48
Claude Opus 5.5
28
GPT-5.6 sol
27
GPT-6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (5) | ||||
| 100% | 0% | 0% | 100% | |
How to passDefine the two endpoints, 30-week usage pattern, test weekday, arrival and return times, maximum changes and target journey time before searching. Use King’s official Strand Campus page to confirm the destination and nearby stations. Record the test date and access dates, exclude known major-disruption days from the baseline, and state how any inaccessible or discontinued source will be replaced. | ||||
| 100% | 0% | 0% | 100% | |
How to passUse TfL Journey Planner, current TfL maps and status information, King’s directions, and relevant operator or National Rail station pages. For each route record every leg, mode, interchange, scheduled time, walking or cycling distance, frequency where published, step-free status, last-mile approach and total door-to-campus time in both directions. Verify that cycling routes use TfL’s cycle map or mapped Cycleways where available. Replace duplicate or infeasible suggestions with a distinct plan from the same required mode category. | ||||
| 100% | 100% | 0% | 100% | |
How to passUse TfL’s fare finder, caps and Travelcard pages, the 18+ Student Oyster eligibility page, and the relevant Railcard terms. Calculate the 30-week cost for each route under pay as you go and every genuinely eligible discount or season option, including application or card fees and any cycle parking, hire or essential maintenance allowance. Separate peak and off-peak rules, avoid combining incompatible discounts, state assumptions and mark eligibility-dependent savings as optional rather than guaranteed. | ||||
| 100% | 0% | 0% | 100% | |
How to passTest the six plans against four scenarios: closure of the main rail or Tube leg, missed connection adding 15 minutes, heavy rain that removes cycling, and an evening return after normal peak service. For each scenario identify the usable fallback, extra changes, expected time penalty and any fare impact using public journey-planning or status pages. Do not claim future punctuality; use route structure, service frequency where published and dependence on single interchange points as the resilience evidence. | ||||
| 100% | 0% | 0% | 100% | |
How to passCompare the six plans using typical round-trip time, 30-week cost, changes and walking or cycling burden, disruption and weather resilience, and fare or eligibility simplicity. Deliver a compact six-route evidence table, fare comparison, scenario matrix, ranked routes, one primary recommendation, wet-weather alternative, two disruption fallbacks and a pre-travel checklist covering live status, planned works and fare revalidation. Include links, access dates, calculations, assumptions, optional eligibility branches and every replacement. | ||||
Failure modes on this task
9 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | — | 1 | 3 | — |
| Standing up the evidence1 mode | — | 2 | 1 | — |
| Reaching the page1 mode | — | — | 1 | — |
| No signal in the judge’s words1 mode · the grader | — | 1 | — | — |