I want to analyze aviation safety fleet ageTravel and Tourism
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
88%
Claude Opus 5.5
84%
GPT-5.6 sol
80%
GPT-6 sol
44%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
58m
Claude Opus 5
26m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
118
Claude Opus 5
87
GPT-6 sol
59
GPT-5.6 sol
54
Claude Opus 5.5
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (25) | ||||
| 100% | 100% | 100% | 100% | |
How to passResearch official European Commission documentation regarding the EU Air Safety List ban on air carriers certified in Nepal. Record legal context in working_dossier.md. | ||||
| 100% | 100% | 100% | 100% | |
How to passInspect official ICAO safety oversight audit results for Nepal across primary aviation legislation, airworthiness, and accident investigation. | ||||
| 100% | 100% | 0% | 0% | |
How to passReview Civil Aviation Authority of Nepal regulatory requirements for twin-engine mandatory operations on mountain STOL routes. | ||||
| 100% | 100% | 100% | 0% | |
How to passExamine safety records for ATR 72-500, DHC-6 Twin Otter, Dornier 228, and Bombardier CRJ-200 aircraft operating in Nepal. | ||||
| 100% | 100% | 100% | 0% | |
How to passDocument scoring weights for fleet age, ICAO compliance, terrain awareness systems (TAWS-B), and accident history in working_dossier.md. | ||||
| 100% | 100% | 100% | 100% | |
How to passSearch official fleet registries and plane spotter databases for Buddha Air. Record aircraft types, counts, and registration dates in scratchpad. | ||||
| 0% | 0% | 0% | 0% | |
How to passCalculate average fleet age and verify installation of Terrain Avoidance and Warning Systems (TAWS) across Buddha Air's ATR fleet. Record what is publicly documented; if not disclosed, note it as not publicly available. | ||||
| 100% | 100% | 100% | 100% | |
How to passSearch official registries for Yeti Airlines fleet details. Record aircraft specifications and operational routes in scratchpad. | ||||
| 100% | 100% | 100% | 0% | |
How to passCalculate average fleet age and verify safety management system (SMS) implementation progress for Yeti Airlines. | ||||
| 100% | 100% | 100% | 100% | |
How to passSearch official registries for Shree Airlines turboprop (Dash 8 Q400) and jet fleet details. Record data in scratchpad. | ||||
| 100% | 100% | 100% | 0% | |
How to passCalculate average fleet age and inspect maintenance tracking compliance for Shree Airlines. | ||||
| 100% | 100% | 100% | 100% | |
How to passSearch official aviation registries for STOL operators Sita Air (Dornier 228) and Tara Air (Twin Otter). Record data in scratchpad. | ||||
| 0% | 0% | 0% | 0% | |
How to passInspect weather radar equipment and ground proximity warning system upgrades across Sita Air and Tara Air STOL fleets. Record what is publicly documented; if not disclosed, note it as not publicly available. | ||||
| 100% | 100% | 100% | 100% | |
How to passReview recent CAAN safety directives regarding mandatory weather checks at destination airfields prior to takeoff. | ||||
| 100% | 100% | 100% | 0% | |
How to passCompute overall safety index scores (0 to 100) for all 5 domestic carriers in working_dossier.md. | ||||
| 100% | 100% | 100% | 100% | |
How to passConfirm that complete fleet age, aircraft registration, and audit data are recorded for all 5 target airlines. Treat items explicitly noted as not publicly available as complete for this check. | ||||
| 100% | 0% | 100% | 0% | |
How to passQuery Flight Safety Foundation ASN records for domestic aviation incidents in Nepal over the past decade. | ||||
| 100% | 100% | 100% | 100% | |
How to passConfirm whether the fallback rule was needed during this research — if any aviation database page failed twice, verify that an alternative official report was used and noted in the scratchpad. If nothing failed, note that all sources remained accessible. | ||||
| 0% | 0% | 100% | 0% | |
How to passInspect carrier documentation regarding simulator training access for high-altitude STOL airstrip landings. Record what is publicly documented; if not disclosed, note it as not publicly available. | ||||
| 100% | 0% | 0% | 0% | |
How to passInspect aircraft maintenance hangar facilities and CAAN MRO approval status for domestic airlines. | ||||
| 100% | 100% | 100% | 0% | |
How to passRank the 5 carriers based on average fleet age, safety equipment, and regulatory compliance records. | ||||
| 100% | 100% | 100% | 100% | |
How to passRe-verify all initial constraints (5 airlines, EU ban context, ICAO compliance, fleet age analysis) against target output. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a detailed markdown table detailing carrier names, fleet size, main aircraft types, average age, safety score, and official links. | ||||
| 100% | 100% | 100% | 100% | |
How to passSummarize the safety and reliability findings in plain language for reference, without issuing direct travel-booking recommendations. | ||||
| 100% | 100% | 100% | 0% | |
How to passWrite and save the complete 25-section safety research report to working_dossier.md. | ||||
Failure modes on this task
26 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | 1 | 3 | 1 | 4 |
| Standing up the evidence1 mode | 1 | 1 | — | 2 |
| Reaching the page1 mode | 1 | — | — | 2 |
| No signal in the judge’s words1 mode · the grader | — | 1 | 3 | 6 |