Hannah Lindqvist lives in a 2 800 sqComputers Electronics and Technology
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
75%
Claude Opus 5.5
75%
Claude Opus 5
50%
GPT-6 sol
38%
GPT-5.6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
16m
Claude Opus 5
10m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
75
Claude Opus 5
33
Claude Opus 5.5
22
GPT-6 sol
22
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (8) | ||||
| 100% | 100% | 100% | 100% | |
How to passConfirm square footage (2,800 sq ft, two stories), the dead zone problem (second floor), and the $300 budget. | ||||
| 100% | 100% | 0% | 100% | |
How to passGo to https://www.amazon.eero.com/, find the pack size (2 or 3 units) rated for coverage matching her square footage, and record price. | ||||
| 100% | 100% | 100% | 100% | |
How to passGo to https://store.google.com/, find the comparable pack size and coverage rating, and record price. | ||||
| 100% | 100% | 100% | 100% | |
How to passGo to https://www.tp-link.com/, find the comparable pack size and coverage rating, and record price. | ||||
| 0% | 0% | 0% | 0% | |
How to passNote that advertised coverage is often optimistic for multi-story homes with more walls; check independent reviews specifically for real-world performance of the exact eero, Google Nest Wifi, and TP-Link Deco models selected for comparison. | ||||
| 0% | 0% | 0% | 0% | |
How to passCompare app-based setup simplicity for each system, since Hannah isn't technical. | ||||
| 100% | 0% | 0% | 100% | |
How to passConfirm the Wi-Fi standard and advertised speed capabilities of each compared system. Explain the practical differences between the systems and note that advertised maximum speeds may not reflect real-world performance. | ||||
| 100% | 0% | 0% | 100% | |
How to passProduce a table with columns: System, Pack size, Coverage rating, Price, Setup ease then state the top pick with reasoning matched to eliminating her second-floor dead zones. | ||||
Failure modes on this task
13 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | — | 1 | 2 | 1 |
| Standing up the evidence1 mode | 1 | — | 1 | — |
| Reaching the page1 mode | — | — | 1 | — |
| No signal in the judge’s words1 mode · the grader | 1 | 3 | 1 | 1 |