I've got a doorknob-sized hole in my bedroomEcommerce & Shopping
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5
83%
Claude Opus 5.5
50%
GPT-5.6 sol
33%
GPT-6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
15m
Claude Opus 5
10m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
64
Claude Opus 5
34
Claude Opus 5.5
30
GPT-6 sol
19
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (6) | ||||
| 100% | 100% | 0% | 100% | |
How to passFind a step-by-step guide to patching a small hole in drywall from an established home-improvement source (for example This Old House or a national home-improvement chain's project library). Open it and record the page title, the full URL, and the complete tools-and-materials list the guide names. | ||||
| 0% | 0% | 0% | 100% | |
How to passOn https://www.youtube.com/, search "how to patch a hole in drywall" and open a video from an established home-improvement channel (100,000+ subscribers) whose description includes a timestamped chapter list. Record the channel name, channel subscriber count, video title, video URL, and video length as displayed. | ||||
| 100% | 0% | 100% | 100% | |
How to passFrom the video's chapter list, record at least four chapters with their start timestamps. Note which chapter addresses a doorknob-sized hole (a small or fist-sized hole repair rather than a full section replacement). | ||||
| 100% | 100% | 100% | 100% | |
How to passSearch for a drywall repair kit and open one product listing priced under $25 that shows a star rating. Record the product name, brand, the items included as shown on the listing, the displayed price, and the star rating. If a field is not shown, record "not shown". | ||||
| 100% | 0% | 100% | 100% | |
How to passCompare the items included in the kit from step 4 with the tools-and-materials list from the guide in step 1 and the method shown in the video chapters from step 3. Decide whether the kit alone covers the doorknob-hole repair. Record a yes/no verdict and one sentence naming the specific items still missing, if any. | ||||
| 100% | 0% | 0% | 100% | |
How to passProduce a simple weekend plan: the materials (the kit from step 4 plus any missing items from step 5), the video chapters to follow in order with their timestamps from step 3, the guide reference from step 1, and a short step-by-step order of work for a doorknob-sized hole, from prepping the hole to being ready to paint. | ||||
Failure modes on this task
8 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Reaching the page1 mode | — | 2 | 1 | — |
| Delivering the result1 mode | — | 1 | 1 | — |
| Standing up the evidence1 mode | 1 | — | — | — |
| No signal in the judge’s words1 mode · the grader | — | 1 | 1 | — |