I have hand tools and a few powerHome and Garden
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5.5
100%
GPT-6 sol
88%
GPT-5.6 sol
88%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
51m
Claude Opus 5
14m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
105
Claude Opus 5
54
Claude Opus 5.5
36
GPT-6 sol
33
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (8) | ||||
| 100% | 100% | 100% | 100% | |
How to passNavigate to the Wikipedia Hand tool page and record the page title, definition of a hand tool, and the tool families or categories explicitly listed there. Then use a reputable beginner tool or safety guide to identify basic safe-use and inspection practices for hand and power tools, such as choosing the correct tool, checking for visible damage before use, keeping guards in place where applicable, and following the manufacturer's instructions. Keep the Wikipedia definitions and the external safety guidance clearly separated, and do not attribute safety recommendations to Wikipedia unless the page explicitly states them. | ||||
| 100% | 100% | 0% | 0% | |
How to passNavigate to https://en.wikipedia.org/wiki/Tool and record the page's definition of a tool, the tool categories it names, and two examples it gives for home or construction use. | ||||
| 100% | 100% | 100% | 100% | |
How to passUse a publicly accessible maintenance or safety guide to identify practical maintenance practices for hand and power tools, including inspection, cleaning, sharpening or edge care where applicable, lubrication where recommended, battery or power-cord checks, and appropriate storage. Record only maintenance practices explicitly stated by the selected source and identify the exact source URL. Do not require the Wikipedia Hand tool or Tool pages to provide maintenance guidance, and do not invent maintenance intervals that the source does not specify. | ||||
| 100% | 100% | 100% | 100% | |
How to passFind a publicly accessible beginner-friendly video or written guide about maintaining and safely using common hand and power tools. Record the source title, publisher/channel, URL, duration where displayed, and the maintenance or safety demonstrations that are actually shown or described. If timestamps are available, record timestamps for the relevant demonstrations; otherwise record the relevant section or topic instead. Do not require a particular subscriber count, YouTube channel, or other audience metric unless the task explicitly specifies one. | ||||
| 100% | 100% | 100% | 100% | |
How to passFind a publicly accessible product page for a tool organizer, small-parts box, or tool bag priced below $30 at the time of research. Record only the product information actually displayed on the page, such as product name, displayed price, dimensions, capacity or compartment information, and rating if shown. If any requested specification or rating is not displayed, record it as "not stated." Record the exact product-page URL and treat the displayed price and availability as time-sensitive. | ||||
| 100% | 100% | 100% | 100% | |
How to passCreate a practical spreadsheet-style tracker for the user's existing tools. Include columns for tool name, tool family/category, hand or power tool, condition, visible damage, sharpening/edge condition where applicable, battery condition where applicable, power-cord or electrical condition where applicable, cleaning status, lubrication/maintenance needed, last maintenance date, next review date, storage location, safety concern, and notes. Use the maintenance intervals explicitly supported by the researched sources; where no interval is provided, use "not stated" rather than inventing a schedule. Include the source URL or source reference for maintenance guidance where useful. | ||||
| 100% | 100% | 100% | 100% | |
How to passCompare the hand tool families, the tool categories, and the maintenance types for a garage with scattered hand and small power tools. Decide which three tools to check first for safety and record a one-sentence reason for each citing one of the sources. | ||||
| 100% | 100% | 100% | 100% | |
How to passCombine the researched hand-tool and power-tool categories, verified safety and maintenance guidance, organizer information, and completed inventory tracker into a concise practical handoff. Include the tool categories, what to record for each tool, maintenance actions supported by the sources, the maintenance schedule or review frequency where explicitly stated, the tools that should be checked first based on documented safety or condition criteria, and the selected organizer with its displayed price and relevant product details. Clearly distinguish source-supported safety/maintenance guidance from practical recommendations. Include the exact source URLs and mark unsupported requested details as "not stated." Do not require timestamped video demonstrations, an Amazon star rating, or any other information that the underlying sources do not explicitly provide. | ||||
Failure modes on this task
2 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Standing up the evidence1 mode | — | — | — | 1 |
| No signal in the judge’s words1 mode · the grader | — | — | 1 | — |