I want to put together a basic vinylArts & Entertainment
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
GPT-6 sol
63%
Claude Opus 5.5
63%
GPT-5.6 sol
0%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
23m
Claude Opus 5
18m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
113
Claude Opus 5
100
Claude Opus 5.5
82
GPT-6 sol
32
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (8) | ||||
| 100% | 100% | 100% | 0% | |
How to passSearch for belt drive turntables with a built-in switchable phono preamp, an adjustable counterweight and anti-skate control, and a magnetic cartridge, and shortlist 2 candidates sourced from at least 2 different retailers, confirming each of these specs is stated directly on the product page rather than assumed from the product name before adding it to the shortlist. | ||||
| 100% | 100% | 0% | 0% | |
How to passOpen each shortlisted turntable's own live product page in its own tab and record the price, shipping cost, drive type, whether the built-in preamp is confirmed, and the cartridge type, noting explicitly if price or shipping does not load on a plain page fetch rather than leaving it blank. | ||||
| 100% | 100% | 100% | 0% | |
How to passSearch for powered bookshelf speaker pairs with an RCA input and a combined output of at least 20 watts, and shortlist 2 candidates sourced from at least 2 different retailers, confirming from the specs that at least one speaker in the pair has a built-in amplifier and is not a passive design requiring an external amp before adding it to the shortlist. | ||||
| 100% | 100% | 0% | 0% | |
How to passOpen each shortlisted speaker pair's own live product page in its own tab and record the price, shipping cost, combined wattage, and confirm the RCA input is listed, noting explicitly if price or shipping does not load on a plain page fetch rather than leaving it blank. | ||||
| 0% | 100% | 100% | 0% | |
How to passSearch for vinyl record cleaning kits that include both a carbon fiber anti-static brush and a liquid cleaning solution of at least 100ml, and shortlist 2 candidates sourced from at least 2 different retailers, confirming both components are included and the result is not just a dry cloth or brush sold alone before adding it to the shortlist. | ||||
| 0% | 100% | 0% | 0% | |
How to passOpen each shortlisted cleaning kit's own live product page in its own tab and record the price, shipping cost, and the stated solution volume converted to ml if needed, noting explicitly if price or shipping does not load on a plain page fetch rather than leaving it blank. | ||||
| 0% | 100% | 100% | 0% | |
How to passCheck every possible combination of one turntable, one pair of powered speakers, and one cleaning kit from the recorded candidates against the $350 combined price and shipping cap, and show the arithmetic for the recommended combination that fits the budget. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a document organizing every candidate across the three categories with item, retailer, price, shipping, and link, clearly highlighting which turntable, speaker pair, and cleaning kit make up the recommended combination along with the final total, leaving the document open for review. | ||||
Failure modes on this task
14 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Standing up the evidence2 modes | 1 | — | — | 3 |
| Delivering the result1 mode | 1 | — | — | 3 |
| No signal in the judge’s words1 mode · the grader | 1 | — | 3 | 2 |