I am looking to buy a new electricHobbies and Leisure
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
83%
Claude Opus 5.5
83%
GPT-5.6 sol
83%
Claude Opus 5
50%
GPT-6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
46m
Claude Opus 5
14m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
119
Claude Opus 5
55
GPT-6 sol
44
Claude Opus 5.5
39
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (6) | ||||
| 100% | 100% | 100% | 100% | |
How to passResearch currently available electric guitars suitable for both rock and blues, using reputable retailers and manufacturer information. Record the exact model, configuration, current price, availability, and finish/shape. Do not treat a generic model listing as sufficient when the task requires a particular exotic finish. Verify that the specific guitar variant satisfies the user's maximum $500 USD budget and is actually purchasable when researched. | ||||
| 100% | 100% | 100% | 100% | |
How to passFrom the researched guitars, identify models priced at $500 USD or less that also have an unusually distinctive color, finish, body shape, or stage-oriented visual design. Verify the exact finish/variant and price rather than assuming that every color of a model costs the same. Preserve the purchase currency and date checked. A guitar qualifies only if both the price and the requested visual characteristic are verified. | ||||
| 100% | 0% | 100% | 100% | |
How to passSelect exactly five qualifying guitar variants. For each, provide the exact model and finish, current price, availability, manufacturer/retailer evidence, and a purchase link that leads to the specific qualifying finish or variant, not merely the model's generic product page. Briefly explain its suitability for rock and blues based on documented pickup configuration, construction, controls, or credible reviews. Do not rank guitars solely by popularity. | ||||
| 0% | 0% | 0% | 0% | |
How to passProvide an accessible image or image-search result for each of the five specific guitar variants selected in Step 3, ensuring the image corresponds to the stated finish and model. Do not require opening a new browser tab; instead, make the images directly accessible in the research output. Do not use an image of a different finish as evidence for the qualifying product. | ||||
| 100% | 100% | 100% | 100% | |
How to passResearch how electric-guitar string gauge affects tension, feel, attack, sustain, tuning stability, and perceived/heavier tone. Compare appropriate gauge options for the selected guitars, considering scale length, tuning, pickup configuration, playing style, and the guitar manufacturer's specifications where available. Explain that heavier strings can provide greater tension and a different feel but do not automatically produce a universally “heavier” sound; amplifier settings, pickups, tuning, picking technique, and setup also materially affect the result. Recommend a practical string gauge for the selected guitars and identify any setup or intonation considerations when substantially changing gauge. | ||||
| 100% | 0% | 100% | 100% | |
How to passCreate a concise document containing the five selected guitar models and exact finishes, current prices and availability, qualifying purchase links, evidence supporting rock/blues suitability, and accessible images. Include the recommended string gauge for each guitar or a clearly justified common gauge where appropriate, followed by an explanation of how string gauge affects tension, feel, attack, and tone. Clearly distinguish verified current retailer/manufacturer information from recommendations or judgments. Do not purchase anything. | ||||
Failure modes on this task
6 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Standing up the evidence1 mode | 1 | 3 | — | — |
| No signal in the judge’s words1 mode · the grader | — | — | 1 | 1 |