I want to shortlist used cars to buyVehicles
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5.5
100%
GPT-5.6 sol
100%
Claude Opus 5
80%
GPT-6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
51m
Claude Opus 5
17m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
107
Claude Opus 5
79
Claude Opus 5.5
66
GPT-6 sol
54
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (5) | ||||
| 100% | 100% | 100% | 100% | |
How to passSearch https://www.autotrader.co.uk/car-search with filters for a petrol or petrol-hybrid hatchback or estate, registration year 2017 or newer, maximum mileage 80,000, and price up to £12,000. Open exactly five different listings meeting all four filters and record each one's price, mileage, registration year, and fuel type. | ||||
| 100% | 0% | 100% | 100% | |
How to passOn each of the five listing pages, scroll to the 'Vehicle history' section. Record the owner count, the MOT valid-until date, and whether the listing is flagged stolen, scrapped, imported, exported, or written-off, plus how many checks passed. This panel appears on both dealer and private-seller listings. | ||||
| 100% | 100% | 100% | 100% | |
How to passOn the same five listing pages, read the vehicle-history checks and insurance write-off disclaimer text near the bottom of the page. Record that AutoTrader's free history flags draw on a private insurance database rather than GOV.UK, and that AutoTrader itself recommends a full paid history check before buying. | ||||
| 100% | 100% | 100% | 100% | |
How to passOpen https://www.gov.uk/check-mot-history and https://www.gov.uk/check-vehicle-tax. Record exactly what each free official tool can confirm once a buyer has the seller's registration number: MOT pass or fail history, recorded mileage, and tax or SORN status. Confirm neither tool covers write-off category or outstanding finance. | ||||
| 100% | 100% | 100% | 100% | |
How to passIn a document, build a table comparing all five candidates by price, mileage, registration year, fuel type, MOT valid-until date, and any AutoTrader history flags. Add a short due-diligence note listing what's free-checkable via GOV.UK, what AutoTrader's panel already covers, and what still needs the seller's registration or a paid report. Do not reserve any vehicle, message any seller, or enter payment details. Leave every listing tab open. | ||||
Failure modes on this task
1 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| No signal in the judge’s words1 mode · the grader | — | 1 | — | — |