I am a first-time used-car buyer in IndiaVehicles
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5.5
88%
GPT-6 sol
63%
GPT-5.6 sol
38%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
23m
Claude Opus 5
12m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
117
Claude Opus 5
69
GPT-6 sol
50
Claude Opus 5.5
21
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (8) | ||||
| 100% | 100% | 100% | 0% | |
How to passEstablish the comparison criteria for three Indian used hatchbacks, including crash safety, running costs, reliability, resale value, used-market price, practicality for everyday city driving, maintenance/service considerations, and ownership risks. Define how manufacturer specifications, independent safety-test results, marketplace listings, and owner/community experiences will be weighted and kept separate. | ||||
| 100% | 100% | 100% | 100% | |
How to passResearch and select exactly three suitable Indian hatchback models for everyday city driving. Prefer models from different manufacturers where practical, but do not make “three different manufacturers” a mandatory requirement if the evidence supports a better comparison. Record the model, relevant model year/variant, fuel/powertrain, transmission, approximate used-market age range, and reason for selection. Ensure each candidate has sufficient publicly accessible manufacturer, safety-test, used-market, and ownership evidence to support a meaningful comparison. | ||||
| 100% | 100% | 0% | 0% | |
How to passFor each selected model, use official manufacturer sources to record the specifications relevant to ownership and comparison, including engine/powertrain, transmission, dimensions, boot capacity, fuel-efficiency claims, safety equipment, service/warranty information where publicly stated, and relevant variant differences. Record the exact model year/variant to which each specification applies and provide the source URL. | ||||
| 100% | 100% | 100% | 100% | |
How to passFor each selected model, research applicable independent crash-test results from Bharat NCAP, Global NCAP, or another recognized independent safety-testing organization. Record the test year, exact tested variant/configuration, adult and child occupant results where available, and important qualifications about applicability to the used vehicle being considered. Do not transfer a rating from one generation, variant, or test protocol to another without evidence. | ||||
| 100% | 100% | 0% | 0% | |
How to passFor each selected model, find at least two current used-car listings from reputable Indian marketplaces or dealers. Record the exact model year, variant, fuel type, transmission, odometer reading, asking price, location, seller type, listing date or date checked, and source URL. Use multiple listings to establish an indicative asking-price range rather than presenting one listing as the market value. Keep asking prices separate from completed transaction prices. | ||||
| 100% | 100% | 100% | 100% | |
How to passResearch maintenance requirements, warranty information, service considerations, common ownership concerns, running-cost factors, and reliability observations using manufacturer information and reputable automotive sources. Where owner experiences are used, identify them explicitly as community evidence rather than verified defect rates. Distinguish documented maintenance requirements from anecdotal reliability claims. | ||||
| 100% | 100% | 100% | 0% | |
How to passFind publicly accessible Indian owner/community discussions containing multiple distinct contributors for at least the selected models. Record recurring observations about reliability, servicing, fuel economy, comfort, defects, dealer/service experience, resale, and long-term ownership. Link to the actual discussions and clearly label these as anecdotal evidence. Do not infer population-wide reliability from a small number of posts or reviews. | ||||
| 100% | 0% | 0% | 0% | |
How to passProduce a final comparison table covering safety, running costs, reliability evidence, resale considerations, used-market asking prices, maintenance/service factors, practicality, and ownership risks. Keep manufacturer facts, independent safety-test evidence, marketplace data, and community experiences visibly separate. Identify the strongest option for a first-time city buyer and explain the recommendation using the documented evidence rather than popularity alone. Include the major uncertainties and model-year/variant qualifications. Finish with a practical used-car inspection checklist covering documents/history, accident or structural damage, tyres, brakes, suspension, engine/transmission, cooling system, electrical equipment, AC, interior, underbody, signs of flood/water damage, service records, and a professional pre-purchase inspection. Do not purchase or arrange a vehicle. | ||||
Failure modes on this task
9 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | — | 1 | 3 | 1 |
| Reaching the page1 mode | — | — | — | 1 |
| No signal in the judge’s words1 mode · the grader | — | — | — | 3 |