I'm a first-time car buyer in IndiaVehicles
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
83%
Claude Opus 5.5
83%
GPT-6 sol
83%
GPT-5.6 sol
17%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
85m
GPT-5.6 sol
30m
Claude Opus 5
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
118
Claude Opus 5
100
Claude Opus 5.5
99
GPT-6 sol
69
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (6) | ||||
| 100% | 100% | 100% | 100% | |
How to passSearch Google for established hatchback models currently available or recently sold in India. Select at least three models from different manufacturers that fall within the practical first-time-buyer range described in the request and have sufficient publicly accessible information for comparison. Prefer models for which official manufacturer pages, safety information, used-market listings, and owner discussions can all be found. Open the relevant official manufacturer or model pages and record the model name, engine options, transmission options, dimensions, boot capacity, claimed fuel-efficiency figure where stated, safety features, warranty information where stated, and exact source URL. If a requested specification is not explicitly stated on the source page, record it as "not stated." Leave the most useful official model pages open for comparison. | ||||
| 100% | 100% | 100% | 0% | |
How to passFor each selected hatchback, search an established publicly accessible vehicle-safety source such as Global NCAP, Bharat NCAP, or another recognised crash-testing organisation. Record the applicable crash-test rating, test year, adult-occupant or child-occupant result where explicitly provided, tested variant or model information, and any important qualification stated by the safety organisation. Do not transfer a rating from one generation, variant, or model year to another unless the source explicitly applies the result to that vehicle. Record the exact safety-source URL. If no applicable public test result is found for a selected model, record "not stated" rather than inferring a rating from another model. | ||||
| 100% | 100% | 100% | 0% | |
How to passGo to an Indian used-car marketplace such as Cars24, Spinny, CarDekho, or OLX and search for the selected hatchback models. Find at least two currently visible listings for each model in a broadly comparable used-car age range, preferably around three to six years old. For every selected listing, record the asking price, model year, variant, fuel type, transmission, kilometres driven, location where shown, and exact listing URL. Prefer listings with enough visible information to make a meaningful comparison and avoid mixing substantially different vehicle configurations without clearly identifying the difference. Prices, mileage, availability, and listing details should be identified as current at the time of research and may change. | ||||
| 100% | 100% | 100% | 0% | |
How to passSearch publicly accessible official manufacturer service information and reputable automotive publications for ownership considerations relevant to the selected models. Record service intervals where explicitly stated, warranty or extended-warranty information, maintenance or running-cost information explicitly reported by the source, and any other documented considerations that would matter when purchasing a used example. Clearly distinguish manufacturer-stated information from observations reported by automotive publications. If a requested ownership detail is not explicitly stated, record it as "not stated." | ||||
| 100% | 100% | 100% | 0% | |
How to passSearch Reddit in a relevant Indian automotive community such as r/CarsIndia or another clearly relevant public subreddit. Find a publicly readable discussion containing comments from multiple users about one or more of the selected hatchbacks, used-car ownership, maintenance, reliability, or common first-time-buyer mistakes. Open the actual thread and record the exact thread URL. Summarise recurring owner-reported experiences, complaints, recommendations, maintenance concerns, and inspection or purchasing advice. Clearly label all such findings as community-reported observations rather than authoritative facts. If the selected Reddit thread is inaccessible or does not contain multiple relevant contributors, search for another qualifying Reddit thread. | ||||
| 0% | 0% | 0% | 0% | |
How to passCombine the research into a structured first-time-buyer guide covering the selected hatchbacks. Include a vehicle specification comparison table covering engine, transmission, dimensions, boot capacity, claimed fuel efficiency, safety features, and warranty information where stated; a safety comparison based specifically on the applicable public crash-test evidence; a used-market comparison showing at least two current listings for each model with asking price, year, variant, fuel type, transmission, kilometres driven, and location; an ownership and maintenance section based on manufacturer and reputable automotive sources; and a clearly labelled community-observations section based on Reddit. Identify one preferred model based on the documented criteria and explain why it best fits the evidence gathered rather than simply calling it the most popular or cheapest option. Calculate a transparent representative purchase-price range using the researched listings and provide a practical first-time-buyer budget that separates the vehicle asking price from any additional costs that are explicitly supported by the researched sources. Finish with a used-car inspection and purchase checklist covering documents, vehicle history, odometer consistency, mechanical condition, tyres, brakes, accident or repair evidence, ownership transfer, insurance, and any other checks explicitly supported by the sources. Do not present a checklist item as a factual requirement unless the cited source supports it; where a requested detail is not explicitly stated, record it as "not stated." | ||||
Failure modes on this task
8 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | 1 | 1 | 1 | 2 |
| Standing up the evidence1 mode | — | — | — | 1 |
| No signal in the judge’s words1 mode · the grader | — | — | — | 2 |