I'm a coffee beginner in India who wantsHobbies and Leisure
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
GPT-6 sol
83%
Claude Opus 5.5
67%
GPT-5.6 sol
0%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
21m
Claude Opus 5
14m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
117
Claude Opus 5
100
Claude Opus 5.5
77
GPT-6 sol
40
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (6) | ||||
| 100% | 100% | 0% | 0% | |
How to passFind a reputable publicly accessible coffee-education source, such as a specialty coffee association, established coffee publication, university resource, or reputable specialty coffee roaster with an educational section. Open the actual page and record what it explicitly states about grind size, coffee-to-water ratio, water quality, water temperature, brewing time, and how these variables affect extraction or taste. Record the exact source URL. If a requested variable or explanation is not explicitly covered, record it as "not stated" rather than supplementing it from general knowledge. | ||||
| 100% | 100% | 100% | 0% | |
How to passDefine objective comparison criteria before comparing the methods. Use criteria that can be supported by the sources, such as initial equipment cost, preparation time where explicitly stated, equipment required, stated capacity, and source-documented ease of use or consistency where available. Select at least two beginner-appropriate brewing methods, such as French press, pour-over, AeroPress, or moka pot. FFor each selected method, find at least one actual entry-level product listing on an accessible Indian retailer such as Amazon India or Flipkart and open the product page. Record the product name, current price, relevant specifications, capacity where stated, and exact product URL. Use the actual product information and prices as the basis for the equipment-cost portion of the comparison. Do not assign subjective scores unless the source explicitly provides information supporting them. Prices and availability are accurate only at the time of research and may change. | ||||
| 100% | 100% | 0% | 0% | |
How to passResearch burr grinders available from an accessible Indian retailer such as Amazon India or Flipkart and select multiple products, including at least one manual grinder and at least one electric grinder. Open the actual retailer or manufacturer product pages rather than relying on search-result snippets. For each grinder, record the model name, manual or electric type, burr type where stated, grind-size range or adjustment mechanism, stated capacity, current price, seller information where visible, and exact product URL. If a requested specification is not stated on the product or manufacturer page, record it as "not stated." Prices and availability should be identified as current at the time of research and subject to change. | ||||
| 100% | 100% | 100% | 0% | |
How to passIdentify Indian specialty coffee roasters with publicly accessible official websites. Open the relevant product or educational pages and record the roast levels offered, bean origin where stated, processing information where stated, available pack sizes where relevant, and any explicit guidance the roaster provides about freshness, roast dates, storage, or recommended use after opening. Record the exact source URL for each roaster. Do not infer freshness periods or storage recommendations that the roaster does not explicitly state; record those details as "not stated." | ||||
| 100% | 100% | 100% | 0% | |
How to passSearch publicly readable home-coffee community discussions, such as a relevant subreddit or coffee-focused forum, where multiple users discuss beginner home coffee setups, grinder choices, brewing methods, common mistakes, or purchasing regrets. Open the actual discussion and record its exact URL. Summarise recurring recommendations, reported mistakes, setup regrets, and practical advice, while clearly labelling all such information as community-reported observations rather than authoritative facts. If a selected discussion is inaccessible or does not contain multiple contributors, use another qualifying public discussion and explain the replacement. | ||||
| 0% | 100% | 100% | 0% | |
How to passSynthesize the collected evidence into a single beginner-oriented setup recommendation. Select one brewing method based on the objective comparison criteria and the actual Indian product information gathered in Step 2. Select one grinder based on the documented grinder specifications and price comparison. Select one Indian coffee bean option based on the information gathered in Step 4. Include a side-by-side brewing-method comparison table, a grinder comparison table covering manual and electric options, a transparent initial setup budget, and a basic step-by-step brewing workflow using only parameters supported by the cited sources. Calculate the initial setup budget using the selected brewer, selected grinder, and one clearly identified standard package of coffee beans for which a current price is available. Distinguish clearly between source-derived facts, current retailer information, and community-reported observations. | ||||
Failure modes on this task
9 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Standing up the evidence1 mode | — | — | 2 | 2 |
| Delivering the result1 mode | 1 | — | — | 2 |
| No signal in the judge’s words1 mode · the grader | — | — | — | 2 |