I am planning to buy some clothes onlineEcommerce & Shopping
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
75%
Claude Opus 5.5
75%
Claude Opus 5
38%
GPT-6 sol
38%
GPT-5.6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
22m
Claude Opus 5
19m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
105
Claude Opus 5
51
GPT-6 sol
50
Claude Opus 5.5
50
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (8) | ||||
| 0% | 0% | 0% | 0% | |
How to passOpen the official websites for Myntra, AJIO, and Marks & Spencer India and find one women's fashion product from each store with a listed price between ₹1,500 and ₹3,000. Record the exact product name, category, listed price, and product-page details needed to identify it. Keep each selected product page open. | ||||
| 100% | 100% | 100% | 100% | |
How to passOpen GrabOn and search for currently listed offers for Myntra, AJIO, and Marks & Spencer India. Record the relevant coupon code or promotional offer, discount amount or percentage, minimum-spend requirement, expiry information if provided, and any important exclusions. Only use offers whose applicability can be reasonably established from the available terms. | ||||
| 100% | 100% | 100% | 100% | |
How to passOpen CashKaro and check the current offers or cashback rates for Myntra, AJIO, and Marks & Spencer India. Record the advertised cashback or coupon benefit and its relevant terms. Clearly distinguish immediate discounts from cashback that would be credited later. Do not create a CashKaro account or activate an offer. | ||||
| 100% | 100% | 100% | 100% | |
How to passOpen CouponDunia and search for current offers for the same three retailers. Record any applicable coupon, cashback, or promotional offer and its relevant conditions. If an offer requires a minimum purchase or has exclusions, record those conditions rather than treating the advertised discount as universally available. | ||||
| 100% | 0% | 0% | 100% | |
How to passFor each of the three retailers, determine the strongest single savings route that can reasonably be applied to the selected product. Calculate the potential rupee saving from the listed coupon or cashback and the resulting effective cost. Do not stack multiple offers unless the terms explicitly permit stacking. Treat cashback that is credited after purchase as cashback rather than an immediate reduction in the checkout price. | ||||
| 100% | 0% | 0% | 100% | |
How to passRank the three shopping routes from #1 to #3 by the potential total saving on the selected product, ordered from highest to lowest. A larger advertised percentage should not automatically receive the higher ranking if the offer has a large minimum-spend requirement or other restriction that makes it less useful for the selected purchase. Record how straightforward each offer is to use and any significant restrictions alongside the ranking as explanatory notes, without letting them change the order. | ||||
| 100% | 0% | 0% | 100% | |
How to passRecheck the selected product prices and the relevant coupon or cashback terms before finalizing the ranking. If an offer cannot be verified as applicable, exclude it from the savings calculation and explain that it could not be confirmed. Do not estimate missing coupon values or assume that an expired offer is still valid. | ||||
| 0% | 0% | 0% | 0% | |
How to passLeave the three selected product pages and the most relevant coupon or cashback pages open in separate tabs for review. Do not add anything to a cart, create an account, enter personal information, or make a purchase. | ||||
Failure modes on this task
14 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Reaching the page1 mode | 2 | 4 | 4 | 1 |
| Delivering the result1 mode | — | — | 1 | — |
| No signal in the judge’s words1 mode · the grader | — | 1 | — | 1 |