task_116Retail trade
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one trial to measure.
Core metrics
Core Pass %
Mean share of the core verifiers passing, averaged over runs. N = 4 trials per model.
73%
Claude Opus 5max
72%
GPT-5.6 Solmax
63%
GPT-5.6 Terramax
53%
Claude Sonnet 5max
47%
Gemini 3.6 Flashhigh
25%
Gemini 3.1 Prohigh
Verifier Pass %
Mean fraction of all verifiers passing, across trials. N = 4 trials per model.
71%
GPT-5.6 Solmax
68%
Claude Opus 5max
64%
GPT-5.6 Terramax
53%
Claude Sonnet 5max
44%
Gemini 3.6 Flashhigh
22%
Gemini 3.1 Prohigh
Avg Cost
Mean API spend across trials; the line under a value is its observed range. N = 4 trials per model.
$10.71$8.19–$11.67
Claude Opus 5max
$8.53$6.26–$9.40
Claude Sonnet 5max
$4.97$3.32–$6.25
GPT-5.6 Solmax
$1.57$1.39–$1.74
GPT-5.6 Terramax
$1.24$1.00–$1.46
Gemini 3.6 Flashhigh
$0.35$0.02–$0.59
Gemini 3.1 Prohigh
Extended metrics
Secondary Pass %
Mean fraction of the secondary verifiers passing. N = 4 trials per model.
67%
GPT-5.6 Solmax
67%
GPT-5.6 Terramax
50%
Claude Sonnet 5max
42%
Claude Opus 5max
33%
Gemini 3.6 Flashhigh
8%
Gemini 3.1 Prohigh
Median Time
Wall-clock minutes, median across trials. N = 4 trials per model.
43m
Claude Sonnet 5max
42m
Claude Opus 5max
27m
GPT-5.6 Solmax
24m
Gemini 3.6 Flashhigh
21m
GPT-5.6 Terramax
9m
Gemini 3.1 Prohigh
Tool Calls
Tool invocations, median across trials — counted from the trajectories. N = 4 trials per model.
102
Claude Sonnet 5max
78
Claude Opus 5max
55
GPT-5.6 Solmax
49
Gemini 3.6 Flashhigh
39
GPT-5.6 Terramax
17
Gemini 3.1 Prohigh
What the reward cost on this task
Reward vs Cost
Mean reward against mean spend per trial — top-left is more reward for less.
- Claude Opus 5
- GPT-5.6 Sol
- GPT-5.6 Terra
- Claude Sonnet 5
- Gemini 3.6 Flash
- Gemini 3.1 Pro
$0.35Cost / trial$10.71
Tokens Consumed vs Reward
Mean reward against cumulative input tokens per trial — not peak context.
- Claude Opus 5
- GPT-5.6 Sol
- GPT-5.6 Terra
- Claude Sonnet 5
- Gemini 3.6 Flash
- Gemini 3.1 Pro
476kInput tokens / trial17.5M
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5maxn=4 | GPT-5.6 Solmaxn=4 | GPT-5.6 Terramaxn=4 | Claude Sonnet 5maxn=4 | Gemini 3.6 Flashhighn=4 | Gemini 3.1 Prohighn=4 |
|---|---|---|---|---|---|---|
| core (15) | ||||||
| 100% | 100% | 75% | 100% | 50% | 50% | |
Why it mattersSushi is the most tempting vendor-led trap because supplier notes conflict with signed launch terms. How to passChecks whether the memo makes the correct route-limited sushi recommendation. | ||||||
| 100% | 100% | 50% | 100% | 100% | 0% | |
Why it mattersA professional reset recommendation must catch the specific physical overfills before Space Planning releases the case set. How to passChecks whether the memo identifies the exact buyer-proposal capacity conflicts by cluster and amount. | ||||||
| 100% | 100% | 100% | 75% | 100% | 50% | |
Why it mattersRetail food-to-go resets must protect mission coverage while avoiding a poor substitute. How to passChecks whether the memo protects credible vegetarian coverage with the correct item logic. | ||||||
| 100% | 100% | 100% | 100% | 100% | 25% | |
Why it mattersKeeping or ambiguously deferring an unsupported winter perishable in July would create avoidable shrink and execution confusion. How to passChecks whether the memo makes a concrete all-cluster zero-facing decision for the winter seasonal SKU. | ||||||
| 75% | 0% | 25% | 0% | 0% | 0% | |
Why it mattersThe task requires a decision that protects lunch traffic and profit, not just deletes weak SKUs. How to passChecks whether the memo prioritizes the core traffic and profitable launch items. | ||||||
| 100% | 100% | 100% | 75% | 50% | 0% | |
Why it mattersThe task's central trap is that supplier projections and proposal comments conflict with the authoritative demand and vendor-term sources. How to passChecks whether the memo substantively applies the source hierarchy instead of merely naming it. | ||||||
| 50% | 50% | 100% | 75% | 75% | 75% | |
Why it mattersThe first four weeks must be executable using only the already available items, case capacity, and signed operating terms. How to passChecks whether the memo avoids invented SKUs, capacity, delivery days, credits, and route assumptions. | ||||||
| 0% | 25% | 0% | 0% | 0% | 0% | |
Why it mattersThe senior merchant needs a first-four-week reset decision that Space Planning and field operations can execute without interpreting generic advice. How to passChecks whether the memo provides an executable revised cluster-level assortment with slot totals. | ||||||
| 100% | 75% | 100% | 75% | 25% | 75% | |
Why it mattersThe workbook must support the memo with granular decisions, not just summary prose. How to passChecks whether the workbook contains a complete cluster-SKU decision matrix. | ||||||
| 0% | 25% | 0% | 0% | 0% | 0% | |
Why it mattersThis keeps the load-bearing final allocation check while avoiding double-counting the Egg Salad, Turkey Cranberry, Caprese, and Hummus decisions. How to passChecks workbook final facing allocations for core traffic, base assortment, profit-launch, and route-limited SKUs not separately covered by weak-SKU or vegetarian-distribution checks. | ||||||
| 0% | 0% | 0% | 0% | 0% | 0% | |
Why it mattersA capacity-constrained reset must prove both that the original proposal was corrected and that the final plan fits live case capacity. How to passChecks whether the workbook numerically reconciles proposal capacity conflicts and final capacity compliance. | ||||||
| 100% | 100% | 100% | 75% | 100% | 75% | |
Why it mattersLeaving these SKUs in the final workbook would contradict the economics and handoff. How to passChecks whether the workbook removes the two weak legacy/seasonal SKUs. | ||||||
| 75% | 100% | 25% | 25% | 0% | 25% | |
Why it mattersThis captures the nuanced vegetarian tradeoff that generic models often miss. How to passChecks whether the workbook handles Caprese and Hummus distribution correctly. | ||||||
| 100% | 100% | 100% | 50% | 0% | 0% | |
Why it mattersVendor-term accuracy is essential because unsupported credits or route eligibility would make the reset economics and execution plan unusable. How to passChecks whether the workbook applies each signed vendor term with exact item, cluster, and credit constraints. | ||||||
| 100% | 100% | 75% | 50% | 100% | 0% | |
Why it mattersA supporting workbook must let reviewers verify margin, shrink, credit, OOS, and capacity logic instead of trusting unexplained conclusions. How to passChecks whether the workbook includes auditable numeric economics and calculation evidence behind the decisions. | ||||||
| secondary (3) | ||||||
| 0% | 25% | 25% | 0% | 0% | 0% | |
Why it mattersPerishable reset discipline means avoiding shrink even when that leaves small amounts of capacity unused. How to passChecks whether the memo accepts specific underfilled capacity rather than force-filling weak perishables. | ||||||
| 100% | 75% | 75% | 50% | 0% | 0% | |
Why it mattersThe requested decision is not only a final assortment; it must show which proposed changes were accepted, changed, or rejected. How to passChecks whether the workbook provides a substantive accept/change/reject reconciliation against the buyer proposal. | ||||||
| 25% | 100% | 100% | 100% | 100% | 25% | |
Why it mattersExtra assumptions or unrelated operational actions can make the recommendation unusable even when the core analysis appears complete. How to passChecks workbook scope control and absence of materially unsolicited tabs, assumptions, or side actions. | ||||||
Failure modes on this task
200 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5max | GPT-5.6 Solmax | GPT-5.6 Terramax | Claude Sonnet 5max | Gemini 3.6 Flashhigh | Gemini 3.1 Prohigh |
|---|---|---|---|---|---|---|
| Reasoning and Computation3 modes | 17 | 13 | 20 | 23 | 25 | 16 |
| Deliverable Completion and Structure2 modes | — | 2 | 1 | 5 | 4 | 29 |
| Presentation and Communication3 modes | 4 | 4 | 5 | 5 | 6 | 11 |
| Input and Instruction Comprehension1 mode | 2 | 2 | — | 1 | 1 | — |
| Grader Error1 mode · the grader | — | — | — | — | 4 | — |