task_116Retail trade
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
Claude Opus 5claude-code · max
Mean reward
65.8%All trials
Across 4 trials on this task, reward ran from 63% to 70% — 0 of 4 cleared every core verifier.
- 11.0 / 15avg core verifiers passed
- 1.3 / 3avg secondary passed
- 136avg turns
- 81avg tool calls
- $10.71avg cost
- 41mavg time
Summary · Claude Opus 5
Claude Opus 5 passed none of its 4 trials. Failures cluster on execution (4/4 trials).
Common pattern
execution, capacity_discipline and core_final_facings and 1 more failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 5's failures on this task were classified as Allocation or Selection Plan Incorrect — 8 verifiers across 4 of 4 trials.
Per-trial notes
- Trial 01Missed 4 of 15 core — scope_control, execution and core_final_facings and 1 more, plus 2 secondary.
- Trial 02Missed 4 of 15 core — traffic_profit, execution and core_final_facings and 1 more, plus 2 secondary.
- Trial 03Missed 5 of 15 core — scope_control, execution and core_final_facings and 2 more, plus 1 secondary.
- Trial 04Missed 3 of 15 core — execution, core_final_facings and capacity, plus 2 secondary.
GPT-5.6 Solcodex · max
Mean reward
69%All trials
Across 4 trials on this task, reward ran from 55% to 83% — 0 of 4 cleared every core verifier.
- 10.8 / 15avg core verifiers passed
- 2.0 / 3avg secondary passed
- 63avg turns
- 49avg tool calls
- $4.97avg cost
- 28mavg time
Summary · GPT-5.6 Sol
GPT-5.6 Sol passed none of its 4 trials. Failures cluster on traffic_profit (4/4 trials).
Common pattern
traffic_profit and capacity failed in every unsuccessful trial; what else broke varies across the other 6 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.6 Sol's failures on this task were classified as Quantity, Cost, Capacity, or Rate Calculation Wrong — 7 verifiers across 4 of 4 trials.
Per-trial notes
- Trial 01Missed 4 of 15 core — traffic_profit, execution and core_final_facings and 1 more, plus 1 secondary.
- Trial 02Missed 6 of 15 core — traffic_profit, scope_control and execution and 3 more, plus 2 secondary.
- Trial 03Missed 3 of 15 core — traffic_profit, scope_control and capacity.
- Trial 04Missed 4 of 15 core — traffic_profit, execution and core_final_facings and 1 more, plus 1 secondary.
GPT-5.6 Terracodex · max
Mean reward
59.2%All trials
Across 4 trials on this task, reward ran from 49% to 63% — 0 of 4 cleared every core verifier.
- 9.5 / 15avg core verifiers passed
- 2.0 / 3avg secondary passed
- 50avg turns
- 39avg tool calls
- $1.57avg cost
- 21mavg time
Summary · GPT-5.6 Terra
GPT-5.6 Terra passed none of its 4 trials. Failures cluster on execution (4/4 trials).
Common pattern
execution, core_final_facings and capacity failed in every unsuccessful trial; what else broke varies across the other 7 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.6 Terra's failures on this task were classified as Allocation or Selection Plan Incorrect — 10 verifiers across 4 of 4 trials.
Per-trial notes
- Trial 01Missed 7 of 15 core — sushi_route, traffic_profit and execution and 4 more, plus 1 secondary.
- Trial 02Missed 5 of 15 core — capacity, execution and core_final_facings and 2 more, plus 1 secondary.
- Trial 03Missed 5 of 15 core — capacity, traffic_profit and execution and 2 more, plus 1 secondary.
- Trial 04Missed 5 of 15 core — traffic_profit, execution and core_final_facings and 2 more, plus 1 secondary.
Claude Sonnet 5claude-code · max
Mean reward
51%All trials
Across 4 trials on this task, reward ran from 42% to 59% — 0 of 4 cleared every core verifier.
- 8.0 / 15avg core verifiers passed
- 1.5 / 3avg secondary passed
- 187avg turns
- 99avg tool calls
- $8.53avg cost
- 44mavg time
Summary · Claude Sonnet 5
Claude Sonnet 5 passed none of its 4 trials. Failures cluster on traffic_profit (4/4 trials).
Common pattern
traffic_profit, execution and capacity_discipline and 2 more failed in every unsuccessful trial; what else broke varies across the other 9 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Sonnet 5's failures on this task were classified as Quantity, Cost, Capacity, or Rate Calculation Wrong — 10 verifiers across 4 of 4 trials.
Per-trial notes
- Trial 01Missed 8 of 15 core — vegetarian_role, traffic_profit and execution and 5 more, plus 2 secondary.
- Trial 02Missed 7 of 15 core — traffic_profit, trust_hierarchy and execution and 4 more, plus 1 secondary.
- Trial 03Missed 6 of 15 core — traffic_profit, scope_control and execution and 3 more, plus 1 secondary.
- Trial 04Missed 7 of 15 core — traffic_profit, execution and workbook_structure and 4 more, plus 2 secondary.
Gemini 3.6 Flashgemini-cli · high
Mean reward
43%All trials
Across 4 trials on this task, reward ran from 37% to 49% — 0 of 4 cleared every core verifier.
- 7.0 / 15avg core verifiers passed
- 1.0 / 3avg secondary passed
- 99avg turns
- 50avg tool calls
- $1.24avg cost
- 25mavg time
Summary · Gemini 3.6 Flash
Gemini 3.6 Flash passed none of its 4 trials. Failures cluster on traffic_profit (4/4 trials).
Common pattern
traffic_profit, execution and capacity_discipline and 5 more failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.6 Flash's failures on this task were classified as Allocation or Selection Plan Incorrect — 10 verifiers across 4 of 4 trials.
Per-trial notes
- Trial 01Missed 8 of 15 core — traffic_profit, trust_hierarchy and scope_control and 5 more, plus 2 secondary.
- Trial 02Missed 9 of 15 core — sushi_route, traffic_profit and trust_hierarchy and 6 more, plus 2 secondary.
- Trial 03Missed 7 of 15 core — traffic_profit, execution and workbook_structure and 4 more, plus 2 secondary.
- Trial 04Missed 8 of 15 core — sushi_route, traffic_profit and execution and 5 more, plus 2 secondary.
Gemini 3.1 Progemini-cli · high
Mean reward
22.5%All trials
Across 4 trials on this task, reward ran from 0% to 36% — 0 of 4 cleared every core verifier.
- 3.8 / 15avg core verifiers passed
- 0.3 / 3avg secondary passed
- 23avg turns
- 17avg tool calls
- $0.35avg cost
- 7mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 4 trials. Failures cluster on capacity (4/4 trials).
Common pattern
capacity, traffic_profit and trust_hierarchy and 7 more failed in every unsuccessful trial; what else broke varies across the other 8 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Required Evidence Fields or Rows Omitted — 11 verifiers across 3 of 4 trials.
Per-trial notes
- Trial 01Missed 12 of 15 core — sushi_route, capacity and vegetarian_role and 9 more, plus 2 secondary.
- Trial 02Missed 9 of 15 core — capacity, seasonal and traffic_profit and 6 more, plus 3 secondary.
- Trial 03Missed 9 of 15 core — capacity, traffic_profit and trust_hierarchy and 6 more, plus 3 secondary.
- Trial 04Missed 15 of 15 core — sushi_route, capacity and vegetarian_role and 12 more, plus 3 secondary.