task_111Manufacturing
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
Claude Opus 5claude-code · max
Mean reward
87%All trials
Across 4 trials on this task, reward ran from 82% to 93% — 0 of 4 cleared every core verifier.
- 13.0 / 15avg core verifiers passed
- 1.0 / 1avg secondary passed
- 137avg turns
- 77avg tool calls
- $13.21avg cost
- 52mavg time
Summary · Claude Opus 5
Claude Opus 5 passed none of its 4 trials. Failures cluster on traceability (4/4 trials).
Common pattern
traceability failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 5's failures on this task were classified as Decision Audit Trail Fragmented or Incomplete — 4 verifiers across 4 of 4 trials.
Per-trial notes
GPT-5.6 Solcodex · max
Mean reward
74.5%All trials
Across 4 trials on this task, reward ran from 64% to 93% — 0 of 4 cleared every core verifier.
- 11.0 / 15avg core verifiers passed
- 1.0 / 1avg secondary passed
- 49avg turns
- 37avg tool calls
- $4.40avg cost
- 38mavg time
Summary · GPT-5.6 Sol
GPT-5.6 Sol passed none of its 4 trials. Failures cluster on traceability (4/4 trials).
Common pattern
traceability failed in every unsuccessful trial; what else broke varies across the other 9 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.6 Sol's failures on this task were classified as Decision Audit Trail Fragmented or Incomplete — 4 verifiers across 4 of 4 trials.
Per-trial notes
- Trial 01Missed 1 of 15 core — traceability.
- Trial 02Missed 5 of 15 core — country_of_origin, traceability and residual_vs_moq and 2 more.
- Trial 03Missed 6 of 15 core — requirement_coverage, traceability and residual_vs_moq and 3 more.
- Trial 04Missed 4 of 15 core — allocation_priority, traceability and cost_reconciliation and 1 more.
GPT-5.6 Terracodex · max
Mean reward
82.5%All trials
Across 4 trials on this task, reward ran from 81% to 87% — 0 of 4 cleared every core verifier.
- 12.3 / 15avg core verifiers passed
- 1.0 / 1avg secondary passed
- 40avg turns
- 29avg tool calls
- $1.49avg cost
- 31mavg time
Summary · GPT-5.6 Terra
GPT-5.6 Terra passed none of its 4 trials. Failures cluster on traceability (4/4 trials).
Common pattern
traceability and memo_recommendation failed in every unsuccessful trial; what else broke varies across the other 2 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.6 Terra's failures on this task were classified as Escalation or Tradeoff Communication Incomplete — 6 verifiers across 4 of 4 trials.
Per-trial notes
- Trial 01Missed 3 of 15 core — traceability, escalation_language and memo_recommendation.
- Trial 02Missed 3 of 15 core — traceability, memo_recommendation and no_fabrication_side_effects.
- Trial 03Missed 3 of 15 core — traceability, escalation_language and memo_recommendation.
- Trial 04Missed 2 of 15 core — traceability and memo_recommendation.
Claude Sonnet 5claude-code · max
Mean reward
68.5%All trials
Across 4 trials on this task, reward ran from 57% to 75% — 0 of 4 cleared every core verifier.
- 10.0 / 15avg core verifiers passed
- 0.8 / 1avg secondary passed
- 236avg turns
- 132avg tool calls
- $12.78avg cost
- 61mavg time
Summary · Claude Sonnet 5
Claude Sonnet 5 passed none of its 4 trials. Failures cluster on traceability (4/4 trials).
Common pattern
traceability and residual_vs_moq failed in every unsuccessful trial; what else broke varies across the other 6 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Sonnet 5's failures on this task were classified as Decision Audit Trail Fragmented or Incomplete — 4 verifiers across 4 of 4 trials.
Per-trial notes
- Trial 01Missed 4 of 15 core — traceability, residual_vs_moq and escalation_language and 1 more.
- Trial 02Missed 5 of 15 core — traceability, residual_vs_moq and cost_reconciliation and 2 more, plus 1 secondary.
- Trial 03Missed 4 of 15 core — traceability, residual_vs_moq and memo_recommendation and 1 more.
- Trial 04Missed 7 of 15 core — traceability, residual_vs_moq and cost_reconciliation and 4 more.
Gemini 3.6 Flashgemini-cli · high
Mean reward
38.5%All trials
Across 4 trials on this task, reward ran from 29% to 54% — 0 of 4 cleared every core verifier.
- 5.5 / 15avg core verifiers passed
- 1.0 / 1avg secondary passed
- 117avg turns
- 47avg tool calls
- $1.51avg cost
- 26mavg time
Summary · Gemini 3.6 Flash
Gemini 3.6 Flash passed none of its 4 trials. Failures cluster on country_of_origin (4/4 trials).
Common pattern
country_of_origin, traceability and residual_vs_moq and 2 more failed in every unsuccessful trial; what else broke varies across the other 6 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.6 Flash's failures on this task were classified as Eligibility or Controlling Rule Applied Incorrectly — 9 verifiers across 4 of 4 trials.
Per-trial notes
- Trial 01Missed 11 of 15 core — country_of_origin, source_hold_status and allocation_priority and 8 more.
- Trial 02Missed 10 of 15 core — country_of_origin, allocation_priority and lead_time_feasibility and 7 more.
- Trial 03Missed 7 of 15 core — country_of_origin, source_hold_status and traceability and 4 more.
- Trial 04Missed 10 of 15 core — country_of_origin, source_hold_status and allocation_priority and 7 more.
Gemini 3.1 Progemini-cli · high
Mean reward
25.8%All trials
Across 4 trials on this task, reward ran from 21% to 32% — 0 of 4 cleared every core verifier.
- 3.8 / 15avg core verifiers passed
- 0.8 / 1avg secondary passed
- 38avg turns
- 25avg tool calls
- $0.60avg cost
- 21mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 4 trials. Failures cluster on country_of_origin (4/4 trials).
Common pattern
country_of_origin, source_hold_status and substitution_policy and 5 more failed in every unsuccessful trial; what else broke varies across the other 7 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Required Evidence Fields or Rows Omitted — 31 verifiers across 4 of 4 trials.
Per-trial notes
- Trial 01Missed 12 of 15 core — usable_quantity, country_of_origin and cert_traceability and 9 more.
- Trial 02Missed 10 of 15 core — country_of_origin, source_hold_status and substitution_policy and 7 more, plus 1 secondary.
- Trial 03Missed 12 of 15 core — country_of_origin, cert_traceability and source_hold_status and 9 more.
- Trial 04Missed 11 of 15 core — country_of_origin, cert_traceability and source_hold_status and 8 more.