task_111Manufacturing
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one trial to measure.
Core metrics
Core Pass %
Mean share of the core verifiers passing, averaged over runs. N = 4 trials per model.
87%
Claude Opus 5max
82%
GPT-5.6 Terramax
73%
GPT-5.6 Solmax
67%
Claude Sonnet 5max
37%
Gemini 3.6 Flashhigh
25%
Gemini 3.1 Prohigh
Verifier Pass %
Mean fraction of all verifiers passing, across trials. N = 4 trials per model.
88%
Claude Opus 5max
83%
GPT-5.6 Terramax
75%
GPT-5.6 Solmax
67%
Claude Sonnet 5max
41%
Gemini 3.6 Flashhigh
28%
Gemini 3.1 Prohigh
Avg Cost
Mean API spend across trials; the line under a value is its observed range. N = 4 trials per model.
$13.21$10.52–$18.68
Claude Opus 5max
$12.78$8.20–$16.55
Claude Sonnet 5max
$4.40$3.86–$5.16
GPT-5.6 Solmax
$1.51$1.16–$2.06
Gemini 3.6 Flashhigh
$1.49$1.32–$1.68
GPT-5.6 Terramax
$0.60$0.37–$0.83
Gemini 3.1 Prohigh
Extended metrics
Secondary Pass %
Mean fraction of the secondary verifiers passing. N = 4 trials per model.
100%
Claude Opus 5max
100%
GPT-5.6 Solmax
100%
GPT-5.6 Terramax
100%
Gemini 3.6 Flashhigh
75%
Claude Sonnet 5max
75%
Gemini 3.1 Prohigh
Median Time
Wall-clock minutes, median across trials. N = 4 trials per model.
65m
Claude Sonnet 5max
50m
Claude Opus 5max
37m
GPT-5.6 Solmax
31m
GPT-5.6 Terramax
26m
Gemini 3.6 Flashhigh
18m
Gemini 3.1 Prohigh
Tool Calls
Tool invocations, median across trials — counted from the trajectories. N = 4 trials per model.
138
Claude Sonnet 5max
70
Claude Opus 5max
46
Gemini 3.6 Flashhigh
36
GPT-5.6 Solmax
30
GPT-5.6 Terramax
23
Gemini 3.1 Prohigh
What the reward cost on this task
Reward vs Cost
Mean reward against mean spend per trial — top-left is more reward for less.
- Claude Opus 5
- GPT-5.6 Sol
- GPT-5.6 Terra
- Claude Sonnet 5
- Gemini 3.6 Flash
- Gemini 3.1 Pro
$0.60Cost / trial$13.21
Tokens Consumed vs Reward
Mean reward against cumulative input tokens per trial — not peak context.
- Claude Opus 5
- GPT-5.6 Sol
- GPT-5.6 Terra
- Claude Sonnet 5
- Gemini 3.6 Flash
- Gemini 3.1 Pro
875kInput tokens / trial29.3M
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5maxn=4 | GPT-5.6 Solmaxn=4 | GPT-5.6 Terramaxn=4 | Claude Sonnet 5maxn=4 | Gemini 3.6 Flashhighn=4 | Gemini 3.1 Prohighn=4 |
|---|---|---|---|---|---|---|
| core (15) | ||||||
| 100% | 100% | 100% | 100% | 100% | 75% | |
Why it mattersCoverage must be assessed on usable yield-adjusted quantity net of allocations, and the original attempt's central risk is treating gross or physically-present pounds as usable. How to passChecks that the analysis workbook reports usable on-hand quantity using the formula (On Hand - Allocated) x Yield Factor for inventory lots rather than gross on-hand pounds. | ||||||
| 100% | 75% | 100% | 100% | 0% | 0% | |
Why it mattersCountry-of-origin restrictions are binding contractual constraints, and the cheapest/fastest options for several critical lines are non-qualifying-origin material that must not be selected. How to passChecks that material melted in China or Russia is excluded for the DFARS programs (Raven, Sable) and the CUST-EU programs (Meridian, Tundra) regardless of price or certificate quality. | ||||||
| 100% | 100% | 100% | 100% | 100% | 25% | |
Why it mattersEN 10204 3.1/3.2 certification and full traceability are minimum eligibility gates, and the original attempt risks accepting C-of-C or non-traceable material that is physically attractive. How to passChecks that Certificate-of-Conformance-only lots/quotes and partial-traceability lots are excluded from certified-production coverage. | ||||||
| 75% | 100% | 100% | 100% | 25% | 0% | |
Why it mattersApproved-source status and hold status are hard eligibility gates, and otherwise-attractive USA-origin material from these sources must not be planned against requirements. How to passChecks that sources in Suspended, Conditional, or Not Approved status, and material under an open quality/MRB or receiving-inspection hold, are excluded from new commitments. | ||||||
| 100% | 100% | 100% | 100% | 100% | 0% | |
Why it mattersThe substitution policy is exhaustive and prohibits interchange (e.g., T651 vs T7351, /11 vs /12, bar vs plate vs extrusion), so a model must not silently treat near-match material as covering a callout. How to passChecks that no alloy, temper/condition, product-form, or specification sub-spec substitution is applied beyond same-spec oversize/remnant cutdown. | ||||||
| 100% | 75% | 100% | 100% | 25% | 75% | |
Why it mattersProgram-priority allocation is an explicit contention rule, and getting it wrong misassigns scarce compliant stock between competing programs. How to passChecks that contested inventory lots usable by more than one requirement are allocated to the higher-priority (lower rank number) program first, with lower-priority shortfalls flowing to new purchase. | ||||||
| 100% | 100% | 100% | 100% | 25% | 50% | |
Why it mattersLead-time feasibility is an explicit gate, and the original attempt risks counting a compliant quote that docks on or after need-by as resolving a requirement on schedule. How to passChecks that a supplier quote is treated as feasible only when its quoted dock/receipt date precedes the requirement need-by date. | ||||||
| 100% | 75% | 100% | 100% | 100% | 100% | |
Why it mattersRequirement-level coverage classification is the core analytical output the task asks for and must be complete across all demand lines. How to passChecks that the workbook classifies each of the 26 requirements into a coverage disposition (covered from compliant inventory, partial inventory plus buy, or full new purchase) at the requirement-ID level. | ||||||
| 0% | 0% | 0% | 0% | 0% | 0% | |
Why it mattersRequirement-level traceability is an explicit auditability obligation and is exactly where a weaker solution omits the why-rejected evidence chain. How to passChecks that for each at-risk requirement the workbook records the selected inventory lot(s) with usable quantity, the rejected lots with reasons, the selected quote, the rejected quote alternatives with reasons, the residual after inventory, and the resulting purchase quantity and cost. | ||||||
| 100% | 50% | 100% | 0% | 0% | 0% | |
Why it mattersConflating need with MOQ-inflated buy quantity was a defect in the earlier attempt and obscures both true demand and committed spend. How to passChecks that requirement residual quantity is reported separately from the minimum-order-quantity-adjusted purchase quantity, with the excess identified. | ||||||
| 100% | 75% | 100% | 50% | 25% | 0% | |
Why it mattersA reconciled, self-consistent committed total was an explicit fix from the earlier attempt, where the memo and PO tab disagreed on the headline figure and shared lines risked double-charging. How to passChecks that the workbook presents a single authoritative total committed new-buy cost with no conflicting totals and a cost chain that reconciles, with shared quote lines not double-counted. | ||||||
| 75% | 75% | 50% | 50% | 25% | 25% | |
Why it mattersClarifying escalation language so that 'every requirement has a feasible path' is not conflated with quality holds awaiting action was an explicit obligation from the earlier revision. How to passChecks that the memo states there is no hard sourcing dead-end while separately listing quality/MRB/process escalations. | ||||||
| 100% | 50% | 0% | 25% | 0% | 0% | |
Why it mattersThe task's core decision principle is distinguishing usable material from merely cheap/fast/available material, and the memo is where that reasoning must be made legible to leadership. How to passChecks that the memo recommends a sourcing/allocation plan that prioritizes compliance over lowest price or shortest lead time and explicitly identifies rejected cheaper/faster non-compliant options. | ||||||
| 75% | 75% | 75% | 25% | 0% | 25% | |
Why it mattersThe task forbids inventing options beyond the source packet, and penalizing fabricated content or unrequested side-additions protects against hallucinated uplift that would look like a stronger answer. How to passChecks that the deliverables do not introduce fabricated alloys, tempers, substitutions, approvals, waivers, expedite options, or other material not present in the source files, and do not add materially unsolicited deliverables. | ||||||
| 75% | 50% | 100% | 50% | 25% | 0% | |
Why it mattersThe shared-PO double-counting failure mode (inflating the committed total by billing one quote line at each requirement's minimum) is exactly where the earlier attempt produced an unreconciled headline figure, and catching it is a fair, materially distinct core check. How to passChecks that the sum of per-requirement residual purchase needs reconciles to the consolidated purchase-order quantities and committed cost, so shared quote lines are not double-counted across the requirements they serve. | ||||||
| secondary (1) | ||||||
| 100% | 100% | 100% | 75% | 100% | 75% | |
Why it mattersIf the memo asserts headline counts or program restriction classes, they must be accurate and self-consistent; this guards against a confidently-stated but wrong summary rather than mandating an unprompted structure. How to passChecks that any coverage counts or per-program restriction classes the memo chooses to state are internally consistent and correct, without forcing the memo to include specific counts. | ||||||
Failure modes on this task
140 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5max | GPT-5.6 Solmax | GPT-5.6 Terramax | Claude Sonnet 5max | Gemini 3.6 Flashhigh | Gemini 3.1 Prohigh |
|---|---|---|---|---|---|---|
| Deliverable Completion and Structure2 modes | 5 | 8 | 4 | 8 | 9 | 31 |
| Reasoning and Computation4 modes | 2 | 3 | 1 | 4 | 19 | 4 |
| Presentation and Communication3 modes | 1 | 3 | 6 | 6 | 7 | 8 |
| Input and Instruction Comprehension2 modes | — | — | — | 3 | 3 | — |
| Grader Error1 mode · the grader | — | 2 | — | — | — | 3 |