task_111Manufacturing

16 verifiers · 15 core · 1 secondary · 16 LLM-graded

TaxonomyManufacturingBuyers and purchasing agents

Task prompt

You are supporting a material sourcing review for an aerospace components manufacturer that builds certified door and hinge mechanism assemblies. Upcoming production lots require aluminum and steel raw materials, including plate, bar, extrusion, and machined-stock inputs. Several material requirements are at risk because available inventory, supplier quotes, lead times, certification status, source approvals, allocations, yield loss, and program priorities do not all align. Production and sourcing leadership need a sourcing plan that identifies which material lots can be used, which supplier quotes should be acted on, which requirements remain at risk, and which program commitments should be prioritized. The objective is not simply to buy the cheapest or fastest available material; the recommendation must distinguish physically available material from material that is actually usable for certified aerospace production. Using the attached material sourcing packet, determine the recommended sourcing and allocation plan for the current review cycle. Identify which aluminum and steel requirements can be covered from compliant on-hand inventory, which require new purchases, which supplier quotes are feasible, and which shortages or compliance issues require escalation. Evaluate the tradeoffs among material specification, temper or condition, product form, mill certification, traceability, approved-source status, country-of-origin or customer restrictions, available quantity after allocation, usable yield, lead time, cost, and program delivery impact. Work only from the attached materials: the material demand forecast, inventory lot and certification snapshot, approved material source list, supplier quote and lead-time matrix, engineering material substitution policy, and program priority / build schedule. Do not invent alternate alloys, tempers, substitutions, engineering approvals, customer waivers, supplier capacity, expedite options, certification corrections, yield assumptions, or allocation changes that are not documented in the source files. Where commercial pressure, supplier promises, or apparent physical availability conflict with material specification, certification, traceability, approved-source, or substitution-policy constraints, treat the documented compliance constraints as authoritative. Do not treat material as usable merely because it is physically on hand or quoted as available. A material lot or quote should be considered usable only if the source materials support that it meets the required alloy/specification, temper or condition, product form, certification and traceability requirements, approved-source rules, and available quantity after existing allocations and yield loss. If a substitution is not explicitly permitted by the attached engineering material substitution policy or documented approval records, do not assume it is allowed. Produce the following deliverables in `/workspace/output/`: - `material_sourcing_recommendation_memo.docx` — a structured recommendation memo for sourcing, production, and quality leadership explaining the recommended material sourcing plan, key constraints, program risks, cost and lead-time tradeoffs, and required escalations. - `material_sourcing_analysis_workbook.xlsx` — a supporting workbook documenting requirement-level material coverage, inventory-lot usability, supplier quote feasibility, allocation decisions, yield-adjusted quantity analysis, at-risk requirements, alternatives considered, and recommended sourcing actions.

System prompt

The system prompt is not available for this bench. Each harness configures its own agent, and the drop records which model ran and how it was tuned — not the prompt it was given.

What it tests

Whether the agent can tell physically available material from usable material. A lot only counts once alloy, temper, product form, mill certification, traceability, approved-source status and post-allocation yield all hold — and the packet states those across six files that do not agree.

Evidence discipline. The brief asks for a plan, but the workbook has to carry the requirement-level chain that shows which lots were rejected and on which rule, or a reviewer cannot check the recommendation at all.

How models fail

Requirement-level traceability is the verifier no trial has ever cleared. Models produce a confident, often defensible plan and omit the why-rejected chain behind it — the answer arrives without the evidence that makes it auditable.

The other two shapes are a deliverable that never gets written at all, and an escalation that is decided rather than escalated: where the packet leaves a rule genuinely unresolved, the trial picks a reading and moves on.

Why it matters

This is certified aerospace production. The paper trail is not documentation of the decision, it is the thing an auditor asks for — and a sourcing plan nobody can retrace is one nobody can act on.

A task is not fully specified by its written instructions. The rest of the spec is in the seed data the run has to read, and in requirements the prompt never spells out — a run only succeeds if it reconstructs enough of the surface.

Source files

xlsx · docx · pdf

Deliverables

docx · xlsx

Golden answer

withheld from the workspace

Verifiers

15 core · 16 LLM-graded

Which of those 6 sources are authoritative and which are planted distractors is not analysed yet — it needs a per-artifact analysis — authoritative vs distractor, extracted requirements, excerpts, the checks that grade each (`tasks/<id>.json spec_surface`, contract 2.8.0).

See the full package