Failure Modes
Every verifier failure is counted under exactly one mode — 11,391 of them across this drop.
- Failure stage
- 5 of them — WHERE in the work a trial broke, at the earliest point that broke.
- Failure mode
- 20 of them — WHAT went wrong, named, inside one stage. Every mode belongs to exactly one stage.
Passing most verifiers is not the same as passing a task
Claude Opus 5 leads on individual verifiers with 76% of them passed, and clears every core verifier on 21% of its trials. The board's headline figure and its pass rate are answering two different questions, and the second is the one a buyer is asking.
Most failures land at Reasoning and Computation
The inputs are available, but extraction, rule application, selection, calculation, or reconciliation produces the wrong substantive result. It accounts for 4,429 of the drop's 11,391 verifier failures — more than any other stage, and every model on the board is in that number.
The most common single failure is Required Evidence Fields or Rows Omitted
It accounts for 2,398 of the drop's 11,391 verifier failures, and shows up on 3 of the 3 tasks this drop publishes — so it is a property of the work rather than of one task. From the runs: A reservation table contained client-slot pairs but omitted start time and reserved-minutes fields required to verify the mappings.
How hard an agent works does not predict how well it does
GPT-5.6 Sol and Gemini 3.6 Flash make almost the same number of tool calls — 47 and 54, at $4.75 and $1.31 — and score 72.5% against 53.7%, 18.8 pts apart. What separates them is what they do with what they read, not how much of it they touch.
Claude Opus 5 carries the smallest share of this drop's failures
1,182 verifier failures out of the 11,012 charged to any model, over 266 trials — 11% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Its characteristic failure is Off-Scope Advice Added
Tagged at Presentation and Communication in 2% of its 266 trials, against 1% for the median of the other 5 models — the widest gap it opens anywhere in the taxonomy. From the trials: A reconciliation memo added diagnostic monitoring and new case-management instructions beyond the requested routing report.
It handles Required Evidence Fields or Rows Omitted better than the field
Tagged at Deliverable Completion and Structure in 0% of its 266 trials, against 3% for the median of the other 5 models. The same measurement as above, read the other way — a failure reading that only ever accuses is not a reading.
GPT-5.6 Sol carries the 5th largest share of this drop's failures
1,275 verifier failures out of the 11,012 charged to any model, over 276 trials — 12% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Its characteristic failure is Off-Scope Advice Added
Tagged at Presentation and Communication in 1% of its 276 trials, against 1% for the median of the other 5 models — the widest gap it opens anywhere in the taxonomy. From the trials: A reconciliation memo added diagnostic monitoring and new case-management instructions beyond the requested routing report.
It handles Required Evidence Fields or Rows Omitted better than the field
Tagged at Deliverable Completion and Structure in 1% of its 276 trials, against 3% for the median of the other 5 models. The same measurement as above, read the other way — a failure reading that only ever accuses is not a reading.
GPT-5.6 Terra carries the 4th largest share of this drop's failures
1,495 verifier failures out of the 11,012 charged to any model, over 276 trials — 14% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Its characteristic failure is Required Numeric Summary Missing or Inconsistent
Tagged at Presentation and Communication in 1% of its 276 trials, against 0% for the median of the other 5 models — the widest gap it opens anywhere in the taxonomy. From the trials: A memo described four clusters as over capacity but omitted each proposal-specific used-versus-capacity comparison.
It handles Required Evidence Fields or Rows Omitted better than the field
Tagged at Deliverable Completion and Structure in 0% of its 276 trials, against 3% for the median of the other 5 models. The same measurement as above, read the other way — a failure reading that only ever accuses is not a reading.
Claude Sonnet 5 carries the 3rd largest share of this drop's failures
1,515 verifier failures out of the 11,012 charged to any model, over 262 trials — 14% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Its characteristic failure is Required Evidence Fields or Rows Omitted
Tagged at Deliverable Completion and Structure in 3% of its 262 trials, against 1% for the median of the other 5 models — the widest gap it opens anywhere in the taxonomy. From the trials: A reservation table contained client-slot pairs but omitted start time and reserved-minutes fields required to verify the mappings.
It handles Off-Scope Advice Added better than the field
Tagged at Presentation and Communication in 1% of its 262 trials, against 1% for the median of the other 5 models. The same measurement as above, read the other way — a failure reading that only ever accuses is not a reading.
Gemini 3.6 Flash carries the 2nd largest share of this drop's failures
2,353 verifier failures out of the 11,012 charged to any model, over 275 trials — 21% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Its characteristic failure is Eligibility or Controlling Rule Applied Incorrectly
Tagged at Reasoning and Computation in 4% of its 275 trials, against 1% for the median of the other 5 models — the widest gap it opens anywhere in the taxonomy. From the trials: China-melt inventory was allocated to a requirement whose governing origin rule excluded it.
It handles Required Numeric Summary Missing or Inconsistent better than the field
Tagged at Presentation and Communication in 0% of its 275 trials, against 0% for the median of the other 5 models. The same measurement as above, read the other way — a failure reading that only ever accuses is not a reading.
Gemini 3.1 Pro carries the largest share of this drop's failures
3,192 verifier failures out of the 11,012 charged to any model, over 276 trials — 29% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Its characteristic failure is Required Evidence Fields or Rows Omitted
Tagged at Deliverable Completion and Structure in 4% of its 276 trials, against 1% for the median of the other 5 models — the widest gap it opens anywhere in the taxonomy. From the trials: A reservation table contained client-slot pairs but omitted start time and reserved-minutes fields required to verify the mappings.
It handles Decision Audit Trail Fragmented or Incomplete better than the field
Tagged at Deliverable Completion and Structure in 0% of its 276 trials, against 1% for the median of the other 5 models. The same measurement as above, read the other way — a failure reading that only ever accuses is not a reading.
Failure modes
Each (trial, verifier) failure is assigned to exactly ONE mode, at the EARLIEST point in the pipeline that broke. The bar under each stage is its share of every classified failure — how much of the drop it accounts for. The dots are how often each model’s own trials were tagged with it, on one scale across every card. Two different denominators, so a large stage is not the same claim as a model that hits one often.
- Claude Opus 5max
- Claude Sonnet 5max
- Gemini 3.1 Prohigh
- Gemini 3.6 Flashhigh
- GPT-5.6 Solmax
- GPT-5.6 Terramax
Reasoning and Computation38.9%
The inputs are available, but extraction, rule application, selection, calculation, or reconciliation produces the wrong substantive result.
6 modes- Quantity, Cost, Capacity, or Rate Calculation Wrong2 tasks · 2021 failures · 17.7% · Capability limitWith the correct source rows and eligibility decisions available, is a required order quantity, excess, cost, credit, capacity, production-mix value, threshold flag, or weighted rate calculated incorrectly?Seen ontask_111task_116
- Eligibility or Controlling Rule Applied Incorrectly3 tasks · 771 failures · 6.8% · Capability limitDoes the artifact keep an option eligible, exclude a required option, or elevate an advisory rule contrary to a documented origin, certification, source-status, schedule, route, seasonality, hold, lock, or authority rule?Seen ontask_111task_116task_53
- Source Mapping or Baseline Extraction Incomplete593 failures · 5.2% · Capability limitIs a required source row, field, or exact baseline set absent, shifted to the wrong model field, hardcoded over, or omitted from the analysis before any downstream calculation?
- Decision Reconciliation or Conclusion Wrong2 tasks · 500 failures · 4.4% · Capability limitDo detailed rows support one result while another table, summary, recommendation, or action classification asserts a materially conflicting result?Seen ontask_111task_53
- Allocation or Selection Plan Incorrect3 tasks · 446 failures · 3.9% · Capability limitAfter eligibility is established, does the artifact assign a scarce resource, facing, shelf position, assortment item, or appointment slot to the wrong recipient or produce the wrong required final set?Seen ontask_111task_116task_53
- Yield or Consumption Path Modeled Incorrectly98 failures · 0.9% · Capability limitDoes the model duplicate a make-item loss across routing operations, omit a consuming or higher-level yield, or otherwise apply survival factors at the wrong points in the BOM and routing path?
Trials tagged, per model33 pts spread
0%33%67%100%
Claude Opus 5max6 / 12 trialsClaude Sonnet 5max6 / 12 trialsGemini 3.1 Prohigh7 / 12 trialsGemini 3.6 Flashhigh10 / 12 trialsGPT-5.6 Solmax6 / 12 trialsGPT-5.6 Terramax6 / 12 trialsDeliverable Completion and Structure28.6%
The substantive work is complete or partly complete, but the required files, audit topology, fields, rows, formulas, or readable values are not delivered in the required form.
4 modes- Required Evidence Fields or Rows Omitted3 tasks · 2398 failures · 21.1% · Harness-fixableIs the decision generally present, but the required exact identifiers, comparison rows, operational columns, governing terms, before-and-after values, or validation fields needed to verify it are absent?Seen ontask_111task_116task_53
- Required Deliverable Missing1 task · 528 failures · 4.6% · Capability limitWhen the run ends, is any required output file absent from its exact required path?Seen ontask_116
- Decision Audit Trail Fragmented or Incomplete1 task · 222 failures · 1.9% · Harness-fixableMust a reviewer join separate sheets or infer unstated links to connect an item or requirement with its inputs, alternatives, decision, rationale, quantities, and resulting economics?Seen ontask_111
- Workbook Formula or Materialization Failure108 failures · 0.9% · Harness-fixableDoes the workbook contain the intended model structure but use malformed cell references, inconsistent summary formulas, or formula-only outputs without readable cached results?
Trials tagged, per model67 pts spread
0%33%67%100%
Claude Opus 5max4 / 12 trialsClaude Sonnet 5max8 / 12 trialsGemini 3.1 Prohigh12 / 12 trialsGemini 3.6 Flashhigh8 / 12 trialsGPT-5.6 Solmax5 / 12 trialsGPT-5.6 Terramax5 / 12 trialsPresentation and Communication25.7%
The underlying analysis or decision exists, but the final memo or summary omits, contradicts, or improperly expands the communication the task requires.
5 modes- Required Rationale or Authority Statement Omitted3 tasks · 1317 failures · 11.6% · Harness-fixableIs the underlying decision correct or substantially present, but the memo fails to state a required rule hierarchy, source-governance principle, item-specific reason, signed-term basis, or operating condition?Seen ontask_111task_116task_53
- Required Numeric Summary Missing or Inconsistent2 tasks · 631 failures · 5.5% · Harness-fixableAre the detailed rows available and substantially correct while the memo or executive summary omits required counts, capacities, overages, unused amounts, totals, or states a value inconsistent with those rows?Seen ontask_111task_116
- Escalation or Tradeoff Communication Incomplete1 task · 483 failures · 4.2% · Harness-fixableDoes the final communication omit or conflate a required escalation, rejected attractive alternative, disqualifying reason, residual risk, or distinction between a feasible plan and separate process actions?Seen ontask_111
- Unsupported Factual Claim Added346 failures · 3.0% · Harness-fixableDoes the final narrative state a numeric benchmark, history, impact, legal consequence, clinical fact, or commercial assertion that is neither sourced from the packet nor derived by the model?
- Off-Scope Advice Added2 tasks · 148 failures · 1.3% · TradeoffWith the requested reconciliation or analysis otherwise present, does the final communication add unrequested clinical, treatment, monitoring, referral, release, or operational workflow directives?Seen ontask_116task_53
Trials tagged, per model42 pts spread
0%33%67%100%
Claude Opus 5max7 / 12 trialsClaude Sonnet 5max12 / 12 trialsGemini 3.1 Prohigh11 / 12 trialsGemini 3.6 Flashhigh12 / 12 trialsGPT-5.6 Solmax11 / 12 trialsGPT-5.6 Terramax11 / 12 trialsInput and Instruction Comprehension3.5%
The artifact adopts an unsupported premise or resolves the governing instructions incorrectly before extracting data or making decisions.
2 modes- Unsupported Permission or Source Fact Introduced2 tasks · 353 failures · 3.1% · Harness-fixableDoes the recommendation depend on a waiver, approval, route, credit, corrective action, operating permission, or factual premise that is not documented in the packet?Seen ontask_111task_116
- Source Ambiguity Collapsed Without Escalation1 task · 49 failures · 0.4% · TradeoffDoes the artifact acknowledge that the packet leaves a recommendation-changing rule unresolved, then choose one interpretation without preserving alternatives or escalating it?Seen ontask_111
Trials tagged, per model33 pts spread
0%33%67%100%
Claude Opus 5max2 / 12 trialsClaude Sonnet 5max4 / 12 trialsGemini 3.1 Prohigh0 / 12 trialsGemini 3.6 Flashhigh4 / 12 trialsGPT-5.6 Solmax2 / 12 trialsGPT-5.6 Terramax0 / 12 trialsGrader Errorthe grader3.3%
The artifact supports the challenged result, but the verifier lacks or misreads evidence, applies a contradictory rule, or penalizes required audit content.
3 modes- Grader Context or Extraction Incomplete2 tasks · 206 failures · 1.8% · Harness-fixableDoes direct inspection show the challenged source fact, companion artifact, row, field, or formula exists, while the verdict relies on a missing packet, truncated extract, unreadable cache, or refusal to inspect the required companion file?Seen ontask_111task_53
- Rubric Conflicts with Task or Benchmark169 failures · 1.5% · Harness-fixableDoes the verdict require or forbid a treatment that conflicts with an explicit task instruction, the reference artifact, a target result, or another governing rubric?
- Audit or Historical Content Misread as Final State1 task · 4 failures · <0.1% · Harness-fixableDoes the verifier treat clearly labeled proposal, history, source, policy, helper, or audit rows as the final recommendation or as prohibited side effects, despite compliant final-state fields?Seen ontask_116
Trials tagged, per model33 pts spread
0%33%67%100%
Claude Opus 5max1 / 12 trialsClaude Sonnet 5max0 / 12 trialsGemini 3.1 Prohigh3 / 12 trialsGemini 3.6 Flashhigh4 / 12 trialsGPT-5.6 Solmax1 / 12 trialsGPT-5.6 Terramax0 / 12 trials
Failure profile
How often each model broke down at each stage of the work, as a share of its trials — a different measurement from the counts below, and the one that compares models. The chart opens on the stages the field does worst at and on the models furthest apart there; pin a model in the picker and it re-picks to that model’s own worst. Every stage and every model stays one chip away.
Point at a shape — or a model above — to read its values.
Incidence — model × failure stageTrials tagged with each plotted stage, as a share of the trials that model made. The figure beside a stage is its spread — the gap between the model that hits it most and the one that hits it least.
1Presentation and Communication42 pts
2Deliverable Completion and Structure67 pts
3Reasoning and Computation33 pts
4Input and Instruction Comprehension33 pts