Failure modes

Machine labels: the harness labels every trial at completion with one of a 9-value closed vocabulary, from the trial outcome alone. Each failed trial carries its label once, on its first failed check, so every count here is a count of trials. A trial that passed carries no mode. The bar under each stage is its share of every classified failure — how much of the drop it accounts for. The dots are how often each model’s own trials were tagged with it, on one scale across every card. Two different denominators, so a large stage is not the same claim as a model that hits one often.

• Claude Opus 4.8
• Gemini 3.1 Pro
• Gemini 3.5 Flash
• GPT-5.5

1. Answer95.7%

   1799 failures · 95.7% of all

   The trial finished and replied, and the reply did not hold up.

   4 modes
   • Wrong answer64 tasks · 1624 failures · 86.4%Seen ontask_ach_debit_hold_early_releasetask_ach_debit_hold_schedule_conformancetask_ach_return_fraud_blast_radius_conformancetask_additional_owner_apa_pipeline_conformancetask_additional_owner_card_activation_funneltask_alloy_would_approve_vs_human_reviewand 58 more
   • Missing field24 tasks · 148 failures · 7.9%Seen ontask_alloy_would_approve_vs_human_reviewtask_atm_decline_vs_usage_limit_reconciliationtask_automation_disbursement_economicstask_cancelled_check_ledger_net_balance_audittask_card_authorizations_after_account_locktask_card_decline_exposure_profileand 18 more
   • Near miss6 tasks · 22 failures · 1.2%Seen ontask_finra_reg_bi_suitability_mastercardtask_instant_deposit_fee_economics_breakeven_analysistask_mastercard_qualifying_signature_volume_2024task_subscription_dead_invoice_ratetask_transfer_balance_check_nsftask_vip_ach_limit_compliance
   • Unparseable JSON5 tasks · 5 failures · 0.3%Seen ontask_cancelled_check_ledger_net_balance_audittask_card_decline_exposure_profiletask_instant_deposit_fee_economics_breakeven_analysistask_mastercard_qualifying_signature_volume_2024task_zeta_bill_goal_feature_adoption_parity_analytics

   Trials tagged, per model30 pts spread

   0%33%67%100%

   Claude Opus 4.8410 / 640 trialsGemini 3.1 Pro503 / 640 trialsGemini 3.5 Flash540 / 640 trialsGPT-5.5346 / 640 trials

2. Execution4.3%

   81 failures · 4.3% of all

   The trial never reached a usable answer.

   2 modes
   • Context overflow17 tasks · 45 failures · 2.4%Seen ontask_ach_debit_hold_schedule_conformancetask_additional_owner_card_activation_funneltask_alloy_would_approve_vs_human_reviewtask_atm_decline_vs_usage_limit_reconciliationtask_automation_disbursement_economicstask_bill_pay_paid_transaction_reconcileand 11 more
   • Exhausted iterations19 tasks · 36 failures · 1.9%Seen ontask_ach_debit_hold_early_releasetask_ach_return_fraud_blast_radius_conformancetask_additional_owner_apa_pipeline_conformancetask_additional_owner_card_activation_funneltask_bill_underfunded_alert_conformancetask_cancelled_check_ledger_net_balance_auditand 13 more

   Trials tagged, per model8 pts spread

   0%33%67%100%

   Claude Opus 4.80 / 640 trialsGemini 3.1 Pro20 / 640 trialsGemini 3.5 Flash51 / 640 trialsGPT-5.510 / 640 trials