ATM decline vs usage-limit reconciliationCards & Interchange
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
60%All trials
Across 10 trials on this task, reward ran from 28.6% to 71.4% — 4 of 10 cleared every check.
- 7.5 / 15avg checks passed
- 29avg turns
- 35avg tool calls
- 5mavg time
Summary · GPT-5.5
GPT-5.5 passed 4 of 10 trials. Failures cluster on cp_distinct_cardholders_affected (10/10 trials).
Common pattern
cp_distinct_cardholders_affected, diag_erroneous_declines and trace_grounded_atm_limit_config and 3 more failed in every unsuccessful trial; what else broke varies across the other 7 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 6 verifiers across 6 of 10 trials.
Per-trial notes
- Trial 01Missed 7 of 15 checks — trace_reconstructed_usage_and_erroneous_set, cp_distinct_cardholders_affected and diag_erroneous_declines and 4 more.
- Trial 02Missed 7 of 15 checks — cp_distinct_cardholders_affected, diag_erroneous_declines and trace_grounded_atm_limit_config and 4 more.
- Trial 03Missed 7 of 15 checks — cp_blocked_withdrawal_usd, cp_distinct_cardholders_affected and diag_erroneous_declines and 4 more.
- Trial 04Missed 7 of 15 checks — cp_distinct_cardholders_affected, diag_erroneous_declines and trace_grounded_atm_limit_config and 4 more.
- Trial 05Missed 7 of 15 checks — cp_distinct_cardholders_affected, diag_erroneous_declines and trace_grounded_atm_limit_config and 4 more.
- Trial 06Missed 8 of 15 checks — trace_reconstructed_usage_and_erroneous_set, cp_distinct_cardholders_affected and diag_erroneous_declines and 5 more.
- Trial 07Missed 7 of 15 checks — trace_reconstructed_usage_and_erroneous_set, cp_distinct_cardholders_affected and diag_erroneous_declines and 4 more.
- Trial 08Missed 11 of 15 checks — cp_blocked_withdrawal_usd, cp_customer_impact_days and cp_distinct_cards_affected and 8 more.
- Trial 09Missed 8 of 15 checks — trace_scope_investigation, cp_distinct_cardholders_affected and diag_erroneous_declines and 5 more.
- Trial 10Missed 6 of 15 checks — cp_distinct_cardholders_affected, diag_erroneous_declines and trace_grounded_atm_limit_config and 3 more.
Claude Opus 4.8
Mean reward
71.9%All trials
Across 10 trials on this task, reward ran from 66.7% to 76.2% — 6 of 10 cleared every check.
- 9.6 / 15avg checks passed
- 24avg turns
- 33avg tool calls
- 6mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 6 of 10 trials. Failures cluster on diag_erroneous_declines (10/10 trials).
Common pattern
trace_reconstructed_usage_and_erroneous_set, diag_erroneous_declines and trace_remediation_reasoning and 1 more failed in every unsuccessful trial; what else broke varies across the other 2 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 4 verifiers across 4 of 10 trials.
Per-trial notes
- Trial 01Missed 5 of 15 checks — trace_reconstructed_usage_and_erroneous_set, diag_erroneous_declines and trace_grounded_atm_limit_config and 2 more.
- Trial 02Missed 5 of 15 checks — trace_reconstructed_usage_and_erroneous_set, trace_scope_investigation and diag_erroneous_declines and 2 more.
- Trial 03Missed 6 of 15 checks — cp_distinct_cardholders_affected, diag_erroneous_declines and trace_grounded_atm_limit_config and 3 more.
- Trial 04Missed 6 of 15 checks — cp_distinct_cardholders_affected, diag_erroneous_declines and trace_grounded_atm_limit_config and 3 more.
- Trial 05Missed 6 of 15 checks — cp_distinct_cardholders_affected, diag_erroneous_declines and trace_grounded_atm_limit_config and 3 more.
- Trial 06Missed 5 of 15 checks — cp_distinct_cardholders_affected, diag_erroneous_declines and trace_grounded_atm_limit_config and 2 more.
- Trial 07Missed 5 of 15 checks — trace_reconstructed_usage_and_erroneous_set, diag_erroneous_declines and trace_grounded_atm_limit_config and 2 more.
- Trial 08Missed 5 of 15 checks — trace_reconstructed_usage_and_erroneous_set, diag_erroneous_declines and trace_grounded_atm_limit_config and 2 more.
- Trial 09Missed 5 of 15 checks — diag_erroneous_declines, trace_grounded_atm_limit_config and trace_remediation_reasoning and 2 more.
- Trial 10Missed 6 of 15 checks — cp_distinct_cardholders_affected, diag_erroneous_declines and trace_grounded_atm_limit_config and 3 more.
Gemini 3.1 Pro
Mean reward
43.8%All trials
Across 10 trials on this task, reward ran from 19% to 61.9% — 0 of 10 cleared every check.
- 5.8 / 15avg checks passed
- 81avg turns
- 81avg tool calls
- 22mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on trace_reconstructed_usage_and_erroneous_set (10/10 trials).
Common pattern
trace_reconstructed_usage_and_erroneous_set, cp_distinct_cardholders_affected and diag_erroneous_declines and 3 more failed in every unsuccessful trial; what else broke varies across the other 8 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 12 of 15 checks — cp_blocked_withdrawal_usd, cp_customer_impact_days and cp_distinct_cards_affected and 9 more.
- Trial 02Missed 11 of 15 checks — cp_blocked_withdrawal_usd, cp_customer_impact_days and cp_distinct_cards_affected and 8 more.
- Trial 03Missed 7 of 15 checks — trace_reconstructed_usage_and_erroneous_set, cp_distinct_cardholders_affected and diag_erroneous_declines and 4 more.
- Trial 04Missed 10 of 15 checks — trace_reconstructed_usage_and_erroneous_set, trace_scope_investigation and cp_distinct_cardholders_affected and 7 more.
- Trial 05Missed 7 of 15 checks — trace_reconstructed_usage_and_erroneous_set, trace_scope_investigation and cp_distinct_cardholders_affected and 4 more.
- Trial 06Missed 9 of 15 checks — trace_reconstructed_usage_and_erroneous_set, trace_scope_investigation and cp_distinct_cardholders_affected and 6 more.
- Trial 07Missed 11 of 15 checks — cp_blocked_withdrawal_usd, cp_customer_impact_days and cp_distinct_cards_affected and 8 more.
- Trial 08Missed 9 of 15 checks — trace_reconstructed_usage_and_erroneous_set, trace_scope_investigation and cp_distinct_cardholders_affected and 6 more.
- Trial 09Missed 9 of 15 checks — trace_reconstructed_usage_and_erroneous_set, trace_scope_investigation and cp_distinct_cardholders_affected and 6 more.
- Trial 10Missed 7 of 15 checks — trace_reconstructed_usage_and_erroneous_set, cp_distinct_cardholders_affected and diag_erroneous_declines and 4 more.
Gemini 3.5 Flash
Mean reward
18.6%All trials
Across 10 trials on this task, reward ran from 9.5% to 28.6% — 0 of 10 cleared every check.
- 2.9 / 15avg checks passed
- 41avg turns
- 40avg tool calls
- 15mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on cp_blocked_withdrawal_usd (10/10 trials).
Common pattern
cp_blocked_withdrawal_usd, cp_customer_impact_days and trace_reconstructed_usage_and_erroneous_set and 7 more failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Missing field — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Missed 12 of 15 checks — cp_blocked_withdrawal_usd, cp_customer_impact_days and trace_reconstructed_usage_and_erroneous_set and 9 more.
- Trial 02Missed 12 of 15 checks — cp_blocked_withdrawal_usd, cp_customer_impact_days and cp_distinct_cards_affected and 9 more.
- Trial 03Missed 13 of 15 checks — cp_blocked_withdrawal_usd, cp_customer_impact_days and cp_distinct_cards_affected and 10 more.
- Trial 04Missed 11 of 15 checks — cp_blocked_withdrawal_usd, cp_customer_impact_days and trace_reconstructed_usage_and_erroneous_set and 8 more.
- Trial 05Missed 13 of 15 checks — cp_blocked_withdrawal_usd, cp_customer_impact_days and cp_distinct_cards_affected and 10 more.
- Trial 06Missed 11 of 15 checks — cp_blocked_withdrawal_usd, cp_customer_impact_days and trace_reconstructed_usage_and_erroneous_set and 8 more.
- Trial 07Missed 13 of 15 checks — cp_blocked_withdrawal_usd, cp_customer_impact_days and cp_distinct_cards_affected and 10 more.
- Trial 08Missed 12 of 15 checks — cp_blocked_withdrawal_usd, cp_customer_impact_days and cp_distinct_cards_affected and 9 more.
- Trial 09Missed 13 of 15 checks — cp_blocked_withdrawal_usd, cp_customer_impact_days and cp_distinct_cards_affected and 10 more.
- Trial 10Missed 11 of 15 checks — cp_blocked_withdrawal_usd, cp_customer_impact_days and trace_reconstructed_usage_and_erroneous_set and 8 more.