VIP ACH Limit Compliance: non-VIP weekly PaySomeone ACH auditPayments & Transfers
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
64.7%All trials
Across 10 trials on this task, reward ran from 46.7% to 93.3% — 0 of 10 cleared every check.
- 9.7 / 15avg checks passed
- 18avg turns
- 35avg tool calls
- 3mavg time
Summary · GPT-5.5
GPT-5.5 passed none of its 10 trials. Failures cluster on cp1_vip_eligible_accounts_count (8/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 10 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 6 of 15 checks — ach_outbound_scope, cp1_vip_eligible_accounts_count and cp5_total_ach_debit_transfers and 3 more.
- Trial 02Missed 6 of 15 checks — ach_outbound_scope, cp5_total_ach_debit_transfers and cp7_non_vip_accounts_exceeding_weekly_limit and 3 more.
- Trial 03Missed 7 of 15 checks — ach_outbound_scope, cp1_vip_eligible_accounts_count and cp5_total_ach_debit_transfers and 4 more.
- Trial 04Missed 7 of 15 checks — cp1_vip_eligible_accounts_count, cp7_non_vip_accounts_exceeding_weekly_limit and jira_pd346_comment_policy and 4 more.
- Trial 05Missed 5 of 15 checks — ach_outbound_scope, cp5_total_ach_debit_transfers and cp7_non_vip_accounts_exceeding_weekly_limit and 2 more.
- Trial 06Missed 8 of 15 checks — ach_outbound_scope, cp1_vip_eligible_accounts_count and cp5_total_ach_debit_transfers and 5 more.
- Trial 07Missed 1 of 15 checks — cp1_vip_eligible_accounts_count.
- Trial 08Missed 3 of 15 checks — cp1_vip_eligible_accounts_count, jira_pd346_comment_policy and process_jira_floor.
- Trial 09Missed 5 of 15 checks.
- Trial 10Missed 5 of 15 checks — cp1_vip_eligible_accounts_count, cp7_non_vip_accounts_exceeding_weekly_limit and vip_derivation_full_tables and 2 more.
Claude Opus 4.8
Mean reward
49.3%All trials
Across 10 trials on this task, reward ran from 40% to 60% — 0 of 10 cleared every check.
- 7.4 / 15avg checks passed
- 15avg turns
- 22avg tool calls
- 2mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed none of its 10 trials. Failures cluster on cp1_vip_eligible_accounts_count (10/10 trials).
Common pattern
cp1_vip_eligible_accounts_count, jira_pd346_comment_policy and vip_derivation_full_tables and 1 more failed in every unsuccessful trial; what else broke varies across the other 7 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 6 of 15 checks — cp1_vip_eligible_accounts_count, cp7_non_vip_accounts_exceeding_weekly_limit and jira_pd346_comment_policy and 3 more.
- Trial 02Missed 6 of 15 checks — cp1_vip_eligible_accounts_count, cp7_non_vip_accounts_exceeding_weekly_limit and jira_pd177_vip_criteria and 3 more.
- Trial 03Missed 9 of 15 checks — ach_outbound_scope, cp1_vip_eligible_accounts_count and cp5_total_ach_debit_transfers and 6 more.
- Trial 04Missed 9 of 15 checks — ach_outbound_scope, cp1_vip_eligible_accounts_count and cp5_total_ach_debit_transfers and 6 more.
- Trial 05Missed 8 of 15 checks.
- Trial 06Missed 6 of 15 checks — cp1_vip_eligible_accounts_count, cp7_non_vip_accounts_exceeding_weekly_limit and jira_pd346_comment_policy and 3 more.
- Trial 07Missed 9 of 15 checks — ach_outbound_scope, cp1_vip_eligible_accounts_count and cp5_total_ach_debit_transfers and 6 more.
- Trial 08Missed 8 of 15 checks — ach_outbound_scope, cp1_vip_eligible_accounts_count and cp5_total_ach_debit_transfers and 5 more.
- Trial 09Missed 9 of 15 checks — ach_outbound_scope, cp1_vip_eligible_accounts_count and cp5_total_ach_debit_transfers and 6 more.
- Trial 10Missed 6 of 15 checks — cp1_vip_eligible_accounts_count, cp7_non_vip_accounts_exceeding_weekly_limit and jira_pd346_comment_policy and 3 more.
Gemini 3.1 Pro
Mean reward
62%All trials
Across 10 trials on this task, reward ran from 40% to 93.3% — 0 of 10 cleared every check.
- 9.3 / 15avg checks passed
- 43avg turns
- 41avg tool calls
- 4mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on jira_pd346_comment_policy (10/10 trials).
Common pattern
jira_pd346_comment_policy failed in every unsuccessful trial; what else broke varies across the other 9 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 15 checks — jira_pd346_comment_policy.
- Trial 02Missed 8 of 15 checks — ach_outbound_scope, cp1_vip_eligible_accounts_count and cp5_total_ach_debit_transfers and 5 more.
- Trial 03Missed 6 of 15 checks — cp1_vip_eligible_accounts_count, cp7_non_vip_accounts_exceeding_weekly_limit and jira_pd346_comment_policy and 3 more.
- Trial 04Missed 6 of 15 checks — cp1_vip_eligible_accounts_count, cp7_non_vip_accounts_exceeding_weekly_limit and jira_pd346_comment_policy and 3 more.
- Trial 05Missed 6 of 15 checks — ach_outbound_scope, cp5_total_ach_debit_transfers and cp7_non_vip_accounts_exceeding_weekly_limit and 3 more.
- Trial 06Missed 6 of 15 checks — cp1_vip_eligible_accounts_count, cp7_non_vip_accounts_exceeding_weekly_limit and jira_pd346_comment_policy and 3 more.
- Trial 07Missed 6 of 15 checks — ach_outbound_scope, cp5_total_ach_debit_transfers and cp7_non_vip_accounts_exceeding_weekly_limit and 3 more.
- Trial 08Missed 9 of 15 checks — ach_outbound_scope, cp1_vip_eligible_accounts_count and cp5_total_ach_debit_transfers and 6 more.
- Trial 09Missed 3 of 15 checks — cp1_vip_eligible_accounts_count, jira_pd346_comment_policy and cp2_vip_eligible_pct.
- Trial 10Missed 6 of 15 checks — cp1_vip_eligible_accounts_count, cp7_non_vip_accounts_exceeding_weekly_limit and jira_pd346_comment_policy and 3 more.
Gemini 3.5 Flash
Mean reward
33.3%All trials
Across 10 trials on this task, reward ran from 13.3% to 60% — 0 of 10 cleared every check.
- 5.0 / 15avg checks passed
- 77avg turns
- 79avg tool calls
- 46mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on cp1_vip_eligible_accounts_count (10/10 trials).
Common pattern
cp1_vip_eligible_accounts_count, cp7_non_vip_accounts_exceeding_weekly_limit and jira_pd346_comment_policy and 3 more failed in every unsuccessful trial; what else broke varies across the other 7 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Missing field — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 10 of 15 checks — ach_outbound_scope, cp1_vip_eligible_accounts_count and cp5_total_ach_debit_transfers and 7 more.
- Trial 02Missed 13 of 15 checks — ach_outbound_scope, cp1_vip_eligible_accounts_count and cp3_non_vip_weekly_limit_usd and 10 more.
- Trial 03Missed 7 of 15 checks — cp1_vip_eligible_accounts_count, cp7_non_vip_accounts_exceeding_weekly_limit and jira_pd346_comment_policy and 4 more.
- Trial 04Missed 11 of 15 checks — ach_outbound_scope, cp1_vip_eligible_accounts_count and cp5_total_ach_debit_transfers and 8 more.
- Trial 05Missed 6 of 15 checks — cp1_vip_eligible_accounts_count, cp7_non_vip_accounts_exceeding_weekly_limit and jira_pd346_comment_policy and 3 more.
- Trial 06Missed 10 of 15 checks.
- Trial 07Missed 10 of 15 checks.
- Trial 08Missed 11 of 15 checks — ach_outbound_scope, cp1_vip_eligible_accounts_count and cp5_total_ach_debit_transfers and 8 more.
- Trial 09Missed 11 of 15 checks — ach_outbound_scope, cp1_vip_eligible_accounts_count and cp5_total_ach_debit_transfers and 8 more.
- Trial 10Missed 11 of 15 checks — ach_outbound_scope, cp1_vip_eligible_accounts_count and cp5_total_ach_debit_transfers and 8 more.