ACH debit hold-schedule conformance auditPayments & Transfers
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
83.3%All trials
Across 10 trials on this task, reward ran from 50% to 100% — 7 of 10 cleared every check.
- 10.0 / 12avg checks passed
- 32avg turns
- 45avg tool calls
- 4mavg time
Summary · GPT-5.5
GPT-5.5 passed 7 of 10 trials. Failures cluster on trace_hold_released_not_appeared_at (9/10 trials).
Common pattern
answer_early_hold_release_usd, trace_all_connectors_and_trap and trace_hold_released_not_appeared_at failed in every unsuccessful trial; what else broke varies across the other 3 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 3 verifiers across 3 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 12 checks — trace_hold_released_not_appeared_at.
- Trial 02Missed 1 of 12 checks — trace_hold_released_not_appeared_at.
- Trial 03Missed 3 of 12 checks — answer_early_hold_release_usd, trace_all_connectors_and_trap and trace_hold_released_not_appeared_at.
- Trial 04Missed 6 of 12 checks — answer_ach_pull_sent_90d_count, answer_early_hold_release_count and answer_early_hold_release_usd and 3 more.
- Trial 05Passed every verifier it was graded on.
- Trial 06Missed 1 of 12 checks — trace_hold_released_not_appeared_at.
- Trial 07Missed 1 of 12 checks — trace_hold_released_not_appeared_at.
- Trial 08Missed 1 of 12 checks — trace_hold_released_not_appeared_at.
- Trial 09Missed 1 of 12 checks — trace_hold_released_not_appeared_at.
- Trial 10Missed 5 of 12 checks — answer_early_hold_release_count, answer_early_hold_release_usd and deliverable_slack_message_content and 2 more.
Claude Opus 4.8
Mean reward
62.5%All trials
Across 10 trials on this task, reward ran from 50% to 91.7% — 1 of 10 cleared every check.
- 7.5 / 12avg checks passed
- 18avg turns
- 21avg tool calls
- 3mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 1 of 10 trials. Failures cluster on trace_hold_released_not_appeared_at (10/10 trials).
Common pattern
deliverable_slack_message_content and trace_hold_released_not_appeared_at failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 6 of 12 checks — answer_ach_pull_sent_90d_count, answer_early_hold_release_count and answer_early_hold_release_usd and 3 more.
- Trial 02Missed 6 of 12 checks — answer_ach_pull_sent_90d_count, answer_early_hold_release_count and answer_early_hold_release_usd and 3 more.
- Trial 03Missed 5 of 12 checks — answer_early_hold_release_count, answer_early_hold_release_usd and deliverable_slack_message_content and 2 more.
- Trial 04Missed 5 of 12 checks — answer_early_hold_release_count, answer_early_hold_release_usd and deliverable_slack_message_content and 2 more.
- Trial 05Missed 5 of 12 checks — answer_early_hold_release_count, answer_early_hold_release_usd and deliverable_slack_message_content and 2 more.
- Trial 06Missed 3 of 12 checks — answer_ach_pull_sent_90d_count, deliverable_slack_message_content and trace_hold_released_not_appeared_at.
- Trial 07Missed 3 of 12 checks — answer_ach_pull_sent_90d_count, deliverable_slack_message_content and trace_hold_released_not_appeared_at.
- Trial 08Missed 5 of 12 checks — answer_early_hold_release_count, answer_early_hold_release_usd and deliverable_slack_message_content and 2 more.
- Trial 09Missed 6 of 12 checks — answer_ach_pull_sent_90d_count, answer_early_hold_release_count and answer_early_hold_release_usd and 3 more.
- Trial 10Missed 1 of 12 checks — trace_hold_released_not_appeared_at.
Gemini 3.1 Pro
Mean reward
66.7%All trials
Across 10 trials on this task, reward ran from 25% to 91.7% — 3 of 10 cleared every check.
- 8.0 / 12avg checks passed
- 49avg turns
- 47avg tool calls
- 27mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 3 of 10 trials. Failures cluster on trace_hold_released_not_appeared_at (10/10 trials).
Common pattern
trace_hold_released_not_appeared_at failed in every unsuccessful trial; what else broke varies across the other 9 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 12 checks — trace_hold_released_not_appeared_at.
- Trial 02Missed 7 of 12 checks — answer_ach_pull_sent_90d_count, answer_early_hold_release_count and answer_early_hold_release_usd and 4 more.
- Trial 03Missed 3 of 12 checks — answer_early_hold_release_usd, trace_all_connectors_and_trap and trace_hold_released_not_appeared_at.
- Trial 04Missed 5 of 12 checks — answer_early_hold_release_count, answer_early_hold_release_usd and deliverable_slack_message_content and 2 more.
- Trial 05Missed 3 of 12 checks — deliverable_jira_ticket_filed, trace_all_connectors_and_trap and trace_hold_released_not_appeared_at.
- Trial 06Missed 1 of 12 checks — trace_hold_released_not_appeared_at.
- Trial 07Missed 1 of 12 checks — trace_hold_released_not_appeared_at.
- Trial 08Missed 9 of 12 checks — answer_early_hold_release_count, answer_early_hold_release_usd and deliverable_jira_ticket_filed and 6 more.
- Trial 09Missed 4 of 12 checks — answer_ach_pull_sent_90d_count, deliverable_jira_ticket_filed and deliverable_slack_message_content and 1 more.
- Trial 10Missed 6 of 12 checks — answer_ach_pull_sent_90d_count, answer_early_hold_release_count and answer_early_hold_release_usd and 3 more.
Gemini 3.5 Flash
Mean reward
20%All trials
Across 10 trials on this task, reward ran from 16.7% to 25% — 0 of 10 cleared every check.
- 2.4 / 12avg checks passed
- 79avg turns
- 77avg tool calls
- 45mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on answer_early_hold_release_count (10/10 trials).
Common pattern
answer_early_hold_release_count, answer_early_hold_release_usd and deliverable_jira_ticket_filed and 6 more failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 9 of 12 checks — answer_early_hold_release_count, answer_early_hold_release_usd and deliverable_jira_ticket_filed and 6 more.
- Trial 02Missed 10 of 12 checks — answer_ach_pull_sent_90d_count, answer_early_hold_release_count and answer_early_hold_release_usd and 7 more.
- Trial 03Missed 10 of 12 checks — answer_ach_pull_sent_90d_count, answer_early_hold_release_count and answer_early_hold_release_usd and 7 more.
- Trial 04Missed 10 of 12 checks — answer_ach_pull_sent_90d_count, answer_early_hold_release_count and answer_early_hold_release_usd and 7 more.
- Trial 05Missed 9 of 12 checks — answer_early_hold_release_count, answer_early_hold_release_usd and deliverable_jira_ticket_filed and 6 more.
- Trial 06Missed 10 of 12 checks — answer_ach_pull_sent_90d_count, answer_early_hold_release_count and answer_early_hold_release_usd and 7 more.
- Trial 07Missed 9 of 12 checks — answer_early_hold_release_count, answer_early_hold_release_usd and deliverable_jira_ticket_filed and 6 more.
- Trial 08Missed 10 of 12 checks — answer_ach_pull_sent_90d_count, answer_early_hold_release_count and answer_early_hold_release_usd and 7 more.
- Trial 09Missed 9 of 12 checks — answer_early_hold_release_count, answer_early_hold_release_usd and deliverable_jira_ticket_filed and 6 more.
- Trial 10Missed 10 of 12 checks — answer_ach_pull_sent_90d_count, answer_early_hold_release_count and answer_early_hold_release_usd and 7 more.