ACH-debit holds released before scheduled settlementPayments & Transfers
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
80%All trials
Across 10 trials on this task, reward ran from 16.7% to 100% — 7 of 10 cleared every check.
- 9.6 / 12avg checks passed
- 52avg turns
- 72avg tool calls
- 9mavg time
Summary · GPT-5.5
GPT-5.5 passed 7 of 10 trials. Failures cluster on trace_hold_release_method (5/10 trials).
Common pattern
answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 3 more failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 3 verifiers across 3 of 10 trials.
Per-trial notes
- Trial 01Missed 6 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 3 more.
- Trial 02Missed 1 of 12 checks — trace_hold_release_method.
- Trial 03Passed every verifier it was graded on.
- Trial 04Passed every verifier it was graded on.
- Trial 05Missed 10 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 7 more.
- Trial 06Missed 6 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 3 more.
- Trial 07Passed every verifier it was graded on.
- Trial 08Missed 1 of 12 checks — trace_hold_release_method.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Claude Opus 4.8
Mean reward
58.3%All trials
Across 10 trials on this task, reward ran from 50% to 91.7% — 2 of 10 cleared every check.
- 7.0 / 12avg checks passed
- 27avg turns
- 37avg tool calls
- 6mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 2 of 10 trials. Failures cluster on trace_hold_release_method (10/10 trials).
Common pattern
All 8 unsuccessful trials broke on exactly the same verifiers — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 3 more failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Missed 6 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 3 more.
- Trial 02Missed 6 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 3 more.
- Trial 03Missed 6 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 3 more.
- Trial 04Missed 6 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 3 more.
- Trial 05Missed 1 of 12 checks — trace_hold_release_method.
- Trial 06Missed 6 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 3 more.
- Trial 07Missed 6 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 3 more.
- Trial 08Missed 6 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 3 more.
- Trial 09Missed 1 of 12 checks — trace_hold_release_method.
- Trial 10Missed 6 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 3 more.
Gemini 3.1 Pro
Mean reward
66.7%All trials
Across 10 trials on this task, reward ran from 16.7% to 100% — 3 of 10 cleared every check.
- 8.0 / 12avg checks passed
- 61avg turns
- 63avg tool calls
- 19mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 3 of 10 trials. Failures cluster on trace_hold_release_method (8/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 10 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 5 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 2 more.
- Trial 02Missed 10 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 7 more.
- Trial 03Missed 1 of 12 checks — deliverable_jira_ticket_filed.
- Trial 04Missed 5 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 2 more.
- Trial 05Missed 6 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 3 more.
- Trial 06Missed 1 of 12 checks — trace_hold_release_method.
- Trial 07Missed 1 of 12 checks — trace_hold_release_method.
- Trial 08Passed every verifier it was graded on.
- Trial 09Missed 6 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 3 more.
- Trial 10Missed 5 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 2 more.
Gemini 3.5 Flash
Mean reward
20%All trials
Across 10 trials on this task, reward ran from 16.7% to 50% — 0 of 10 cleared every check.
- 2.4 / 12avg checks passed
- 53avg turns
- 64avg tool calls
- 25mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on answer_early_release_count (10/10 trials).
Common pattern
answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 3 more failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 10 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 7 more.
- Trial 02Missed 10 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 7 more.
- Trial 03Missed 10 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 7 more.
- Trial 04Missed 10 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 7 more.
- Trial 05Missed 10 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 7 more.
- Trial 06Missed 10 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 7 more.
- Trial 07Missed 10 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 7 more.
- Trial 08Missed 6 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 3 more.
- Trial 09Missed 10 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 7 more.
- Trial 10Missed 10 of 12 checks — answer_early_release_count, answer_exposure_usd and answer_timing_and_service_split and 7 more.