Goal Automation Throughput AnalysisPayments & Transfers
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
92.6%All trials
Across 10 trials on this task, reward ran from 78.9% to 100% — 3 of 10 cleared every check.
- 17.6 / 19avg checks passed
- 18avg turns
- 21avg tool calls
- 3mavg time
Summary · GPT-5.5
GPT-5.5 passed 3 of 10 trials. Failures cluster on cp16_accounts_with_error_and_landed (5/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 4 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 19 checks — cp16_accounts_with_error_and_landed.
- Trial 02Passed every verifier it was graded on.
- Trial 03Missed 4 of 19 checks — cp13_peak_month_landed, cp15_peak_month_accounts and cp16_accounts_with_error_and_landed and 1 more.
- Trial 04Missed 3 of 19 checks — cp13_peak_month_landed, cp15_peak_month_accounts and cp18_peak_month_landed_deleted.
- Trial 05Missed 1 of 19 checks — cp16_accounts_with_error_and_landed.
- Trial 06Missed 3 of 19 checks — cp13_peak_month_landed, cp15_peak_month_accounts and cp18_peak_month_landed_deleted.
- Trial 07Missed 1 of 19 checks — cp16_accounts_with_error_and_landed.
- Trial 08Missed 1 of 19 checks — cp16_accounts_with_error_and_landed.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Claude Opus 4.8
Mean reward
95.8%All trials
Across 10 trials on this task, reward ran from 89.5% to 100% — 3 of 10 cleared every check.
- 18.2 / 19avg checks passed
- 11avg turns
- 15avg tool calls
- 2mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 3 of 10 trials. Failures cluster on cp16_accounts_with_error_and_landed (7/10 trials).
Common pattern
cp16_accounts_with_error_and_landed failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Missed 1 of 19 checks — cp16_accounts_with_error_and_landed.
- Trial 03Missed 1 of 19 checks — cp16_accounts_with_error_and_landed.
- Trial 04Missed 1 of 19 checks — cp16_accounts_with_error_and_landed.
- Trial 05Missed 1 of 19 checks — cp16_accounts_with_error_and_landed.
- Trial 06Missed 1 of 19 checks — cp16_accounts_with_error_and_landed.
- Trial 07Passed every verifier it was graded on.
- Trial 08Missed 1 of 19 checks — cp16_accounts_with_error_and_landed.
- Trial 09Missed 2 of 19 checks — cp16_accounts_with_error_and_landed and cp2_success_rate.
- Trial 10Passed every verifier it was graded on.
Gemini 3.1 Pro
Mean reward
83.7%All trials
Across 10 trials on this task, reward ran from 0% to 94.7% — 0 of 10 cleared every check.
- 15.9 / 19avg checks passed
- 21avg turns
- 19avg tool calls
- 3mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on cp16_accounts_with_error_and_landed (10/10 trials).
Common pattern
cp16_accounts_with_error_and_landed failed in every unsuccessful trial; what else broke varies across the other 18 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 19 checks — cp16_accounts_with_error_and_landed.
- Trial 02Missed 1 of 19 checks — cp16_accounts_with_error_and_landed.
- Trial 03Missed 19 of 19 checks — cp10_pending_on_deleted_automations, cp11_account_concentration and cp12_median_landed and 16 more.
- Trial 04Missed 1 of 19 checks — cp16_accounts_with_error_and_landed.
- Trial 05Missed 1 of 19 checks — cp16_accounts_with_error_and_landed.
- Trial 06Missed 4 of 19 checks — cp13_peak_month_landed, cp15_peak_month_accounts and cp16_accounts_with_error_and_landed and 1 more.
- Trial 07Missed 1 of 19 checks — cp16_accounts_with_error_and_landed.
- Trial 08Missed 1 of 19 checks — cp16_accounts_with_error_and_landed.
- Trial 09Missed 1 of 19 checks — cp16_accounts_with_error_and_landed.
- Trial 10Missed 1 of 19 checks — cp16_accounts_with_error_and_landed.
Gemini 3.5 Flash
Mean reward
30.5%All trials
Across 10 trials on this task, reward ran from 0% to 100% — 1 of 10 cleared every check.
- 5.8 / 19avg checks passed
- 74avg turns
- 81avg tool calls
- 40mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed 1 of 10 trials. Failures cluster on cp10_pending_on_deleted_automations (8/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 19 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Missing field — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Missed 17 of 19 checks — cp10_pending_on_deleted_automations, cp11_account_concentration and cp12_median_landed and 14 more.
- Trial 02Missed 18 of 19 checks — cp10_pending_on_deleted_automations, cp11_account_concentration and cp12_median_landed and 15 more.
- Trial 03Missed 17 of 19 checks — cp10_pending_on_deleted_automations, cp11_account_concentration and cp12_median_landed and 14 more.
- Trial 04Missed 18 of 19 checks — cp10_pending_on_deleted_automations, cp11_account_concentration and cp12_median_landed and 15 more.
- Trial 05Missed 19 of 19 checks — cp10_pending_on_deleted_automations, cp11_account_concentration and cp12_median_landed and 16 more.
- Trial 06Missed 14 of 19 checks — cp10_pending_on_deleted_automations, cp11_account_concentration and cp12_median_landed and 11 more.
- Trial 07Missed 10 of 19 checks — cp10_pending_on_deleted_automations, cp11_account_concentration and cp14_accounts_for_three_quarters and 7 more.
- Trial 08Passed every verifier it was graded on.
- Trial 09Missed 18 of 19 checks — cp10_pending_on_deleted_automations, cp11_account_concentration and cp12_median_landed and 15 more.
- Trial 10Missed 1 of 19 checks — cp2_success_rate.