Audit manual-review check deposit release latency & stranded fundsDeposits & Funding
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
98.5%All trials
Across 10 trials on this task, reward ran from 92.3% to 100% — 8 of 10 cleared every check.
- 12.8 / 13avg checks passed
- 20avg turns
- 26avg tool calls
- 3mavg time
Summary · GPT-5.5
GPT-5.5 passed 8 of 10 trials. Failures cluster on rubric_root_cause (2/10 trials).
Common pattern
All 2 unsuccessful trials broke on exactly the same verifiers — rubric_root_cause failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 2 verifiers across 2 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 13 checks — rubric_root_cause.
- Trial 02Passed every verifier it was graded on.
- Trial 03Passed every verifier it was graded on.
- Trial 04Passed every verifier it was graded on.
- Trial 05Passed every verifier it was graded on.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Missed 1 of 13 checks — rubric_root_cause.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Claude Opus 4.8
Mean reward
76.2%All trials
Across 10 trials on this task, reward ran from 23.1% to 92.3% — 0 of 10 cleared every check.
- 9.9 / 13avg checks passed
- 20avg turns
- 34avg tool calls
- 3mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed none of its 10 trials. Failures cluster on rubric_root_cause (10/10 trials).
Common pattern
rubric_root_cause failed in every unsuccessful trial; what else broke varies across the other 10 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 13 checks — rubric_root_cause.
- Trial 02Missed 10 of 13 checks — gold_accounts_multi_stranded, gold_affected_accounts and gold_max_account_stranded_usd and 7 more.
- Trial 03Missed 2 of 13 checks — rubric_root_cause and scopes_stranded_set_and_cohorts.
- Trial 04Missed 1 of 13 checks — rubric_root_cause.
- Trial 05Missed 2 of 13 checks — rubric_root_cause and scopes_stranded_set_and_cohorts.
- Trial 06Missed 1 of 13 checks — rubric_root_cause.
- Trial 07Missed 1 of 13 checks — rubric_root_cause.
- Trial 08Missed 2 of 13 checks — rubric_root_cause and scopes_stranded_set_and_cohorts.
- Trial 09Missed 1 of 13 checks — rubric_root_cause.
- Trial 10Missed 10 of 13 checks — gold_accounts_multi_stranded, gold_affected_accounts and gold_max_account_stranded_usd and 7 more.
Gemini 3.1 Pro
Mean reward
70.8%All trials
Across 10 trials on this task, reward ran from 15.4% to 92.3% — 0 of 10 cleared every check.
- 9.2 / 13avg checks passed
- 37avg turns
- 45avg tool calls
- 4mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on rubric_root_cause (10/10 trials).
Common pattern
rubric_root_cause failed in every unsuccessful trial; what else broke varies across the other 10 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 13 checks — rubric_root_cause.
- Trial 02Missed 11 of 13 checks — gold_accounts_multi_stranded, gold_affected_accounts and gold_max_account_stranded_usd and 8 more.
- Trial 03Missed 1 of 13 checks — rubric_root_cause.
- Trial 04Missed 9 of 13 checks — gold_accounts_multi_stranded, gold_affected_accounts and gold_max_account_stranded_usd and 6 more.
- Trial 05Missed 2 of 13 checks — rubric_root_cause and scopes_stranded_set_and_cohorts.
- Trial 06Missed 1 of 13 checks — rubric_root_cause.
- Trial 07Missed 2 of 13 checks — rubric_root_cause and scopes_stranded_set_and_cohorts.
- Trial 08Missed 1 of 13 checks — rubric_root_cause.
- Trial 09Missed 1 of 13 checks — rubric_root_cause.
- Trial 10Missed 9 of 13 checks — gold_accounts_multi_stranded, gold_affected_accounts and gold_max_account_stranded_usd and 6 more.
Gemini 3.5 Flash
Mean reward
8.5%All trials
Across 10 trials on this task, reward ran from 7.7% to 15.4% — 0 of 10 cleared every check.
- 1.1 / 13avg checks passed
- 38avg turns
- 51avg tool calls
- 13mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on gold_accounts_multi_stranded (10/10 trials).
Common pattern
gold_accounts_multi_stranded, gold_affected_accounts and gold_max_account_stranded_usd and 8 more failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Missed 12 of 13 checks — gold_accounts_multi_stranded, gold_affected_accounts and gold_max_account_stranded_usd and 9 more.
- Trial 02Missed 12 of 13 checks — gold_accounts_multi_stranded, gold_affected_accounts and gold_max_account_stranded_usd and 9 more.
- Trial 03Missed 12 of 13 checks — gold_accounts_multi_stranded, gold_affected_accounts and gold_max_account_stranded_usd and 9 more.
- Trial 04Missed 12 of 13 checks — gold_accounts_multi_stranded, gold_affected_accounts and gold_max_account_stranded_usd and 9 more.
- Trial 05Missed 12 of 13 checks — gold_accounts_multi_stranded, gold_affected_accounts and gold_max_account_stranded_usd and 9 more.
- Trial 06Missed 12 of 13 checks — gold_accounts_multi_stranded, gold_affected_accounts and gold_max_account_stranded_usd and 9 more.
- Trial 07Missed 12 of 13 checks — gold_accounts_multi_stranded, gold_affected_accounts and gold_max_account_stranded_usd and 9 more.
- Trial 08Missed 11 of 13 checks — gold_accounts_multi_stranded, gold_affected_accounts and gold_max_account_stranded_usd and 8 more.
- Trial 09Missed 12 of 13 checks — gold_accounts_multi_stranded, gold_affected_accounts and gold_max_account_stranded_usd and 9 more.
- Trial 10Missed 12 of 13 checks — gold_accounts_multi_stranded, gold_affected_accounts and gold_max_account_stranded_usd and 9 more.