Confluence Overdraft & NSF PolicyRisk & Fraud
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
77.5%All trials
Across 10 trials on this task, reward ran from 50% to 100% — 5 of 10 cleared every check.
- 3.1 / 4avg checks passed
- 32avg turns
- 445avg tool calls
- 19mavg time
Summary · GPT-5.5
GPT-5.5 passed 5 of 10 trials. Failures cluster on cp4_payday_nsf_transfers (5/10 trials).
Common pattern
cp4_payday_nsf_transfers failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 5 verifiers across 5 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Missed 1 of 4 checks — cp4_payday_nsf_transfers.
- Trial 03Missed 2 of 4 checks — cp3_healing_attempts and cp4_payday_nsf_transfers.
- Trial 04Missed 2 of 4 checks — cp3_healing_attempts and cp4_payday_nsf_transfers.
- Trial 05Passed every verifier it was graded on.
- Trial 06Passed every verifier it was graded on.
- Trial 07Missed 2 of 4 checks — cp3_healing_attempts and cp4_payday_nsf_transfers.
- Trial 08Passed every verifier it was graded on.
- Trial 09Passed every verifier it was graded on.
- Trial 10Missed 2 of 4 checks — cp3_healing_attempts and cp4_payday_nsf_transfers.
Claude Opus 4.8
Mean reward
70%All trials
Across 10 trials on this task, reward ran from 50% to 100% — 1 of 10 cleared every check.
- 2.8 / 4avg checks passed
- 29avg turns
- 56avg tool calls
- 4mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 1 of 10 trials. Failures cluster on cp3_healing_attempts (7/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 2 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 4 checks — cp4_payday_nsf_transfers.
- Trial 02Missed 2 of 4 checks — cp3_healing_attempts and cp4_payday_nsf_transfers.
- Trial 03Missed 1 of 4 checks — cp3_healing_attempts.
- Trial 04Missed 1 of 4 checks — cp3_healing_attempts.
- Trial 05Missed 1 of 4 checks — cp4_payday_nsf_transfers.
- Trial 06Passed every verifier it was graded on.
- Trial 07Missed 1 of 4 checks — cp3_healing_attempts.
- Trial 08Missed 2 of 4 checks — cp3_healing_attempts and cp4_payday_nsf_transfers.
- Trial 09Missed 1 of 4 checks — cp3_healing_attempts.
- Trial 10Missed 2 of 4 checks — cp3_healing_attempts and cp4_payday_nsf_transfers.
Gemini 3.1 Pro
Mean reward
92.5%All trials
Across 10 trials on this task, reward ran from 75% to 100% — 7 of 10 cleared every check.
- 3.7 / 4avg checks passed
- 71avg turns
- 102avg tool calls
- 32mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 7 of 10 trials. Failures cluster on cp3_healing_attempts (2/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 2 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 3 verifiers across 3 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Passed every verifier it was graded on.
- Trial 03Missed 1 of 4 checks — cp3_healing_attempts.
- Trial 04Passed every verifier it was graded on.
- Trial 05Missed 1 of 4 checks — cp3_healing_attempts.
- Trial 06Passed every verifier it was graded on.
- Trial 07Missed 1 of 4 checks — cp4_payday_nsf_transfers.
- Trial 08Passed every verifier it was graded on.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Gemini 3.5 Flash
Mean reward
72.5%All trials
Across 10 trials on this task, reward ran from 0% to 100% — 5 of 10 cleared every check.
- 2.9 / 4avg checks passed
- 57avg turns
- 58avg tool calls
- 17mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed 5 of 10 trials. Failures cluster on cp4_payday_nsf_transfers (5/10 trials).
Common pattern
cp4_payday_nsf_transfers failed in every unsuccessful trial; what else broke varies across the other 3 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Missing field — 3 verifiers across 3 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Passed every verifier it was graded on.
- Trial 03Missed 1 of 4 checks — cp4_payday_nsf_transfers.
- Trial 04Missed 4 of 4 checks — cp1_overdraft_offered, cp2_negative_balance_count and cp3_healing_attempts and 1 more.
- Trial 05Passed every verifier it was graded on.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Missed 4 of 4 checks — cp1_overdraft_offered, cp2_negative_balance_count and cp3_healing_attempts and 1 more.
- Trial 09Missed 1 of 4 checks — cp4_payday_nsf_transfers.
- Trial 10Missed 1 of 4 checks — cp4_payday_nsf_transfers.