Reg CC large-deposit exception (aggregate-per-banking-day) hold exposureCompliance & Regulatory
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
35.7%All trials
Across 10 trials on this task, reward ran from 28.6% to 42.9% — 0 of 10 cleared every check.
- 2.5 / 7avg checks passed
- 23avg turns
- 22avg tool calls
- 2mavg time
Summary · GPT-5.5
GPT-5.5 passed none of its 10 trials. Failures cluster on cp1_large_deposit_count (10/10 trials).
Common pattern
cp1_large_deposit_count, cp2_total_excess_held_usd and rubric_slack_message_completeness and 1 more failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 4 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and rubric_slack_message_completeness and 1 more.
- Trial 02Missed 4 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and rubric_slack_message_completeness and 1 more.
- Trial 03Missed 5 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 2 more.
- Trial 04Missed 4 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and rubric_slack_message_completeness and 1 more.
- Trial 05Missed 4 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and rubric_slack_message_completeness and 1 more.
- Trial 06Missed 5 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 2 more.
- Trial 07Missed 5 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 2 more.
- Trial 08Missed 4 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and rubric_slack_message_completeness and 1 more.
- Trial 09Missed 5 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 2 more.
- Trial 10Missed 5 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 2 more.
Claude Opus 4.8
Mean reward
44.3%All trials
Across 10 trials on this task, reward ran from 28.6% to 85.7% — 1 of 10 cleared every check.
- 3.1 / 7avg checks passed
- 13avg turns
- 13avg tool calls
- 1mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 1 of 10 trials. Failures cluster on rubric_aggregate_availability_and_datefield (10/10 trials).
Common pattern
cp1_large_deposit_count, cp2_total_excess_held_usd and rubric_slack_message_completeness and 1 more failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 4 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and rubric_slack_message_completeness and 1 more.
- Trial 02Missed 5 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 2 more.
- Trial 03Missed 4 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and rubric_slack_message_completeness and 1 more.
- Trial 04Missed 4 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and rubric_slack_message_completeness and 1 more.
- Trial 05Missed 1 of 7 checks — rubric_aggregate_availability_and_datefield.
- Trial 06Missed 4 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and rubric_slack_message_completeness and 1 more.
- Trial 07Missed 4 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and rubric_slack_message_completeness and 1 more.
- Trial 08Missed 4 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and rubric_slack_message_completeness and 1 more.
- Trial 09Missed 5 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 2 more.
- Trial 10Missed 4 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and rubric_slack_message_completeness and 1 more.
Gemini 3.1 Pro
Mean reward
45.7%All trials
Across 10 trials on this task, reward ran from 28.6% to 71.4% — 0 of 10 cleared every check.
- 3.2 / 7avg checks passed
- 25avg turns
- 23avg tool calls
- 2mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on rubric_aggregate_availability_and_datefield (10/10 trials).
Common pattern
rubric_aggregate_availability_and_datefield failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 3 of 7 checks — cp3_avg_days_to_available, rubric_slack_message_completeness and rubric_aggregate_availability_and_datefield.
- Trial 02Missed 2 of 7 checks — cp3_avg_days_to_available and rubric_aggregate_availability_and_datefield.
- Trial 03Missed 5 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 2 more.
- Trial 04Missed 5 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 2 more.
- Trial 05Missed 5 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 2 more.
- Trial 06Missed 2 of 7 checks — cp3_avg_days_to_available and rubric_aggregate_availability_and_datefield.
- Trial 07Missed 4 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and rubric_slack_message_completeness and 1 more.
- Trial 08Missed 2 of 7 checks — cp3_avg_days_to_available and rubric_aggregate_availability_and_datefield.
- Trial 09Missed 5 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 2 more.
- Trial 10Missed 5 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 2 more.
Gemini 3.5 Flash
Mean reward
0%All trials
Across 10 trials on this task, reward ran from 0% to 0% — 0 of 10 cleared every check.
- 0.0 / 7avg checks passed
- 42avg turns
- 40avg tool calls
- 15mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on cp1_large_deposit_count (10/10 trials).
Common pattern
All 10 unsuccessful trials broke on exactly the same verifiers — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 4 more failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 7 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 4 more.
- Trial 02Missed 7 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 4 more.
- Trial 03Missed 7 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 4 more.
- Trial 04Missed 7 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 4 more.
- Trial 05Missed 7 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 4 more.
- Trial 06Missed 7 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 4 more.
- Trial 07Missed 7 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 4 more.
- Trial 08Missed 7 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 4 more.
- Trial 09Missed 7 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 4 more.
- Trial 10Missed 7 of 7 checks — cp1_large_deposit_count, cp2_total_excess_held_usd and cp3_avg_days_to_available and 4 more.