Mobile check deposit clearing-time auditDeposits & Funding
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
69.3%All trials
Across 10 trials on this task, reward ran from 33.3% to 93.3% — 3 of 10 cleared every check.
- 10.4 / 15avg checks passed
- 94avg turns
- 161avg tool calls
- 49mavg time
Summary · GPT-5.5
GPT-5.5 passed 3 of 10 trials. Failures cluster on trace_check_deposit_lifecycle_method (9/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 12 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 9 of 15 checks — answer_in_scope_deposit_count, answer_promised_date_accuracy and answer_stuck_exposure_metrics and 6 more.
- Trial 02Missed 2 of 15 checks — answer_stuck_reconciliation_split and trace_check_deposit_lifecycle_method.
- Trial 03Missed 4 of 15 checks — answer_in_scope_deposit_count, answer_stuck_exposure_metrics and deliverable_slack_message_content and 1 more.
- Trial 04Missed 9 of 15 checks — answer_stuck_exposure_metrics, answer_stuck_reconciliation_split and deliverable_jira_ticket_filed and 6 more.
- Trial 05Missed 10 of 15 checks — answer_cleared_timing_metrics, answer_in_scope_deposit_count and answer_promised_date_accuracy and 7 more.
- Trial 06Missed 4 of 15 checks — answer_in_scope_deposit_count, answer_stuck_exposure_metrics and deliverable_slack_message_content and 1 more.
- Trial 07Missed 1 of 15 checks — trace_check_deposit_lifecycle_method.
- Trial 08Missed 1 of 15 checks — trace_check_deposit_lifecycle_method.
- Trial 09Missed 1 of 15 checks — trace_check_deposit_lifecycle_method.
- Trial 10Missed 5 of 15 checks — answer_in_scope_deposit_count, answer_stuck_exposure_metrics and answer_stuck_reconciliation_split and 2 more.
Claude Opus 4.8
Mean reward
80%All trials
Across 10 trials on this task, reward ran from 66.7% to 100% — 2 of 10 cleared every check.
- 12.0 / 15avg checks passed
- 23avg turns
- 28avg tool calls
- 4mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 2 of 10 trials. Failures cluster on deliverable_slack_message_content (7/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 7 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Missed 1 of 15 checks — trace_check_deposit_lifecycle_method.
- Trial 03Missed 4 of 15 checks — answer_stuck_reconciliation_split, deliverable_jira_unfunded_item and deliverable_slack_message_content and 1 more.
- Trial 04Missed 4 of 15 checks — answer_stuck_reconciliation_split, deliverable_jira_unfunded_item and deliverable_slack_message_content and 1 more.
- Trial 05Missed 4 of 15 checks — answer_cleared_timing_metrics, answer_in_scope_deposit_count and answer_stuck_exposure_metrics and 1 more.
- Trial 06Missed 4 of 15 checks — answer_in_scope_deposit_count, answer_stuck_exposure_metrics and deliverable_slack_message_content and 1 more.
- Trial 07Missed 2 of 15 checks — answer_in_scope_deposit_count and trace_check_deposit_lifecycle_method.
- Trial 08Missed 3 of 15 checks — answer_in_scope_deposit_count, deliverable_slack_message_content and trace_check_deposit_lifecycle_method.
- Trial 09Missed 5 of 15 checks — answer_cleared_timing_metrics, answer_in_scope_deposit_count and answer_stuck_exposure_metrics and 2 more.
- Trial 10Missed 3 of 15 checks — answer_stuck_reconciliation_split, deliverable_jira_unfunded_item and deliverable_slack_message_content.
Gemini 3.1 Pro
Mean reward
60.7%All trials
Across 10 trials on this task, reward ran from 40% to 80% — 0 of 10 cleared every check.
- 9.1 / 15avg checks passed
- 27avg turns
- 25avg tool calls
- 3mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on trace_check_deposit_lifecycle_method (10/10 trials).
Common pattern
trace_check_deposit_lifecycle_method failed in every unsuccessful trial; what else broke varies across the other 9 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 3 of 15 checks — deliverable_jira_ticket_filed, deliverable_jira_unfunded_item and trace_check_deposit_lifecycle_method.
- Trial 02Missed 8 of 15 checks — answer_in_scope_deposit_count, answer_promised_date_accuracy and answer_stuck_exposure_metrics and 5 more.
- Trial 03Missed 3 of 15 checks — deliverable_jira_ticket_filed, deliverable_jira_unfunded_item and trace_check_deposit_lifecycle_method.
- Trial 04Missed 4 of 15 checks — answer_cleared_timing_metrics, deliverable_jira_ticket_filed and deliverable_jira_unfunded_item and 1 more.
- Trial 05Missed 7 of 15 checks — answer_promised_date_accuracy, answer_stuck_exposure_metrics and answer_stuck_reconciliation_split and 4 more.
- Trial 06Missed 6 of 15 checks — answer_cleared_timing_metrics, answer_in_scope_deposit_count and answer_promised_date_accuracy and 3 more.
- Trial 07Missed 7 of 15 checks — answer_cleared_timing_metrics, answer_in_scope_deposit_count and answer_stuck_exposure_metrics and 4 more.
- Trial 08Missed 9 of 15 checks — answer_cleared_timing_metrics, answer_in_scope_deposit_count and answer_promised_date_accuracy and 6 more.
- Trial 09Missed 8 of 15 checks — answer_promised_date_accuracy, answer_stuck_exposure_metrics and answer_stuck_reconciliation_split and 5 more.
- Trial 10Missed 4 of 15 checks — answer_cleared_timing_metrics, answer_in_scope_deposit_count and answer_promised_date_accuracy and 1 more.
Gemini 3.5 Flash
Mean reward
32%All trials
Across 10 trials on this task, reward ran from 13.3% to 86.7% — 0 of 10 cleared every check.
- 4.8 / 15avg checks passed
- 38avg turns
- 35avg tool calls
- 4mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on answer_in_scope_deposit_count (10/10 trials).
Common pattern
answer_in_scope_deposit_count and trace_check_deposit_lifecycle_method failed in every unsuccessful trial; what else broke varies across the other 11 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 11 of 15 checks — answer_in_scope_deposit_count, answer_promised_date_accuracy and answer_stuck_reconciliation_split and 8 more.
- Trial 02Missed 13 of 15 checks — answer_cleared_timing_metrics, answer_in_scope_deposit_count and answer_promised_date_accuracy and 10 more.
- Trial 03Missed 13 of 15 checks — answer_cleared_timing_metrics, answer_in_scope_deposit_count and answer_promised_date_accuracy and 10 more.
- Trial 04Missed 13 of 15 checks — answer_cleared_timing_metrics, answer_in_scope_deposit_count and answer_promised_date_accuracy and 10 more.
- Trial 05Missed 10 of 15 checks — answer_in_scope_deposit_count, answer_promised_date_accuracy and answer_stuck_reconciliation_split and 7 more.
- Trial 06Missed 2 of 15 checks — answer_in_scope_deposit_count and trace_check_deposit_lifecycle_method.
- Trial 07Missed 13 of 15 checks — answer_cleared_timing_metrics, answer_in_scope_deposit_count and answer_promised_date_accuracy and 10 more.
- Trial 08Missed 12 of 15 checks — answer_cleared_timing_metrics, answer_in_scope_deposit_count and answer_promised_date_accuracy and 9 more.
- Trial 09Missed 13 of 15 checks — answer_cleared_timing_metrics, answer_in_scope_deposit_count and answer_promised_date_accuracy and 10 more.
- Trial 10Missed 2 of 15 checks — answer_in_scope_deposit_count and trace_check_deposit_lifecycle_method.