Returned-transfer account lock reconciliationPayments & Transfers
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
82.9%All trials
Across 10 trials on this task, reward ran from 64.3% to 100% — 6 of 10 cleared every check.
- 11.6 / 14avg checks passed
- 23avg turns
- 27avg tool calls
- 3mavg time
Summary · GPT-5.5
GPT-5.5 passed 6 of 10 trials. Failures cluster on floor_all_connectors (7/10 trials).
Common pattern
answer_returned_transfer_lock_cohort failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 4 verifiers across 4 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Missed 3 of 14 checks — answer_lock_exceptions, floor_all_connectors and floor_read_source_slack.
- Trial 03Missed 2 of 14 checks — floor_all_connectors and floor_read_source_slack.
- Trial 04Missed 5 of 14 checks — answer_returned_transfer_lock_cohort, deliverable_slack_message_content and answer_lock_exceptions and 2 more.
- Trial 05Missed 2 of 14 checks — answer_returned_transfer_lock_cohort and deliverable_slack_message_content.
- Trial 06Missed 2 of 14 checks — floor_all_connectors and floor_read_source_slack.
- Trial 07Missed 4 of 14 checks — answer_returned_transfer_lock_cohort, deliverable_slack_message_content and floor_all_connectors and 1 more.
- Trial 08Missed 2 of 14 checks — floor_all_connectors and floor_read_source_slack.
- Trial 09Passed every verifier it was graded on.
- Trial 10Missed 4 of 14 checks — answer_returned_transfer_lock_cohort, answer_lock_exceptions and floor_all_connectors and 1 more.
Claude Opus 4.8
Mean reward
85%All trials
Across 10 trials on this task, reward ran from 78.6% to 100% — 3 of 10 cleared every check.
- 11.9 / 14avg checks passed
- 22avg turns
- 30avg tool calls
- 4mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 3 of 10 trials. Failures cluster on answer_returned_transfer_lock_cohort (7/10 trials).
Common pattern
answer_returned_transfer_lock_cohort and deliverable_slack_message_content failed in every unsuccessful trial; what else broke varies across the other 2 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Missed 3 of 14 checks — answer_returned_transfer_lock_cohort, deliverable_slack_message_content and trace_all_connectors_and_trap.
- Trial 03Missed 2 of 14 checks — answer_returned_transfer_lock_cohort and deliverable_slack_message_content.
- Trial 04Missed 2 of 14 checks — answer_returned_transfer_lock_cohort and deliverable_slack_message_content.
- Trial 05Missed 1 of 14 checks — trace_all_connectors_and_trap.
- Trial 06Missed 1 of 14 checks — answer_lock_exceptions.
- Trial 07Missed 3 of 14 checks — answer_returned_transfer_lock_cohort, deliverable_slack_message_content and answer_lock_exceptions.
- Trial 08Missed 3 of 14 checks — answer_returned_transfer_lock_cohort, deliverable_slack_message_content and trace_all_connectors_and_trap.
- Trial 09Missed 3 of 14 checks — answer_returned_transfer_lock_cohort, deliverable_slack_message_content and answer_lock_exceptions.
- Trial 10Missed 3 of 14 checks — answer_returned_transfer_lock_cohort, deliverable_slack_message_content and trace_all_connectors_and_trap.
Gemini 3.1 Pro
Mean reward
72.9%All trials
Across 10 trials on this task, reward ran from 42.9% to 92.9% — 1 of 10 cleared every check.
- 10.2 / 14avg checks passed
- 52avg turns
- 51avg tool calls
- 6mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 1 of 10 trials. Failures cluster on answer_returned_transfer_lock_cohort (8/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 8 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 6 of 14 checks — answer_returned_transfer_lock_cohort, deliverable_slack_message_content and answer_lock_exceptions and 3 more.
- Trial 02Missed 2 of 14 checks — answer_returned_transfer_lock_cohort and deliverable_slack_message_content.
- Trial 03Missed 1 of 14 checks — answer_lock_exceptions.
- Trial 04Missed 3 of 14 checks — answer_returned_transfer_lock_cohort, deliverable_slack_message_content and trace_all_connectors_and_trap.
- Trial 05Missed 4 of 14 checks — answer_returned_transfer_lock_cohort, deliverable_slack_message_content and answer_lock_exceptions and 1 more.
- Trial 06Missed 3 of 14 checks — deliverable_jira_comment_next_steps, deliverable_jira_ticket_filed and answer_lock_exceptions.
- Trial 07Missed 8 of 14 checks — answer_returned_transfer_lock_cohort, deliverable_jira_comment_next_steps and deliverable_jira_ticket_filed and 5 more.
- Trial 08Missed 5 of 14 checks — answer_returned_transfer_lock_cohort, deliverable_slack_message_content and floor_all_connectors and 2 more.
- Trial 09Missed 4 of 14 checks — answer_returned_transfer_lock_cohort, deliverable_jira_comment_next_steps and deliverable_jira_ticket_filed and 1 more.
- Trial 10Missed 2 of 14 checks — answer_returned_transfer_lock_cohort and deliverable_slack_message_content.
Gemini 3.5 Flash
Mean reward
17.1%All trials
Across 10 trials on this task, reward ran from 14.3% to 28.6% — 0 of 10 cleared every check.
- 2.4 / 14avg checks passed
- 49avg turns
- 47avg tool calls
- 18mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on answer_returned_transfer_lock_cohort (10/10 trials).
Common pattern
answer_returned_transfer_lock_cohort, deliverable_jira_comment_next_steps and deliverable_jira_ticket_filed and 7 more failed in every unsuccessful trial; what else broke varies across the other 2 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 12 of 14 checks.
- Trial 02Missed 12 of 14 checks.
- Trial 03Missed 11 of 14 checks.
- Trial 04Missed 10 of 14 checks — answer_returned_transfer_lock_cohort, deliverable_jira_comment_next_steps and deliverable_jira_ticket_filed and 7 more.
- Trial 05Missed 12 of 14 checks.
- Trial 06Missed 12 of 14 checks.
- Trial 07Missed 11 of 14 checks.
- Trial 08Missed 12 of 14 checks.
- Trial 09Missed 12 of 14 checks.
- Trial 10Missed 12 of 14 checks.