Dropped incremental-auth phantom-hold reconcileCards & Interchange
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
93.3%All trials
Across 10 trials on this task, reward ran from 86.7% to 100% — 10 of 10 cleared every check.
- 16.0 / 17avg checks passed
- 10avg turns
- 12avg tool calls
- 2mavg time
Summary · GPT-5.5
GPT-5.5 cleared every check in all 10 trials.
Common pattern
Every trial cleared the core tier, so there is no failure pattern to read.
Per-trial notes
- Trial 01Missed 2 of 17 checks — floor_all_connectors and floor_read_source_ticket.
- Trial 02Passed every verifier it was graded on.
- Trial 03Missed 2 of 17 checks — floor_all_connectors and floor_read_source_ticket.
- Trial 04Passed every verifier it was graded on.
- Trial 05Passed every verifier it was graded on.
- Trial 06Missed 2 of 17 checks — floor_all_connectors and floor_read_source_ticket.
- Trial 07Passed every verifier it was graded on.
- Trial 08Missed 2 of 17 checks — floor_all_connectors and floor_read_source_ticket.
- Trial 09Passed every verifier it was graded on.
- Trial 10Missed 2 of 17 checks — floor_all_connectors and floor_read_source_ticket.
Claude Opus 4.8
Mean reward
79.3%All trials
Across 10 trials on this task, reward ran from 50% to 86.7% — 8 of 10 cleared every check.
- 13.6 / 17avg checks passed
- 11avg turns
- 17avg tool calls
- 2mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 8 of 10 trials. Failures cluster on floor_all_connectors (10/10 trials).
Common pattern
All 2 unsuccessful trials broke on exactly the same verifiers — cp_affected_trace_count, cp_repeat_offender_accounts and deliverable_slack_message_content and 6 more failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 2 verifiers across 2 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 17 checks — floor_all_connectors and floor_read_source_ticket.
- Trial 02Missed 2 of 17 checks — floor_all_connectors and floor_read_source_ticket.
- Trial 03Missed 2 of 17 checks — floor_all_connectors and floor_read_source_ticket.
- Trial 04Missed 2 of 17 checks — floor_all_connectors and floor_read_source_ticket.
- Trial 05Missed 9 of 17 checks — cp_affected_trace_count, cp_repeat_offender_accounts and deliverable_slack_message_content and 6 more.
- Trial 06Missed 2 of 17 checks — floor_all_connectors and floor_read_source_ticket.
- Trial 07Missed 2 of 17 checks — floor_all_connectors and floor_read_source_ticket.
- Trial 08Missed 2 of 17 checks — floor_all_connectors and floor_read_source_ticket.
- Trial 09Missed 9 of 17 checks — cp_affected_trace_count, cp_repeat_offender_accounts and deliverable_slack_message_content and 6 more.
- Trial 10Missed 2 of 17 checks — floor_all_connectors and floor_read_source_ticket.
Gemini 3.1 Pro
Mean reward
78%All trials
Across 10 trials on this task, reward ran from 36.7% to 100% — 5 of 10 cleared every check.
- 13.4 / 17avg checks passed
- 26avg turns
- 24avg tool calls
- 5mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 5 of 10 trials. Failures cluster on trace_all_connectors_and_trap (7/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 11 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 5 verifiers across 5 of 10 trials.
Per-trial notes
- Trial 01Missed 4 of 17 checks — deliverable_jira_ticket_filed, floor_all_connectors and floor_read_source_ticket and 1 more.
- Trial 02Passed every verifier it was graded on.
- Trial 03Passed every verifier it was graded on.
- Trial 04Missed 1 of 17 checks — deliverable_jira_ticket_filed.
- Trial 05Missed 3 of 17 checks — floor_all_connectors, floor_read_source_ticket and trace_all_connectors_and_trap.
- Trial 06Missed 3 of 17 checks — floor_all_connectors, floor_read_source_ticket and trace_all_connectors_and_trap.
- Trial 07Missed 7 of 17 checks — cp_affected_trace_count, cp_repeat_offender_accounts and deliverable_slack_message_content and 4 more.
- Trial 08Missed 3 of 17 checks — floor_all_connectors, floor_read_source_ticket and trace_all_connectors_and_trap.
- Trial 09Missed 11 of 17 checks — cp_affected_trace_count, cp_repeat_offender_accounts and deliverable_jira_ticket_filed and 8 more.
- Trial 10Missed 4 of 17 checks — deliverable_jira_ticket_filed, floor_all_connectors and floor_read_source_ticket and 1 more.
Gemini 3.5 Flash
Mean reward
33.3%All trials
Across 10 trials on this task, reward ran from 20% to 63.3% — 0 of 10 cleared every check.
- 5.2 / 17avg checks passed
- 55avg turns
- 66avg tool calls
- 19mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on cp_affected_trace_count (10/10 trials).
Common pattern
cp_affected_trace_count, cp_repeat_offender_accounts and deliverable_slack_message_content and 4 more failed in every unsuccessful trial; what else broke varies across the other 8 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 4 verifiers across 4 of 10 trials.
Per-trial notes
- Trial 01Missed 14 of 17 checks — cp_affected_trace_count, cp_repeat_offender_accounts and cp_top_affected_account_id and 11 more.
- Trial 02Missed 14 of 17 checks — cp_affected_trace_count, cp_repeat_offender_accounts and cp_top_affected_account_id and 11 more.
- Trial 03Missed 14 of 17 checks — cp_affected_trace_count, cp_repeat_offender_accounts and cp_top_affected_account_id and 11 more.
- Trial 04Missed 11 of 17 checks — cp_affected_trace_count, cp_repeat_offender_accounts and deliverable_jira_ticket_filed and 8 more.
- Trial 05Missed 13 of 17 checks — cp_affected_trace_count, cp_repeat_offender_accounts and cp_top_affected_account_id and 10 more.
- Trial 06Missed 7 of 17 checks — cp_affected_trace_count, cp_repeat_offender_accounts and deliverable_slack_message_content and 4 more.
- Trial 07Missed 11 of 17 checks — cp_affected_trace_count, cp_repeat_offender_accounts and deliverable_jira_ticket_filed and 8 more.
- Trial 08Missed 14 of 17 checks — cp_affected_trace_count, cp_repeat_offender_accounts and cp_top_affected_account_id and 11 more.
- Trial 09Missed 13 of 17 checks — cp_affected_trace_count, cp_repeat_offender_accounts and cp_top_affected_account_id and 10 more.
- Trial 10Missed 7 of 17 checks — cp_affected_trace_count, cp_repeat_offender_accounts and deliverable_slack_message_content and 4 more.