Duplicate card auth holds never releasedCards & Interchange
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
79.3%All trials
Across 10 trials on this task, reward ran from 57.1% to 92.9% — 3 of 10 cleared every check.
- 11.1 / 14avg checks passed
- 23avg turns
- 31avg tool calls
- 5mavg time
Summary · GPT-5.5
GPT-5.5 passed 3 of 10 trials. Failures cluster on trace_duplicate_hold_reconstruct (10/10 trials).
Common pattern
answer_p2_dollar_impact and trace_duplicate_hold_reconstruct failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 6 of 14 checks — answer_p1_scope, answer_p2_dollar_impact and answer_p3_cohort and 3 more.
- Trial 02Missed 2 of 14 checks — answer_p2_dollar_impact and trace_duplicate_hold_reconstruct.
- Trial 03Missed 1 of 14 checks — trace_duplicate_hold_reconstruct.
- Trial 04Missed 6 of 14 checks — answer_p1_scope, answer_p2_dollar_impact and answer_p3_cohort and 3 more.
- Trial 05Missed 2 of 14 checks — answer_p2_dollar_impact and trace_duplicate_hold_reconstruct.
- Trial 06Missed 6 of 14 checks — answer_p1_scope, answer_p2_dollar_impact and answer_p3_cohort and 3 more.
- Trial 07Missed 1 of 14 checks — trace_duplicate_hold_reconstruct.
- Trial 08Missed 1 of 14 checks — trace_duplicate_hold_reconstruct.
- Trial 09Missed 2 of 14 checks — answer_p2_dollar_impact and trace_duplicate_hold_reconstruct.
- Trial 10Missed 2 of 14 checks — answer_p2_dollar_impact and trace_duplicate_hold_reconstruct.
Claude Opus 4.8
Mean reward
86.4%All trials
Across 10 trials on this task, reward ran from 57.1% to 100% — 3 of 10 cleared every check.
- 12.1 / 14avg checks passed
- 22avg turns
- 33avg tool calls
- 5mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 3 of 10 trials. Failures cluster on answer_p1_scope (5/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 7 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 3 of 14 checks — answer_p1_scope, answer_p2_dollar_impact and trace_duplicate_hold_reconstruct.
- Trial 02Missed 1 of 14 checks — answer_p1_scope.
- Trial 03Missed 2 of 14 checks — answer_p2_dollar_impact and answer_p5_recommendation.
- Trial 04Passed every verifier it was graded on.
- Trial 05Missed 6 of 14 checks — answer_p1_scope, answer_p2_dollar_impact and answer_p3_cohort and 3 more.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Missed 1 of 14 checks — answer_p1_scope.
- Trial 09Missed 1 of 14 checks — answer_p2_dollar_impact.
- Trial 10Missed 5 of 14 checks — answer_p1_scope, answer_p2_dollar_impact and answer_p3_cohort and 2 more.
Gemini 3.1 Pro
Mean reward
53.6%All trials
Across 10 trials on this task, reward ran from 7.1% to 100% — 2 of 10 cleared every check.
- 7.5 / 14avg checks passed
- 23avg turns
- 21avg tool calls
- 7mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 2 of 10 trials. Failures cluster on deliverable_jira_ticket_filed (7/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 13 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Missed 3 of 14 checks — deliverable_jira_ticket_filed, floor_write_back and trace_all_connectors_and_trap.
- Trial 02Passed every verifier it was graded on.
- Trial 03Missed 11 of 14 checks — answer_p1_scope, answer_p2_dollar_impact and answer_p3_cohort and 8 more.
- Trial 04Missed 11 of 14 checks — answer_p1_scope, answer_p2_dollar_impact and answer_p3_cohort and 8 more.
- Trial 05Missed 9 of 14 checks — answer_p1_scope, answer_p2_dollar_impact and answer_p3_cohort and 6 more.
- Trial 06Passed every verifier it was graded on.
- Trial 07Missed 13 of 14 checks — answer_p1_scope, answer_p2_dollar_impact and answer_p3_cohort and 10 more.
- Trial 08Missed 3 of 14 checks — deliverable_jira_ticket_filed, floor_write_back and trace_all_connectors_and_trap.
- Trial 09Missed 6 of 14 checks — answer_p1_scope, answer_p2_dollar_impact and answer_p3_cohort and 3 more.
- Trial 10Missed 9 of 14 checks — answer_p1_scope, answer_p2_dollar_impact and answer_p3_cohort and 6 more.
Gemini 3.5 Flash
Mean reward
43.6%All trials
Across 10 trials on this task, reward ran from 14.3% to 92.9% — 0 of 10 cleared every check.
- 6.1 / 14avg checks passed
- 39avg turns
- 37avg tool calls
- 5mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on deliverable_jira_ticket_filed (9/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 12 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 12 of 14 checks — answer_p1_scope, answer_p2_dollar_impact and answer_p3_cohort and 9 more.
- Trial 02Missed 6 of 14 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 3 more.
- Trial 03Missed 6 of 14 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 3 more.
- Trial 04Missed 11 of 14 checks — answer_p1_scope, answer_p2_dollar_impact and answer_p3_cohort and 8 more.
- Trial 05Missed 1 of 14 checks — answer_p3_cohort.
- Trial 06Missed 7 of 14 checks — answer_p3_cohort, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 4 more.
- Trial 07Missed 11 of 14 checks — answer_p1_scope, answer_p2_dollar_impact and answer_p3_cohort and 8 more.
- Trial 08Missed 6 of 14 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 3 more.
- Trial 09Missed 12 of 14 checks — answer_p1_scope, answer_p2_dollar_impact and answer_p3_cohort and 9 more.
- Trial 10Missed 7 of 14 checks — answer_p2_dollar_impact, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 4 more.