Stuck card auth holds after POS declinesCards & Interchange
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
80%All trials
Across 10 trials on this task, reward ran from 50% to 100% — 5 of 10 cleared every check.
- 9.6 / 12avg checks passed
- 21avg turns
- 22avg tool calls
- 2mavg time
Summary · GPT-5.5
GPT-5.5 passed 5 of 10 trials. Failures cluster on trace_auth_decline_pairing (6/10 trials).
Common pattern
answer_frozen_and_release_needed and deliverable_slack_message_content failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 5 verifiers across 5 of 10 trials.
Per-trial notes
- Trial 01Missed 3 of 12 checks — answer_frozen_and_release_needed, deliverable_slack_message_content and trace_auth_decline_pairing.
- Trial 02Missed 1 of 12 checks — trace_auth_decline_pairing.
- Trial 03Missed 3 of 12 checks — answer_event_window, answer_frozen_and_release_needed and deliverable_slack_message_content.
- Trial 04Missed 1 of 12 checks — trace_auth_decline_pairing.
- Trial 05Missed 6 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 3 more.
- Trial 06Missed 6 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 3 more.
- Trial 07Missed 4 of 12 checks — answer_event_window, answer_frozen_and_release_needed and deliverable_slack_message_content and 1 more.
- Trial 08Passed every verifier it was graded on.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Claude Opus 4.8
Mean reward
72.5%All trials
Across 10 trials on this task, reward ran from 50% to 100% — 4 of 10 cleared every check.
- 8.7 / 12avg checks passed
- 15avg turns
- 17avg tool calls
- 2mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 4 of 10 trials. Failures cluster on answer_declined_with_hold_and_stuck_counts (5/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 9 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 6 verifiers across 6 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Missed 6 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 3 more.
- Trial 03Missed 3 of 12 checks — deliverable_jira_ticket_filed, floor_all_connectors and floor_write_back.
- Trial 04Missed 6 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 3 more.
- Trial 05Missed 6 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 3 more.
- Trial 06Missed 6 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 3 more.
- Trial 07Passed every verifier it was graded on.
- Trial 08Passed every verifier it was graded on.
- Trial 09Missed 6 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 3 more.
- Trial 10Passed every verifier it was graded on.
Gemini 3.1 Pro
Mean reward
64.2%All trials
Across 10 trials on this task, reward ran from 33.3% to 100% — 3 of 10 cleared every check.
- 7.7 / 12avg checks passed
- 38avg turns
- 36avg tool calls
- 4mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 3 of 10 trials. Failures cluster on trace_auth_decline_pairing (9/10 trials).
Common pattern
answer_frozen_and_release_needed, deliverable_slack_message_content and trace_auth_decline_pairing failed in every unsuccessful trial; what else broke varies across the other 5 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 6 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 3 more.
- Trial 02Missed 8 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 5 more.
- Trial 03Missed 3 of 12 checks — answer_frozen_and_release_needed, deliverable_slack_message_content and trace_auth_decline_pairing.
- Trial 04Missed 6 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 3 more.
- Trial 05Passed every verifier it was graded on.
- Trial 06Missed 6 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 3 more.
- Trial 07Missed 6 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 3 more.
- Trial 08Missed 1 of 12 checks — trace_auth_decline_pairing.
- Trial 09Missed 6 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 3 more.
- Trial 10Missed 1 of 12 checks — trace_auth_decline_pairing.
Gemini 3.5 Flash
Mean reward
26.7%All trials
Across 10 trials on this task, reward ran from 16.7% to 50% — 0 of 10 cleared every check.
- 3.2 / 12avg checks passed
- 66avg turns
- 64avg tool calls
- 36mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on answer_declined_with_hold_and_stuck_counts (10/10 trials).
Common pattern
answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 3 more failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 10 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 7 more.
- Trial 02Missed 6 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 3 more.
- Trial 03Missed 10 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 7 more.
- Trial 04Missed 6 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 3 more.
- Trial 05Missed 10 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 7 more.
- Trial 06Missed 10 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 7 more.
- Trial 07Missed 10 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 7 more.
- Trial 08Missed 10 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 7 more.
- Trial 09Missed 6 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 3 more.
- Trial 10Missed 10 of 12 checks — answer_declined_with_hold_and_stuck_counts, answer_event_window and answer_frozen_and_release_needed and 7 more.