Card Decline Exposure ProfileCards & Interchange
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
77.3%All trials
Across 10 trials on this task, reward ran from 40% to 100% — 1 of 10 cleared every check.
- 11.6 / 15avg checks passed
- 16avg turns
- 31avg tool calls
- 3mavg time
Summary · GPT-5.5
GPT-5.5 passed 1 of 10 trials. Failures cluster on deliverable_slack_summary_content (9/10 trials).
Common pattern
deliverable_slack_summary_content failed in every unsuccessful trial; what else broke varies across the other 8 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 6 of 15 checks — cp1_declined_transaction_count, cp4_top_reason_share_of_declined_usd_pct and cp6_mean_declined_amount_usd and 3 more.
- Trial 02Missed 1 of 15 checks — deliverable_slack_summary_content.
- Trial 03Missed 3 of 15 checks — cp7_authorization_decline_rate_pct, deliverable_slack_summary_content and rubric_decline_exposure.
- Trial 04Missed 6 of 15 checks — cp1_declined_transaction_count, cp4_top_reason_share_of_declined_usd_pct and cp6_mean_declined_amount_usd and 3 more.
- Trial 05Missed 9 of 15 checks — cp1_declined_transaction_count, cp2_top_decline_reason and cp3_top_reason_declined_usd and 6 more.
- Trial 06Missed 1 of 15 checks — deliverable_slack_summary_content.
- Trial 07Missed 3 of 15 checks — cp7_authorization_decline_rate_pct, deliverable_slack_summary_content and rubric_decline_exposure.
- Trial 08Missed 2 of 15 checks — cp1_declined_transaction_count and deliverable_slack_summary_content.
- Trial 09Missed 3 of 15 checks — cp7_authorization_decline_rate_pct, deliverable_slack_summary_content and rubric_decline_exposure.
- Trial 10Passed every verifier it was graded on.
Claude Opus 4.8
Mean reward
88%All trials
Across 10 trials on this task, reward ran from 66.7% to 100% — 5 of 10 cleared every check.
- 13.2 / 15avg checks passed
- 14avg turns
- 25avg tool calls
- 2mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 5 of 10 trials. Failures cluster on deliverable_slack_summary_content (5/10 trials).
Common pattern
deliverable_slack_summary_content failed in every unsuccessful trial; what else broke varies across the other 5 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 5 verifiers across 5 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Passed every verifier it was graded on.
- Trial 03Missed 2 of 15 checks — cp1_declined_transaction_count and deliverable_slack_summary_content.
- Trial 04Missed 3 of 15 checks — cp7_authorization_decline_rate_pct, deliverable_slack_summary_content and rubric_decline_exposure.
- Trial 05Missed 3 of 15 checks — cp7_authorization_decline_rate_pct, deliverable_slack_summary_content and rubric_decline_exposure.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Missed 5 of 15 checks — cp1_declined_transaction_count, cp4_top_reason_share_of_declined_usd_pct and cp6_mean_declined_amount_usd and 2 more.
- Trial 09Missed 5 of 15 checks — cp1_declined_transaction_count, cp4_top_reason_share_of_declined_usd_pct and cp6_mean_declined_amount_usd and 2 more.
- Trial 10Passed every verifier it was graded on.
Gemini 3.1 Pro
Mean reward
94.7%All trials
Across 10 trials on this task, reward ran from 93.3% to 100% — 2 of 10 cleared every check.
- 14.2 / 15avg checks passed
- 27avg turns
- 25avg tool calls
- 4mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 2 of 10 trials. Failures cluster on deliverable_jira_ticket_filed (8/10 trials).
Common pattern
All 8 unsuccessful trials broke on exactly the same verifiers — deliverable_jira_ticket_filed failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Missed 1 of 15 checks — deliverable_jira_ticket_filed.
- Trial 03Missed 1 of 15 checks — deliverable_jira_ticket_filed.
- Trial 04Passed every verifier it was graded on.
- Trial 05Missed 1 of 15 checks — deliverable_jira_ticket_filed.
- Trial 06Missed 1 of 15 checks — deliverable_jira_ticket_filed.
- Trial 07Missed 1 of 15 checks — deliverable_jira_ticket_filed.
- Trial 08Missed 1 of 15 checks — deliverable_jira_ticket_filed.
- Trial 09Missed 1 of 15 checks — deliverable_jira_ticket_filed.
- Trial 10Missed 1 of 15 checks — deliverable_jira_ticket_filed.
Gemini 3.5 Flash
Mean reward
28.7%All trials
Across 10 trials on this task, reward ran from 6.7% to 80% — 0 of 10 cleared every check.
- 4.3 / 15avg checks passed
- 80avg turns
- 78avg tool calls
- 47mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on deliverable_slack_summary_content (10/10 trials).
Common pattern
deliverable_slack_summary_content and rubric_decline_exposure failed in every unsuccessful trial; what else broke varies across the other 12 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Missing field — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 3 of 15 checks — cp7_authorization_decline_rate_pct, deliverable_slack_summary_content and rubric_decline_exposure.
- Trial 02Missed 14 of 15 checks — cp1_declined_transaction_count, cp2_top_decline_reason and cp3_top_reason_declined_usd and 11 more.
- Trial 03Missed 12 of 15 checks — cp1_declined_transaction_count, cp2_top_decline_reason and cp3_top_reason_declined_usd and 9 more.
- Trial 04Missed 13 of 15 checks — cp1_declined_transaction_count, cp2_top_decline_reason and cp3_top_reason_declined_usd and 10 more.
- Trial 05Missed 6 of 15 checks — cp7_authorization_decline_rate_pct, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 3 more.
- Trial 06Missed 14 of 15 checks — cp1_declined_transaction_count, cp2_top_decline_reason and cp3_top_reason_declined_usd and 11 more.
- Trial 07Missed 14 of 15 checks — cp1_declined_transaction_count, cp2_top_decline_reason and cp3_top_reason_declined_usd and 11 more.
- Trial 08Missed 9 of 15 checks — cp5_typical_declined_amount_usd, cp7_authorization_decline_rate_pct and cp8_realtime_decline_risk_score and 6 more.
- Trial 09Missed 14 of 15 checks — cp1_declined_transaction_count, cp2_top_decline_reason and cp3_top_reason_declined_usd and 11 more.
- Trial 10Missed 8 of 15 checks — cp7_authorization_decline_rate_pct, cp8_realtime_decline_risk_score and cp9_documented_oon_atm_fee_usd and 5 more.