Card network mix review + Reg BI confirmationsCompliance & Regulatory
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
92.5%All trials
Across 10 trials on this task, reward ran from 75% to 100% — 6 of 10 cleared every check.
- 7.4 / 8avg checks passed
- 16avg turns
- 18avg tool calls
- 2mavg time
Summary · GPT-5.5
GPT-5.5 passed 6 of 10 trials. Failures cluster on cp4_mastercard_pct (4/10 trials).
Common pattern
cp4_mastercard_pct failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 3 verifiers across 3 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 8 checks — cp4_mastercard_pct.
- Trial 02Passed every verifier it was graded on.
- Trial 03Missed 2 of 8 checks — cp4_mastercard_pct and process_floor.
- Trial 04Missed 2 of 8 checks — cp4_mastercard_pct and process_floor.
- Trial 05Passed every verifier it was graded on.
- Trial 06Missed 1 of 8 checks — cp4_mastercard_pct.
- Trial 07Passed every verifier it was graded on.
- Trial 08Passed every verifier it was graded on.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Claude Opus 4.8
Mean reward
87.5%All trials
Across 10 trials on this task, reward ran from 75% to 100% — 1 of 10 cleared every check.
- 7.0 / 8avg checks passed
- 11avg turns
- 12avg tool calls
- 1mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 1 of 10 trials. Failures cluster on cp4_mastercard_pct (9/10 trials).
Common pattern
cp4_mastercard_pct failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 5 verifiers across 5 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 8 checks — cp4_mastercard_pct.
- Trial 02Missed 1 of 8 checks — cp4_mastercard_pct.
- Trial 03Missed 2 of 8 checks — cp1_reg_bi_effective_year and cp4_mastercard_pct.
- Trial 04Missed 1 of 8 checks — cp4_mastercard_pct.
- Trial 05Missed 1 of 8 checks — cp4_mastercard_pct.
- Trial 06Missed 1 of 8 checks — cp4_mastercard_pct.
- Trial 07Missed 1 of 8 checks — cp4_mastercard_pct.
- Trial 08Missed 1 of 8 checks — cp4_mastercard_pct.
- Trial 09Passed every verifier it was graded on.
- Trial 10Missed 1 of 8 checks — cp4_mastercard_pct.
Gemini 3.1 Pro
Mean reward
73.8%All trials
Across 10 trials on this task, reward ran from 37.5% to 87.5% — 0 of 10 cleared every check.
- 5.9 / 8avg checks passed
- 23avg turns
- 21avg tool calls
- 3mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on cp4_mastercard_pct (10/10 trials).
Common pattern
cp4_mastercard_pct failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 8 checks — cp4_mastercard_pct and deliverable_jira_ticket_filed.
- Trial 02Missed 2 of 8 checks — cp4_mastercard_pct and deliverable_jira_ticket_filed.
- Trial 03Missed 2 of 8 checks — cp4_mastercard_pct and deliverable_jira_ticket_filed.
- Trial 04Missed 1 of 8 checks — cp4_mastercard_pct.
- Trial 05Missed 5 of 8 checks — cp1_reg_bi_effective_year, cp2_reg_bi_broker_dealers_not_rias and cp3_reg_bi_higher_than_suitability and 2 more.
- Trial 06Missed 2 of 8 checks — cp4_mastercard_pct and deliverable_jira_ticket_filed.
- Trial 07Missed 2 of 8 checks — cp4_mastercard_pct and deliverable_jira_ticket_filed.
- Trial 08Missed 1 of 8 checks — cp4_mastercard_pct.
- Trial 09Missed 2 of 8 checks — cp4_mastercard_pct and deliverable_jira_ticket_filed.
- Trial 10Missed 2 of 8 checks — cp4_mastercard_pct and deliverable_jira_ticket_filed.
Gemini 3.5 Flash
Mean reward
53.8%All trials
Across 10 trials on this task, reward ran from 12.5% to 87.5% — 0 of 10 cleared every check.
- 4.3 / 8avg checks passed
- 40avg turns
- 39avg tool calls
- 14mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on cp4_mastercard_pct (10/10 trials).
Common pattern
cp4_mastercard_pct failed in every unsuccessful trial; what else broke varies across the other 6 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 4 of 8 checks — cp4_mastercard_pct, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 1 more.
- Trial 02Missed 4 of 8 checks — cp4_mastercard_pct, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 1 more.
- Trial 03Missed 1 of 8 checks — cp4_mastercard_pct.
- Trial 04Missed 4 of 8 checks — cp4_mastercard_pct, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 1 more.
- Trial 05Missed 7 of 8 checks — cp1_reg_bi_effective_year, cp2_reg_bi_broker_dealers_not_rias and cp3_reg_bi_higher_than_suitability and 4 more.
- Trial 06Missed 4 of 8 checks — cp4_mastercard_pct, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 1 more.
- Trial 07Missed 4 of 8 checks — cp4_mastercard_pct, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 1 more.
- Trial 08Missed 4 of 8 checks — cp4_mastercard_pct, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 1 more.
- Trial 09Missed 4 of 8 checks — cp4_mastercard_pct, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 1 more.
- Trial 10Missed 1 of 8 checks — cp4_mastercard_pct.