Additional-owner card activation funnelCards & Interchange
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
40.8%All trials
Across 10 trials on this task, reward ran from 7.7% to 100% — 1 of 10 cleared every check.
- 5.3 / 13avg checks passed
- 68avg turns
- 291avg tool calls
- 15mavg time
Summary · GPT-5.5
GPT-5.5 passed 1 of 10 trials. Failures cluster on answer_additional_owner_count (9/10 trials).
Common pattern
answer_additional_owner_count, answer_paid_extra_card_users and answer_virtual_funnel_activation_wallet and 3 more failed in every unsuccessful trial; what else broke varies across the other 6 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Missed 7 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 4 more.
- Trial 02Missed 12 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 9 more.
- Trial 03Passed every verifier it was graded on.
- Trial 04Missed 8 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 5 more.
- Trial 05Missed 8 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 5 more.
- Trial 06Missed 8 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 5 more.
- Trial 07Missed 7 of 13 checks — answer_additional_owner_count, answer_paid_extra_card_users and answer_virtual_funnel_activation_wallet and 4 more.
- Trial 08Missed 8 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 5 more.
- Trial 09Missed 11 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 8 more.
- Trial 10Missed 8 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 5 more.
Claude Opus 4.8
Mean reward
88.5%All trials
Across 10 trials on this task, reward ran from 46.2% to 100% — 6 of 10 cleared every check.
- 11.5 / 13avg checks passed
- 33avg turns
- 58avg tool calls
- 5mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 6 of 10 trials. Failures cluster on answer_virtual_funnel_activation_wallet (4/10 trials).
Common pattern
answer_virtual_funnel_activation_wallet and deliverable_slack_message_content failed in every unsuccessful trial; what else broke varies across the other 6 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 4 verifiers across 4 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Passed every verifier it was graded on.
- Trial 03Passed every verifier it was graded on.
- Trial 04Passed every verifier it was graded on.
- Trial 05Passed every verifier it was graded on.
- Trial 06Missed 3 of 13 checks — answer_virtual_funnel_activation_wallet, deliverable_slack_message_content and trace_trap_avoidance.
- Trial 07Missed 7 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 4 more.
- Trial 08Passed every verifier it was graded on.
- Trial 09Missed 3 of 13 checks — answer_virtual_funnel_activation_wallet, deliverable_slack_message_content and trace_trap_avoidance.
- Trial 10Missed 2 of 13 checks — answer_virtual_funnel_activation_wallet and deliverable_slack_message_content.
Gemini 3.1 Pro
Mean reward
52.3%All trials
Across 10 trials on this task, reward ran from 30.8% to 100% — 1 of 10 cleared every check.
- 6.8 / 13avg checks passed
- 58avg turns
- 56avg tool calls
- 10mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 1 of 10 trials. Failures cluster on answer_additional_owner_count (7/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 9 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 8 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 5 more.
- Trial 02Passed every verifier it was graded on.
- Trial 03Missed 9 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 6 more.
- Trial 04Missed 7 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 4 more.
- Trial 05Missed 9 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 6 more.
- Trial 06Missed 8 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 5 more.
- Trial 07Missed 9 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 6 more.
- Trial 08Missed 2 of 13 checks — deliverable_jira_ticket_filed and trace_trap_avoidance.
- Trial 09Missed 9 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 6 more.
- Trial 10Missed 1 of 13 checks — deliverable_jira_ticket_filed.
Gemini 3.5 Flash
Mean reward
14.6%All trials
Across 10 trials on this task, reward ran from 7.7% to 15.4% — 0 of 10 cleared every check.
- 1.9 / 13avg checks passed
- 31avg turns
- 28avg tool calls
- 9mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on answer_additional_owner_count (10/10 trials).
Common pattern
answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 8 more failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 12 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 9 more.
- Trial 02Missed 11 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 8 more.
- Trial 03Missed 11 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 8 more.
- Trial 04Missed 11 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 8 more.
- Trial 05Missed 11 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 8 more.
- Trial 06Missed 11 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 8 more.
- Trial 07Missed 11 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 8 more.
- Trial 08Missed 11 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 8 more.
- Trial 09Missed 11 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 8 more.
- Trial 10Missed 11 of 13 checks — answer_additional_owner_count, answer_never_ordered_count and answer_paid_extra_card_users and 8 more.