IDA suspect-fraud resolution codeRisk & Fraud
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
84.6%All trials
Across 10 trials on this task, reward ran from 46.2% to 100% — 6 of 10 cleared every check.
- 11.0 / 13avg checks passed
- 56avg turns
- 206avg tool calls
- 12mavg time
Summary · GPT-5.5
GPT-5.5 passed 6 of 10 trials. Failures cluster on deliverable_slack_message_content (4/10 trials).
Common pattern
deliverable_slack_message_content and trace_ida_dual_method failed in every unsuccessful trial; what else broke varies across the other 6 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 3 verifiers across 3 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Missed 7 of 13 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 4 more.
- Trial 03Missed 3 of 13 checks — answer_both_net_fraud_loss_cents, deliverable_slack_message_content and trace_ida_dual_method.
- Trial 04Passed every verifier it was graded on.
- Trial 05Passed every verifier it was graded on.
- Trial 06Missed 7 of 13 checks — answer_both_net_fraud_loss_cents, deliverable_jira_ticket_filed and deliverable_slack_message_content and 4 more.
- Trial 07Passed every verifier it was graded on.
- Trial 08Passed every verifier it was graded on.
- Trial 09Missed 3 of 13 checks — answer_both_net_fraud_loss_cents, deliverable_slack_message_content and trace_ida_dual_method.
- Trial 10Passed every verifier it was graded on.
Claude Opus 4.8
Mean reward
56.2%All trials
Across 10 trials on this task, reward ran from 38.5% to 76.9% — 0 of 10 cleared every check.
- 7.3 / 13avg checks passed
- 41avg turns
- 67avg tool calls
- 7mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed none of its 10 trials. Failures cluster on deliverable_slack_message_content (10/10 trials).
Common pattern
deliverable_slack_message_content and trace_ida_dual_method failed in every unsuccessful trial; what else broke varies across the other 9 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 7 of 13 checks — answer_ida_alert_closure_accounts_count, deliverable_jira_ticket_filed and deliverable_slack_message_content and 4 more.
- Trial 02Missed 3 of 13 checks — answer_both_net_fraud_loss_cents, deliverable_slack_message_content and trace_ida_dual_method.
- Trial 03Missed 8 of 13 checks — answer_both_net_fraud_loss_cents, answer_case_a_ida_resolution_code and answer_case_b_ida_resolution_code and 5 more.
- Trial 04Missed 8 of 13 checks — answer_both_net_fraud_loss_cents, answer_ida_alert_closure_accounts_count and deliverable_jira_ticket_filed and 5 more.
- Trial 05Missed 4 of 13 checks — answer_both_net_fraud_loss_cents, answer_opened_and_flag_contrast and deliverable_slack_message_content and 1 more.
- Trial 06Missed 5 of 13 checks.
- Trial 07Missed 6 of 13 checks — answer_both_net_fraud_loss_cents, answer_ida_alert_closure_accounts_count and answer_opened_and_flag_contrast and 3 more.
- Trial 08Missed 5 of 13 checks.
- Trial 09Missed 5 of 13 checks.
- Trial 10Missed 6 of 13 checks — answer_both_net_fraud_loss_cents, answer_ida_alert_closure_accounts_count and answer_opened_and_flag_contrast and 3 more.
Gemini 3.1 Pro
Mean reward
88.5%All trials
Across 10 trials on this task, reward ran from 69.2% to 92.3% — 0 of 10 cleared every check.
- 11.5 / 13avg checks passed
- 50avg turns
- 48avg tool calls
- 5mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on deliverable_jira_ticket_filed (9/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 5 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 13 checks — deliverable_jira_ticket_filed.
- Trial 02Missed 1 of 13 checks — deliverable_jira_ticket_filed.
- Trial 03Missed 4 of 13 checks — answer_both_net_fraud_loss_cents, deliverable_slack_message_content and trace_all_connectors_and_trap and 1 more.
- Trial 04Missed 1 of 13 checks — deliverable_jira_ticket_filed.
- Trial 05Missed 1 of 13 checks — deliverable_jira_ticket_filed.
- Trial 06Missed 2 of 13 checks — deliverable_jira_ticket_filed and trace_all_connectors_and_trap.
- Trial 07Missed 1 of 13 checks — deliverable_jira_ticket_filed.
- Trial 08Missed 1 of 13 checks — deliverable_jira_ticket_filed.
- Trial 09Missed 2 of 13 checks — deliverable_jira_ticket_filed and trace_all_connectors_and_trap.
- Trial 10Missed 1 of 13 checks — deliverable_jira_ticket_filed.
Gemini 3.5 Flash
Mean reward
43.1%All trials
Across 10 trials on this task, reward ran from 0% to 92.3% — 1 of 10 cleared every check.
- 5.6 / 13avg checks passed
- 59avg turns
- 57avg tool calls
- 10mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed 1 of 10 trials. Failures cluster on deliverable_slack_message_content (9/10 trials).
Common pattern
deliverable_slack_message_content and trace_ida_dual_method failed in every unsuccessful trial; what else broke varies across the other 11 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 13 checks — trace_all_connectors_and_trap.
- Trial 02Missed 8 of 13 checks — answer_ida_alert_closure_accounts_count, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 5 more.
- Trial 03Missed 8 of 13 checks — answer_ida_alert_closure_accounts_count, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 5 more.
- Trial 04Missed 8 of 13 checks — answer_ida_alert_closure_accounts_count, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 5 more.
- Trial 05Missed 7 of 13 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 4 more.
- Trial 06Missed 8 of 13 checks — answer_ida_alert_closure_accounts_count, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 5 more.
- Trial 07Missed 8 of 13 checks — answer_ida_alert_closure_accounts_count, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 5 more.
- Trial 08Missed 3 of 13 checks — answer_both_net_fraud_loss_cents, deliverable_slack_message_content and trace_ida_dual_method.
- Trial 09Missed 10 of 13 checks.
- Trial 10Missed 13 of 13 checks — answer_both_net_fraud_loss_cents, answer_case_a_ida_resolution_code and answer_case_b_ida_resolution_code and 10 more.