Q4 Fraud Impact Report - Email & Ticket InvestigationRisk & Fraud
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
47.3%All trials
Across 10 trials on this task, reward ran from 36.4% to 90.9% — 0 of 10 cleared every check.
- 5.2 / 11avg checks passed
- 22avg turns
- 69avg tool calls
- 3mavg time
Summary · GPT-5.5
GPT-5.5 passed none of its 10 trials. Failures cluster on slack_lists_ticket_11533 (10/10 trials).
Common pattern
slack_lists_ticket_11533 failed in every unsuccessful trial; what else broke varies across the other 7 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 6 of 11 checks — slack_lists_ticket_11533, slack_lists_ticket_20644 and checked_conversations_ticket_11533 and 3 more.
- Trial 02Missed 1 of 11 checks — slack_lists_ticket_11533.
- Trial 03Missed 7 of 11 checks — found_fraudulent_activity_thread, slack_lists_ticket_11533 and slack_lists_ticket_20644 and 4 more.
- Trial 04Missed 4 of 11 checks — found_fraudulent_activity_thread, slack_lists_ticket_11533 and checked_conversations_ticket_11533 and 1 more.
- Trial 05Missed 7 of 11 checks — found_fraudulent_activity_thread, slack_lists_ticket_11533 and slack_lists_ticket_20644 and 4 more.
- Trial 06Missed 6 of 11 checks — slack_lists_ticket_11533, slack_lists_ticket_20644 and checked_conversations_ticket_11533 and 3 more.
- Trial 07Missed 7 of 11 checks — found_fraudulent_activity_thread, slack_lists_ticket_11533 and slack_lists_ticket_20644 and 4 more.
- Trial 08Missed 6 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and slack_lists_ticket_11533 and 3 more.
- Trial 09Missed 7 of 11 checks — found_unauthorized_charges_thread, slack_lists_ticket_11533 and slack_lists_ticket_20644 and 4 more.
- Trial 10Missed 7 of 11 checks — found_fraudulent_activity_thread, slack_lists_ticket_11533 and slack_lists_ticket_20644 and 4 more.
Claude Opus 4.8
Mean reward
44.5%All trials
Across 10 trials on this task, reward ran from 27.3% to 72.7% — 0 of 10 cleared every check.
- 4.9 / 11avg checks passed
- 13avg turns
- 36avg tool calls
- 3mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed none of its 10 trials. Failures cluster on checked_conversations_ticket_20644 (10/10 trials).
Common pattern
checked_conversations_ticket_20644 and found_fraud_ticket_20644 failed in every unsuccessful trial; what else broke varies across the other 6 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 6 of 11 checks — slack_lists_ticket_11533, slack_lists_ticket_20644 and checked_conversations_ticket_11533 and 3 more.
- Trial 02Missed 5 of 11 checks — found_unauthorized_charges_thread, checked_conversations_ticket_11533 and checked_conversations_ticket_20644 and 2 more.
- Trial 03Missed 7 of 11 checks — found_fraudulent_activity_thread, slack_lists_ticket_11533 and slack_lists_ticket_20644 and 4 more.
- Trial 04Missed 8 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and slack_lists_ticket_11533 and 5 more.
- Trial 05Missed 6 of 11 checks — slack_lists_ticket_11533, slack_lists_ticket_20644 and checked_conversations_ticket_11533 and 3 more.
- Trial 06Missed 3 of 11 checks — found_unauthorized_charges_thread, checked_conversations_ticket_20644 and found_fraud_ticket_20644.
- Trial 07Missed 5 of 11 checks — found_fraudulent_activity_thread, checked_conversations_ticket_11533 and checked_conversations_ticket_20644 and 2 more.
- Trial 08Missed 5 of 11 checks — found_fraudulent_activity_thread, checked_conversations_ticket_11533 and checked_conversations_ticket_20644 and 2 more.
- Trial 09Missed 8 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and slack_lists_ticket_11533 and 5 more.
- Trial 10Missed 8 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and slack_lists_ticket_11533 and 5 more.
Gemini 3.1 Pro
Mean reward
18.2%All trials
Across 10 trials on this task, reward ran from 18.2% to 18.2% — 0 of 10 cleared every check.
- 2.0 / 11avg checks passed
- 9avg turns
- 13avg tool calls
- 1mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on found_fraudulent_activity_thread (10/10 trials).
Common pattern
All 10 unsuccessful trials broke on exactly the same verifiers — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.
- Trial 02Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.
- Trial 03Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.
- Trial 04Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.
- Trial 05Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.
- Trial 06Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.
- Trial 07Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.
- Trial 08Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.
- Trial 09Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.
- Trial 10Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.
Gemini 3.5 Flash
Mean reward
18.2%All trials
Across 10 trials on this task, reward ran from 18.2% to 18.2% — 0 of 10 cleared every check.
- 2.0 / 11avg checks passed
- 4avg turns
- 1avg tool calls
- 1mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on found_fraudulent_activity_thread (10/10 trials).
Common pattern
All 10 unsuccessful trials broke on exactly the same verifiers — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.
- Trial 02Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.
- Trial 03Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.
- Trial 04Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.
- Trial 05Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.
- Trial 06Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.
- Trial 07Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.
- Trial 08Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.
- Trial 09Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.
- Trial 10Missed 9 of 11 checks — found_fraudulent_activity_thread, found_unauthorized_charges_thread and q4_fraud_impact_channel_in_database and 6 more.