Fraud Dispute Triage - Freshdesk & Slack CoordinationRisk & Fraud
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
100%All trials
Across 10 trials on this task, reward ran from 100% to 100% — 10 of 10 cleared every check.
- 25.0 / 25avg checks passed
- 9avg turns
- 20avg tool calls
- 2mavg time
Summary · GPT-5.5
GPT-5.5 cleared every check in all 10 trials.
Common pattern
Every trial cleared the core tier, so there is no failure pattern to read.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Passed every verifier it was graded on.
- Trial 03Passed every verifier it was graded on.
- Trial 04Passed every verifier it was graded on.
- Trial 05Passed every verifier it was graded on.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Passed every verifier it was graded on.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Claude Opus 4.8
Mean reward
98.8%All trials
Across 10 trials on this task, reward ran from 88% to 100% — 9 of 10 cleared every check.
- 24.7 / 25avg checks passed
- 10avg turns
- 16avg tool calls
- 2mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 9 of 10 trials. Failures cluster on only_updated_25675_priority (1/10 trials).
Common pattern
All 1 unsuccessful trials broke on exactly the same verifiers — only_updated_25675_priority, ticket_25675_priority_in_database and updated_ticket_25675_priority failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 1 verifier across 1 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Passed every verifier it was graded on.
- Trial 03Missed 3 of 25 checks — only_updated_25675_priority, ticket_25675_priority_in_database and updated_ticket_25675_priority.
- Trial 04Passed every verifier it was graded on.
- Trial 05Passed every verifier it was graded on.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Passed every verifier it was graded on.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Gemini 3.1 Pro
Mean reward
93.6%All trials
Across 10 trials on this task, reward ran from 92% to 96% — 0 of 10 cleared every check.
- 23.4 / 25avg checks passed
- 6avg turns
- 10avg tool calls
- 2mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on retrieved_ticket_25675 (10/10 trials).
Common pattern
retrieved_ticket_25675 failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 25 checks — retrieved_ticket_25675 and retrieved_ticket_52850.
- Trial 02Missed 2 of 25 checks — retrieved_ticket_25675 and retrieved_ticket_52850.
- Trial 03Missed 2 of 25 checks — retrieved_ticket_25675 and retrieved_ticket_52850.
- Trial 04Missed 2 of 25 checks — retrieved_ticket_25675 and retrieved_ticket_52850.
- Trial 05Missed 1 of 25 checks — retrieved_ticket_25675.
- Trial 06Missed 2 of 25 checks — retrieved_ticket_25675 and retrieved_ticket_52850.
- Trial 07Missed 2 of 25 checks — retrieved_ticket_25675 and retrieved_ticket_52850.
- Trial 08Missed 1 of 25 checks — retrieved_ticket_25675.
- Trial 09Missed 1 of 25 checks — retrieved_ticket_25675.
- Trial 10Missed 1 of 25 checks — retrieved_ticket_25675.
Gemini 3.5 Flash
Mean reward
60.4%All trials
Across 10 trials on this task, reward ran from 32% to 100% — 4 of 10 cleared every check.
- 15.1 / 25avg checks passed
- 21avg turns
- 23avg tool calls
- 4mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed 4 of 10 trials. Failures cluster on added_note_ticket_25675 (6/10 trials).
Common pattern
added_note_ticket_25675, added_note_ticket_52850 and channel_purpose_in_database and 13 more failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 6 verifiers across 6 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Missed 16 of 25 checks — added_note_ticket_25675, added_note_ticket_52850 and channel_purpose_in_database and 13 more.
- Trial 03Passed every verifier it was graded on.
- Trial 04Missed 17 of 25 checks — added_note_ticket_25675, added_note_ticket_52850 and channel_purpose_in_database and 14 more.
- Trial 05Passed every verifier it was graded on.
- Trial 06Missed 16 of 25 checks — added_note_ticket_25675, added_note_ticket_52850 and channel_purpose_in_database and 13 more.
- Trial 07Missed 16 of 25 checks — added_note_ticket_25675, added_note_ticket_52850 and channel_purpose_in_database and 13 more.
- Trial 08Missed 17 of 25 checks — added_note_ticket_25675, added_note_ticket_52850 and channel_purpose_in_database and 14 more.
- Trial 09Missed 17 of 25 checks — added_note_ticket_25675, added_note_ticket_52850 and channel_purpose_in_database and 14 more.
- Trial 10Passed every verifier it was graded on.