Fraud Dispute Triage - Freshdesk & Slack CoordinationRisk & Fraud
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one trial to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over trials. N = 10 trials per model.
100%
GPT-5.5
99%
Claude Opus 4.8
94%
Gemini 3.1 Pro
60%
Gemini 3.5 Flash
Extended metrics
Median Time
Wall-clock minutes, median across trials. N = 10 trials per model.
3m
Gemini 3.5 Flash
2m
GPT-5.5
2m
Claude Opus 4.8
2m
Gemini 3.1 Pro
Tool Calls
Tool invocations, median across trials — counted from the trajectories. N = 10 trials per model.
21
GPT-5.5
21
Gemini 3.5 Flash
16
Claude Opus 4.8
10
Gemini 3.1 Pro
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | GPT-5.5n=10 | Claude Opus 4.8n=10 | Gemini 3.1 Pron=10 | Gemini 3.5 Flashn=10 |
|---|---|---|---|---|
| checks (25) | ||||
| added_note_ticket_25675 | 100% | 100% | 100% | 40% |
| added_note_ticket_52850 | 100% | 100% | 100% | 40% |
| avoided_destructive_ticket_actions | 100% | 100% | 100% | 100% |
| avoided_public_replies | 100% | 100% | 100% | 100% |
| channel_purpose_in_database | 100% | 100% | 100% | 40% |
| conversations_ticket_25675 | 100% | 100% | 100% | 100% |
| conversations_ticket_52850 | 100% | 100% | 100% | 100% |
| created_fraud_triage_channel | 100% | 100% | 100% | 40% |
| created_only_one_slack_channel | 100% | 100% | 100% | 40% |
| fraud_triage_channel_in_database | 100% | 100% | 100% | 40% |
| only_updated_25675_priority | 100% | 90% | 100% | 40% |
| other_dispute_priorities_unchanged | 100% | 100% | 100% | 100% |
| pinned_triage_summary | 100% | 100% | 100% | 40% |
| posted_triage_summary | 100% | 100% | 100% | 40% |
| retrieved_ticket_25675 | 100% | 100% | 0% | 100% |
| retrieved_ticket_52850 | 100% | 100% | 40% | 100% |
| searched_open_dispute_tickets | 100% | 100% | 100% | 70% |
| set_channel_purpose | 100% | 100% | 100% | 40% |
| ticket_25675_priority_in_database | 100% | 90% | 100% | 40% |
| triage_message_mentions_25675 | 100% | 100% | 100% | 40% |
| triage_message_mentions_52850 | 100% | 100% | 100% | 40% |
| triage_message_states_4_open_disputes | 100% | 100% | 100% | 40% |
| triage_notes_only_on_25675_and_52850 | 100% | 100% | 100% | 100% |
| triage_summary_pinned_in_database | 100% | 100% | 100% | 40% |
| updated_ticket_25675_priority | 100% | 90% | 100% | 40% |
Failure modes on this task
17 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | GPT-5.5 | Claude Opus 4.8 | Gemini 3.1 Pro | Gemini 3.5 Flash |
|---|---|---|---|---|
| Answer1 mode | — | 1 | 10 | 6 |