Duplicate-pending complaint audit — CX hold-confusion triageCards & Interchange
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
81.5%All trials
Across 10 trials on this task, reward ran from 69.2% to 92.3% — 0 of 10 cleared every check.
- 10.6 / 13avg checks passed
- 23avg turns
- 29avg tool calls
- 3mavg time
Summary · GPT-5.5
GPT-5.5 passed none of its 10 trials. Failures cluster on deliverable_slack_message_content (10/10 trials).
Common pattern
deliverable_slack_message_content failed in every unsuccessful trial; what else broke varies across the other 3 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 3 of 13 checks — answer_harm_split_and_refund, deliverable_slack_message_content and answer_incidents_and_holds.
- Trial 02Missed 2 of 13 checks — answer_harm_split_and_refund and deliverable_slack_message_content.
- Trial 03Missed 2 of 13 checks — answer_harm_split_and_refund and deliverable_slack_message_content.
- Trial 04Missed 2 of 13 checks — answer_harm_split_and_refund and deliverable_slack_message_content.
- Trial 05Missed 3 of 13 checks — answer_harm_split_and_refund, deliverable_slack_message_content and answer_incidents_and_holds.
- Trial 06Missed 2 of 13 checks — answer_harm_split_and_refund and deliverable_slack_message_content.
- Trial 07Missed 4 of 13 checks — answer_harm_split_and_refund, deliverable_slack_message_content and answer_incidents_and_holds and 1 more.
- Trial 08Missed 1 of 13 checks — deliverable_slack_message_content.
- Trial 09Missed 2 of 13 checks — answer_harm_split_and_refund and deliverable_slack_message_content.
- Trial 10Missed 3 of 13 checks — answer_harm_split_and_refund, deliverable_slack_message_content and answer_incidents_and_holds.
Claude Opus 4.8
Mean reward
80%All trials
Across 10 trials on this task, reward ran from 76.9% to 84.6% — 0 of 10 cleared every check.
- 10.4 / 13avg checks passed
- 17avg turns
- 19avg tool calls
- 3mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed none of its 10 trials. Failures cluster on answer_harm_split_and_refund (10/10 trials).
Common pattern
answer_harm_split_and_refund and deliverable_slack_message_content failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 3 of 13 checks — answer_harm_split_and_refund, deliverable_slack_message_content and answer_incidents_and_holds.
- Trial 02Missed 3 of 13 checks — answer_harm_split_and_refund, deliverable_slack_message_content and answer_incidents_and_holds.
- Trial 03Missed 3 of 13 checks — answer_harm_split_and_refund, deliverable_slack_message_content and answer_incidents_and_holds.
- Trial 04Missed 3 of 13 checks — answer_harm_split_and_refund, deliverable_slack_message_content and answer_incidents_and_holds.
- Trial 05Missed 3 of 13 checks — answer_harm_split_and_refund, deliverable_slack_message_content and answer_incidents_and_holds.
- Trial 06Missed 2 of 13 checks — answer_harm_split_and_refund and deliverable_slack_message_content.
- Trial 07Missed 3 of 13 checks — answer_harm_split_and_refund, deliverable_slack_message_content and answer_incidents_and_holds.
- Trial 08Missed 2 of 13 checks — answer_harm_split_and_refund and deliverable_slack_message_content.
- Trial 09Missed 2 of 13 checks — answer_harm_split_and_refund and deliverable_slack_message_content.
- Trial 10Missed 2 of 13 checks — answer_harm_split_and_refund and deliverable_slack_message_content.
Gemini 3.1 Pro
Mean reward
57.7%All trials
Across 10 trials on this task, reward ran from 38.5% to 76.9% — 0 of 10 cleared every check.
- 7.5 / 13avg checks passed
- 20avg turns
- 18avg tool calls
- 3mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on answer_harm_split_and_refund (10/10 trials).
Common pattern
answer_harm_split_and_refund and deliverable_slack_message_content failed in every unsuccessful trial; what else broke varies across the other 8 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 7 of 13 checks — answer_complaints_reviewed, answer_harm_split_and_refund and deliverable_jira_ticket_filed and 4 more.
- Trial 02Missed 3 of 13 checks — answer_harm_split_and_refund, deliverable_slack_message_content and answer_incidents_and_holds.
- Trial 03Missed 4 of 13 checks — answer_complaints_reviewed, answer_harm_split_and_refund and deliverable_slack_message_content and 1 more.
- Trial 04Missed 7 of 13 checks — answer_complaints_reviewed, answer_harm_split_and_refund and deliverable_jira_ticket_filed and 4 more.
- Trial 05Missed 8 of 13 checks — answer_harm_split_and_refund, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 5 more.
- Trial 06Missed 4 of 13 checks — answer_complaints_reviewed, answer_harm_split_and_refund and deliverable_slack_message_content and 1 more.
- Trial 07Missed 4 of 13 checks — answer_complaints_reviewed, answer_harm_split_and_refund and deliverable_slack_message_content and 1 more.
- Trial 08Missed 7 of 13 checks — answer_complaints_reviewed, answer_harm_split_and_refund and deliverable_jira_ticket_filed and 4 more.
- Trial 09Missed 4 of 13 checks — answer_complaints_reviewed, answer_harm_split_and_refund and deliverable_slack_message_content and 1 more.
- Trial 10Missed 7 of 13 checks — answer_complaints_reviewed, answer_harm_split_and_refund and deliverable_jira_ticket_filed and 4 more.
Gemini 3.5 Flash
Mean reward
40%All trials
Across 10 trials on this task, reward ran from 15.4% to 100% — 2 of 10 cleared every check.
- 5.2 / 13avg checks passed
- 48avg turns
- 46avg tool calls
- 17mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed 2 of 10 trials. Failures cluster on answer_harm_split_and_refund (8/10 trials).
Common pattern
answer_harm_split_and_refund, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 4 more failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Missed 8 of 13 checks — answer_harm_split_and_refund, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 5 more.
- Trial 03Missed 11 of 13 checks — answer_harm_split_and_refund, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 8 more.
- Trial 04Missed 10 of 13 checks — answer_harm_split_and_refund, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 7 more.
- Trial 05Missed 11 of 13 checks — answer_harm_split_and_refund, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 8 more.
- Trial 06Missed 10 of 13 checks — answer_harm_split_and_refund, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 7 more.
- Trial 07Passed every verifier it was graded on.
- Trial 08Missed 10 of 13 checks — answer_harm_split_and_refund, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 7 more.
- Trial 09Missed 7 of 13 checks — answer_harm_split_and_refund, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 4 more.
- Trial 10Missed 11 of 13 checks — answer_harm_split_and_refund, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 8 more.