CX outbound-wire reconcilePayments & Transfers
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
85%All trials
Across 10 trials on this task, reward ran from 75% to 100% — 8 of 10 cleared every check.
- 3.4 / 4avg checks passed
- 11avg turns
- 26avg tool calls
- 2mavg time
Summary · GPT-5.5
GPT-5.5 passed 8 of 10 trials. Failures cluster on process_floor (4/10 trials).
Common pattern
All 2 unsuccessful trials broke on exactly the same verifiers — cp2_cx_squad_wire_mentions failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 2 verifiers across 2 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Missed 1 of 4 checks — cp2_cx_squad_wire_mentions.
- Trial 03Missed 1 of 4 checks — process_floor.
- Trial 04Missed 1 of 4 checks — process_floor.
- Trial 05Passed every verifier it was graded on.
- Trial 06Missed 1 of 4 checks — cp2_cx_squad_wire_mentions.
- Trial 07Missed 1 of 4 checks — process_floor.
- Trial 08Passed every verifier it was graded on.
- Trial 09Passed every verifier it was graded on.
- Trial 10Missed 1 of 4 checks — process_floor.
Claude Opus 4.8
Mean reward
82.5%All trials
Across 10 trials on this task, reward ran from 75% to 100% — 10 of 10 cleared every check.
- 3.3 / 4avg checks passed
- 22avg turns
- 71avg tool calls
- 4mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 cleared every check in all 10 trials.
Common pattern
Every trial cleared the core tier, so there is no failure pattern to read.
Per-trial notes
- Trial 01Missed 1 of 4 checks — process_floor.
- Trial 02Missed 1 of 4 checks — process_floor.
- Trial 03Missed 1 of 4 checks — process_floor.
- Trial 04Passed every verifier it was graded on.
- Trial 05Passed every verifier it was graded on.
- Trial 06Missed 1 of 4 checks — process_floor.
- Trial 07Missed 1 of 4 checks — process_floor.
- Trial 08Missed 1 of 4 checks — process_floor.
- Trial 09Missed 1 of 4 checks — process_floor.
- Trial 10Passed every verifier it was graded on.
Gemini 3.1 Pro
Mean reward
47.5%All trials
Across 10 trials on this task, reward ran from 25% to 75% — 4 of 10 cleared every check.
- 1.9 / 4avg checks passed
- 12avg turns
- 12avg tool calls
- 1mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 4 of 10 trials. Failures cluster on process_floor (10/10 trials).
Common pattern
cp2_cx_squad_wire_mentions and process_floor failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 6 verifiers across 6 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 4 checks — process_floor.
- Trial 02Missed 3 of 4 checks — cp1_outbound_wire_notifications, cp2_cx_squad_wire_mentions and process_floor.
- Trial 03Missed 3 of 4 checks — cp1_outbound_wire_notifications, cp2_cx_squad_wire_mentions and process_floor.
- Trial 04Missed 3 of 4 checks — cp1_outbound_wire_notifications, cp2_cx_squad_wire_mentions and process_floor.
- Trial 05Missed 3 of 4 checks — cp1_outbound_wire_notifications, cp2_cx_squad_wire_mentions and process_floor.
- Trial 06Missed 3 of 4 checks — cp1_outbound_wire_notifications, cp2_cx_squad_wire_mentions and process_floor.
- Trial 07Missed 1 of 4 checks — process_floor.
- Trial 08Missed 2 of 4 checks — cp2_cx_squad_wire_mentions and process_floor.
- Trial 09Missed 1 of 4 checks — process_floor.
- Trial 10Missed 1 of 4 checks — process_floor.
Gemini 3.5 Flash
Mean reward
42.5%All trials
Across 10 trials on this task, reward ran from 0% to 75% — 0 of 10 cleared every check.
- 1.7 / 4avg checks passed
- 39avg turns
- 49avg tool calls
- 15mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on cp2_cx_squad_wire_mentions (10/10 trials).
Common pattern
cp2_cx_squad_wire_mentions failed in every unsuccessful trial; what else broke varies across the other 3 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 4 checks — cp2_cx_squad_wire_mentions.
- Trial 02Missed 4 of 4 checks — cp1_outbound_wire_notifications, cp2_cx_squad_wire_mentions and deliverable_freshdesk_ticket and 1 more.
- Trial 03Missed 4 of 4 checks — cp1_outbound_wire_notifications, cp2_cx_squad_wire_mentions and deliverable_freshdesk_ticket and 1 more.
- Trial 04Missed 2 of 4 checks — cp1_outbound_wire_notifications and cp2_cx_squad_wire_mentions.
- Trial 05Missed 1 of 4 checks — cp2_cx_squad_wire_mentions.
- Trial 06Missed 1 of 4 checks — cp2_cx_squad_wire_mentions.
- Trial 07Missed 1 of 4 checks — cp2_cx_squad_wire_mentions.
- Trial 08Missed 4 of 4 checks — cp1_outbound_wire_notifications, cp2_cx_squad_wire_mentions and deliverable_freshdesk_ticket and 1 more.
- Trial 09Missed 1 of 4 checks — cp2_cx_squad_wire_mentions.
- Trial 10Missed 4 of 4 checks — cp1_outbound_wire_notifications, cp2_cx_squad_wire_mentions and deliverable_freshdesk_ticket and 1 more.