Bill-pay lifecycle: reconcile paid bills to ledger transactionsReconciliation & Settlement
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
83.1%All trials
Across 10 trials on this task, reward ran from 76.9% to 92.3% — 3 of 10 cleared every check.
- 10.8 / 13avg checks passed
- 27avg turns
- 45avg tool calls
- 4mavg time
Summary · GPT-5.5
GPT-5.5 passed 3 of 10 trials. Failures cluster on trace_reconcile_method (9/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 4 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 13 checks — trace_reconcile_method.
- Trial 02Missed 3 of 13 checks — answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method.
- Trial 03Missed 3 of 13 checks — answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method.
- Trial 04Missed 3 of 13 checks — answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method.
- Trial 05Missed 1 of 13 checks — trace_reconcile_method.
- Trial 06Missed 3 of 13 checks — answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method.
- Trial 07Missed 3 of 13 checks — answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method.
- Trial 08Missed 3 of 13 checks — answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method.
- Trial 09Missed 1 of 13 checks — trace_reconcile_method.
- Trial 10Missed 1 of 13 checks — rubric_phantom_root_cause.
Claude Opus 4.8
Mean reward
73.1%All trials
Across 10 trials on this task, reward ran from 61.5% to 100% — 1 of 10 cleared every check.
- 9.5 / 13avg checks passed
- 19avg turns
- 28avg tool calls
- 3mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 1 of 10 trials. Failures cluster on answer_exception_totals (9/10 trials).
Common pattern
answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method failed in every unsuccessful trial; what else broke varies across the other 2 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 5 of 13 checks — answer_exception_totals, deliverable_jira_ticket_filed and floor_write_back and 2 more.
- Trial 02Missed 3 of 13 checks — answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method.
- Trial 03Missed 3 of 13 checks — answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method.
- Trial 04Missed 5 of 13 checks — answer_exception_totals, deliverable_jira_ticket_filed and floor_write_back and 2 more.
- Trial 05Missed 3 of 13 checks — answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method.
- Trial 06Missed 5 of 13 checks — answer_exception_totals, deliverable_jira_ticket_filed and floor_write_back and 2 more.
- Trial 07Passed every verifier it was graded on.
- Trial 08Missed 5 of 13 checks — answer_exception_totals, deliverable_jira_ticket_filed and floor_write_back and 2 more.
- Trial 09Missed 3 of 13 checks — answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method.
- Trial 10Missed 3 of 13 checks — answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method.
Gemini 3.1 Pro
Mean reward
72.3%All trials
Across 10 trials on this task, reward ran from 46.2% to 76.9% — 0 of 10 cleared every check.
- 9.4 / 13avg checks passed
- 30avg turns
- 29avg tool calls
- 13mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on answer_exception_totals (10/10 trials).
Common pattern
answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 3 of 13 checks — answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method.
- Trial 02Missed 3 of 13 checks — answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method.
- Trial 03Missed 4 of 13 checks — answer_exception_totals, deliverable_jira_ticket_filed and trace_all_connectors_and_trap and 1 more.
- Trial 04Missed 7 of 13 checks — answer_exception_totals, answer_lifecycle_buckets and answer_reconciliation_rate and 4 more.
- Trial 05Missed 3 of 13 checks — answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method.
- Trial 06Missed 3 of 13 checks — answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method.
- Trial 07Missed 3 of 13 checks — answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method.
- Trial 08Missed 3 of 13 checks — answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method.
- Trial 09Missed 4 of 13 checks — answer_exception_totals, deliverable_jira_ticket_filed and trace_all_connectors_and_trap and 1 more.
- Trial 10Missed 3 of 13 checks — answer_exception_totals, trace_all_connectors_and_trap and trace_reconcile_method.
Gemini 3.5 Flash
Mean reward
43.8%All trials
Across 10 trials on this task, reward ran from 23.1% to 100% — 1 of 10 cleared every check.
- 5.7 / 13avg checks passed
- 61avg turns
- 59avg tool calls
- 28mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed 1 of 10 trials. Failures cluster on deliverable_jira_ticket_filed (9/10 trials).
Common pattern
deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 3 more failed in every unsuccessful trial; what else broke varies across the other 6 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 6 verifiers across 6 of 10 trials.
Per-trial notes
- Trial 01Missed 10 of 13 checks — answer_exception_totals, answer_lifecycle_buckets and answer_reconciliation_rate and 7 more.
- Trial 02Missed 10 of 13 checks — answer_exception_totals, answer_lifecycle_buckets and answer_reconciliation_rate and 7 more.
- Trial 03Missed 7 of 13 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 4 more.
- Trial 04Missed 10 of 13 checks — answer_exception_totals, answer_lifecycle_buckets and answer_reconciliation_rate and 7 more.
- Trial 05Missed 8 of 13 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 5 more.
- Trial 06Missed 6 of 13 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 3 more.
- Trial 07Passed every verifier it was graded on.
- Trial 08Missed 8 of 13 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 5 more.
- Trial 09Missed 6 of 13 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 3 more.
- Trial 10Missed 8 of 13 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 5 more.