Reconstruct cancelled-check return-then-re-debit ledger and net balance impactReconciliation & Settlement
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
39.3%All trials
Across 10 trials on this task, reward ran from 6.7% to 60% — 0 of 10 cleared every check.
- 5.9 / 15avg checks passed
- 115avg turns
- 223avg tool calls
- 27mavg time
Summary · GPT-5.5
GPT-5.5 passed none of its 10 trials. Failures cluster on cp_gross_dollars_moved (10/10 trials).
Common pattern
cp_gross_dollars_moved, cp_net_exposure_usd and cp_outstanding_overdebit_usd and 1 more failed in every unsuccessful trial; what else broke varies across the other 10 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 6 of 15 checks — cp_gross_dollars_moved, cp_net_exposure_usd and cp_outstanding_overdebit_usd and 3 more.
- Trial 02Missed 8 of 15 checks — cp_gross_dollars_moved, cp_net_exposure_usd and cp_outstanding_overdebit_usd and 5 more.
- Trial 03Missed 12 of 15 checks — cp_fully_corrected_count, cp_gross_dollars_moved and cp_net_exposure_usd and 9 more.
- Trial 04Missed 8 of 15 checks — cp_gross_dollars_moved, cp_net_exposure_usd and cp_outstanding_overdebit_usd and 5 more.
- Trial 05Missed 7 of 15 checks — cp_gross_dollars_moved, cp_net_exposure_usd and cp_outstanding_overdebit_usd and 4 more.
- Trial 06Missed 8 of 15 checks — cp_gross_dollars_moved, cp_net_exposure_usd and cp_outstanding_overdebit_usd and 5 more.
- Trial 07Missed 8 of 15 checks — cp_gross_dollars_moved, cp_net_exposure_usd and cp_outstanding_overdebit_usd and 5 more.
- Trial 08Missed 7 of 15 checks — cp_gross_dollars_moved, cp_net_exposure_usd and cp_outstanding_overdebit_usd and 4 more.
- Trial 09Missed 14 of 15 checks — cp_fully_corrected_count, cp_gross_dollars_moved and cp_net_exposure_usd and 11 more.
- Trial 10Missed 13 of 15 checks — cp_fully_corrected_count, cp_gross_dollars_moved and cp_net_exposure_usd and 10 more.
Claude Opus 4.8
Mean reward
65.3%All trials
Across 10 trials on this task, reward ran from 40% to 93.3% — 1 of 10 cleared every check.
- 9.8 / 15avg checks passed
- 32avg turns
- 42avg tool calls
- 6mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 1 of 10 trials. Failures cluster on cohort_and_remediation (10/10 trials).
Common pattern
cohort_and_remediation failed in every unsuccessful trial; what else broke varies across the other 8 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 8 of 15 checks — cp_fully_corrected_count, cp_gross_dollars_moved and cp_net_exposure_usd and 5 more.
- Trial 02Missed 6 of 15 checks — cp_gross_dollars_moved, cp_net_exposure_usd and cp_outstanding_overdebit_usd and 3 more.
- Trial 03Missed 8 of 15 checks — cp_gross_dollars_moved, cp_net_exposure_usd and cp_outstanding_overdebit_usd and 5 more.
- Trial 04Missed 2 of 15 checks — reconstruction_method and cohort_and_remediation.
- Trial 05Missed 2 of 15 checks — reconstruction_method and cohort_and_remediation.
- Trial 06Missed 9 of 15 checks — cp_fully_corrected_count, cp_gross_dollars_moved and cp_net_exposure_usd and 6 more.
- Trial 07Missed 8 of 15 checks — cp_gross_dollars_moved, cp_net_exposure_usd and cp_outstanding_overdebit_usd and 5 more.
- Trial 08Missed 2 of 15 checks — reconstruction_method and cohort_and_remediation.
- Trial 09Missed 6 of 15 checks — cp_gross_dollars_moved, cp_net_exposure_usd and cp_outstanding_overdebit_usd and 3 more.
- Trial 10Missed 1 of 15 checks — cohort_and_remediation.
Gemini 3.1 Pro
Mean reward
68%All trials
Across 10 trials on this task, reward ran from 6.7% to 86.7% — 0 of 10 cleared every check.
- 10.2 / 15avg checks passed
- 63avg turns
- 61avg tool calls
- 9mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on reconstruction_method (10/10 trials).
Common pattern
reconstruction_method and cohort_and_remediation failed in every unsuccessful trial; what else broke varies across the other 12 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 14 of 15 checks — cp_fully_corrected_count, cp_gross_dollars_moved and cp_net_exposure_usd and 11 more.
- Trial 02Missed 9 of 15 checks — cp_gross_dollars_moved, cp_net_exposure_usd and cp_outstanding_overdebit_usd and 6 more.
- Trial 03Missed 2 of 15 checks — reconstruction_method and cohort_and_remediation.
- Trial 04Missed 2 of 15 checks — reconstruction_method and cohort_and_remediation.
- Trial 05Missed 2 of 15 checks — reconstruction_method and cohort_and_remediation.
- Trial 06Missed 2 of 15 checks — reconstruction_method and cohort_and_remediation.
- Trial 07Missed 8 of 15 checks — cp_gross_dollars_moved, cp_net_exposure_usd and cp_outstanding_overdebit_usd and 5 more.
- Trial 08Missed 3 of 15 checks — reconstruction_method, cohort_and_remediation and posted_deliverable_to_slack.
- Trial 09Missed 3 of 15 checks — reconstruction_method, cohort_and_remediation and root_cause_and_control.
- Trial 10Missed 3 of 15 checks — reconstruction_method, cohort_and_remediation and posted_deliverable_to_slack.
Gemini 3.5 Flash
Mean reward
2%All trials
Across 10 trials on this task, reward ran from 0% to 6.7% — 0 of 10 cleared every check.
- 0.3 / 15avg checks passed
- 39avg turns
- 49avg tool calls
- 9mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on cp_fully_corrected_count (10/10 trials).
Common pattern
cp_fully_corrected_count, cp_gross_dollars_moved and cp_net_exposure_usd and 10 more failed in every unsuccessful trial; what else broke varies across the other 2 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Missing field — 6 verifiers across 6 of 10 trials.
Per-trial notes
- Trial 01Missed 15 of 15 checks — cp_fully_corrected_count, cp_gross_dollars_moved and cp_net_exposure_usd and 12 more.
- Trial 02Missed 14 of 15 checks — cp_fully_corrected_count, cp_gross_dollars_moved and cp_net_exposure_usd and 11 more.
- Trial 03Missed 15 of 15 checks — cp_fully_corrected_count, cp_gross_dollars_moved and cp_net_exposure_usd and 12 more.
- Trial 04Missed 15 of 15 checks — cp_fully_corrected_count, cp_gross_dollars_moved and cp_net_exposure_usd and 12 more.
- Trial 05Missed 15 of 15 checks — cp_fully_corrected_count, cp_gross_dollars_moved and cp_net_exposure_usd and 12 more.
- Trial 06Missed 14 of 15 checks — cp_fully_corrected_count, cp_gross_dollars_moved and cp_net_exposure_usd and 11 more.
- Trial 07Missed 15 of 15 checks — cp_fully_corrected_count, cp_gross_dollars_moved and cp_net_exposure_usd and 12 more.
- Trial 08Missed 15 of 15 checks — cp_fully_corrected_count, cp_gross_dollars_moved and cp_net_exposure_usd and 12 more.
- Trial 09Missed 15 of 15 checks — cp_fully_corrected_count, cp_gross_dollars_moved and cp_net_exposure_usd and 12 more.
- Trial 10Missed 14 of 15 checks — cp_fully_corrected_count, cp_gross_dollars_moved and cp_net_exposure_usd and 11 more.