Monthly snapshot ledger continuity auditReconciliation & Settlement
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
91.7%All trials
Across 10 trials on this task, reward ran from 58.3% to 100% — 8 of 10 cleared every check.
- 11.0 / 12avg checks passed
- 16avg turns
- 18avg tool calls
- 3mavg time
Summary · GPT-5.5
GPT-5.5 passed 8 of 10 trials. Failures cluster on answer_cluster_2023_08_count (2/10 trials).
Common pattern
answer_cluster_2023_08_count, answer_continuity_break_counts and answer_rollover_population_and_incomplete_count and 1 more failed in every unsuccessful trial; what else broke varies across the other 2 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 2 verifiers across 2 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Passed every verifier it was graded on.
- Trial 03Missed 5 of 12 checks.
- Trial 04Passed every verifier it was graded on.
- Trial 05Missed 5 of 12 checks.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Passed every verifier it was graded on.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Claude Opus 4.8
Mean reward
77.5%All trials
Across 10 trials on this task, reward ran from 41.7% to 100% — 4 of 10 cleared every check.
- 9.3 / 12avg checks passed
- 14avg turns
- 15avg tool calls
- 2mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 4 of 10 trials. Failures cluster on answer_cluster_2023_08_count (6/10 trials).
Common pattern
answer_cluster_2023_08_count, answer_continuity_break_counts and answer_rollover_population_and_incomplete_count and 1 more failed in every unsuccessful trial; what else broke varies across the other 3 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 6 verifiers across 6 of 10 trials.
Per-trial notes
- Trial 01Missed 4 of 12 checks.
- Trial 02Missed 4 of 12 checks.
- Trial 03Missed 4 of 12 checks.
- Trial 04Passed every verifier it was graded on.
- Trial 05Missed 4 of 12 checks.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Missed 4 of 12 checks.
- Trial 09Passed every verifier it was graded on.
- Trial 10Missed 7 of 12 checks.
Gemini 3.1 Pro
Mean reward
55%All trials
Across 10 trials on this task, reward ran from 50% to 66.7% — 0 of 10 cleared every check.
- 6.6 / 12avg checks passed
- 19avg turns
- 17avg tool calls
- 3mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on answer_cluster_2023_08_count (10/10 trials).
Common pattern
answer_cluster_2023_08_count, answer_continuity_break_counts and answer_rollover_population_and_incomplete_count and 1 more failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 5 of 12 checks.
- Trial 02Missed 6 of 12 checks.
- Trial 03Missed 6 of 12 checks.
- Trial 04Missed 4 of 12 checks.
- Trial 05Missed 6 of 12 checks.
- Trial 06Missed 4 of 12 checks.
- Trial 07Missed 6 of 12 checks.
- Trial 08Missed 6 of 12 checks.
- Trial 09Missed 6 of 12 checks.
- Trial 10Missed 5 of 12 checks.
Gemini 3.5 Flash
Mean reward
46.7%All trials
Across 10 trials on this task, reward ran from 16.7% to 91.7% — 1 of 10 cleared every check.
- 5.6 / 12avg checks passed
- 78avg turns
- 89avg tool calls
- 54mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed 1 of 10 trials. Failures cluster on deliverable_slack_message_content (9/10 trials).
Common pattern
deliverable_slack_message_content failed in every unsuccessful trial; what else broke varies across the other 9 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Missed 10 of 12 checks.
- Trial 02Missed 6 of 12 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 3 more.
- Trial 03Missed 10 of 12 checks.
- Trial 04Missed 9 of 12 checks — answer_cluster_2023_08_count, answer_continuity_break_counts and deliverable_jira_ticket_filed and 6 more.
- Trial 05Missed 1 of 12 checks — trace_break_classification_and_scope.
- Trial 06Missed 5 of 12 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 2 more.
- Trial 07Missed 2 of 12 checks — answer_rollover_population_and_incomplete_count and deliverable_slack_message_content.
- Trial 08Missed 5 of 12 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 2 more.
- Trial 09Missed 10 of 12 checks.
- Trial 10Missed 6 of 12 checks — answer_cluster_2023_08_count, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 3 more.