Recurring-bill double-pay audit after due-date editsSubscriptions & Billing
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
95%All trials
Across 10 trials on this task, reward ran from 91.7% to 100% — 4 of 10 cleared every check.
- 11.4 / 12avg checks passed
- 16avg turns
- 25avg tool calls
- 2mavg time
Summary · GPT-5.5
GPT-5.5 passed 4 of 10 trials. Failures cluster on answer_edit_linked_pairs (6/10 trials).
Common pattern
All 6 unsuccessful trials broke on exactly the same verifiers — answer_edit_linked_pairs failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 6 verifiers across 6 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Passed every verifier it was graded on.
- Trial 03Passed every verifier it was graded on.
- Trial 04Missed 1 of 12 checks — answer_edit_linked_pairs.
- Trial 05Missed 1 of 12 checks — answer_edit_linked_pairs.
- Trial 06Missed 1 of 12 checks — answer_edit_linked_pairs.
- Trial 07Missed 1 of 12 checks — answer_edit_linked_pairs.
- Trial 08Missed 1 of 12 checks — answer_edit_linked_pairs.
- Trial 09Passed every verifier it was graded on.
- Trial 10Missed 1 of 12 checks — answer_edit_linked_pairs.
Claude Opus 4.8
Mean reward
93.3%All trials
Across 10 trials on this task, reward ran from 83.3% to 100% — 8 of 10 cleared every check.
- 11.2 / 12avg checks passed
- 12avg turns
- 16avg tool calls
- 1mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 8 of 10 trials. Failures cluster on deliverable_jira_ticket_filed (2/10 trials).
Common pattern
All 2 unsuccessful trials broke on exactly the same verifiers — deliverable_jira_ticket_filed and floor_write_back failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 2 verifiers across 2 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Missed 2 of 12 checks — floor_all_connectors and floor_read_source_confluence.
- Trial 03Missed 2 of 12 checks — floor_all_connectors and floor_read_source_confluence.
- Trial 04Missed 2 of 12 checks — deliverable_jira_ticket_filed and floor_write_back.
- Trial 05Passed every verifier it was graded on.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Passed every verifier it was graded on.
- Trial 09Missed 2 of 12 checks — deliverable_jira_ticket_filed and floor_write_back.
- Trial 10Passed every verifier it was graded on.
Gemini 3.1 Pro
Mean reward
99.2%All trials
Across 10 trials on this task, reward ran from 91.7% to 100% — 9 of 10 cleared every check.
- 11.9 / 12avg checks passed
- 24avg turns
- 22avg tool calls
- 2mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 9 of 10 trials. Failures cluster on deliverable_jira_ticket_filed (1/10 trials).
Common pattern
All 1 unsuccessful trials broke on exactly the same verifiers — deliverable_jira_ticket_filed failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 1 verifier across 1 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Passed every verifier it was graded on.
- Trial 03Passed every verifier it was graded on.
- Trial 04Missed 1 of 12 checks — deliverable_jira_ticket_filed.
- Trial 05Passed every verifier it was graded on.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Passed every verifier it was graded on.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Gemini 3.5 Flash
Mean reward
50%All trials
Across 10 trials on this task, reward ran from 16.7% to 100% — 1 of 10 cleared every check.
- 6.0 / 12avg checks passed
- 29avg turns
- 28avg tool calls
- 18mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed 1 of 10 trials. Failures cluster on deliverable_jira_ticket_filed (9/10 trials).
Common pattern
deliverable_jira_ticket_filed and floor_write_back failed in every unsuccessful trial; what else broke varies across the other 8 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 6 of 12 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 3 more.
- Trial 02Missed 6 of 12 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 3 more.
- Trial 03Missed 10 of 12 checks — answer_collision_population, answer_dup_exposure_usd and answer_edit_linked_pairs and 7 more.
- Trial 04Passed every verifier it was graded on.
- Trial 05Missed 6 of 12 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 3 more.
- Trial 06Missed 9 of 12 checks — answer_collision_population, answer_dup_exposure_usd and answer_edit_linked_pairs and 6 more.
- Trial 07Missed 6 of 12 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 3 more.
- Trial 08Missed 9 of 12 checks — answer_collision_population, answer_dup_exposure_usd and answer_edit_linked_pairs and 6 more.
- Trial 09Missed 2 of 12 checks — deliverable_jira_ticket_filed and floor_write_back.
- Trial 10Missed 6 of 12 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 3 more.