Additional-owner OwnerSync write-back coverage auditCustomer & Account Analytics
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
88.3%All trials
Across 10 trials on this task, reward ran from 58.3% to 91.7% — 9 of 10 cleared every check.
- 10.6 / 12avg checks passed
- 26avg turns
- 39avg tool calls
- 3mavg time
Summary · GPT-5.5
GPT-5.5 passed 9 of 10 trials. Failures cluster on floor_read_source_confluence (10/10 trials).
Common pattern
All 1 unsuccessful trials broke on exactly the same verifiers — answer_approved_and_kyc_coverage, deliverable_slack_message_content and floor_read_source_confluence and 2 more failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 1 verifier across 1 of 10 trials.
Per-trial notes
- Trial 01Missed 5 of 12 checks — answer_approved_and_kyc_coverage, deliverable_slack_message_content and floor_read_source_confluence and 2 more.
- Trial 02Missed 1 of 12 checks — floor_read_source_confluence.
- Trial 03Missed 1 of 12 checks — floor_read_source_confluence.
- Trial 04Missed 1 of 12 checks — floor_read_source_confluence.
- Trial 05Missed 1 of 12 checks — floor_read_source_confluence.
- Trial 06Missed 1 of 12 checks — floor_read_source_confluence.
- Trial 07Missed 1 of 12 checks — floor_read_source_confluence.
- Trial 08Missed 1 of 12 checks — floor_read_source_confluence.
- Trial 09Missed 1 of 12 checks — floor_read_source_confluence.
- Trial 10Missed 1 of 12 checks — floor_read_source_confluence.
Claude Opus 4.8
Mean reward
65.8%All trials
Across 10 trials on this task, reward ran from 58.3% to 91.7% — 2 of 10 cleared every check.
- 7.9 / 12avg checks passed
- 16avg turns
- 22avg tool calls
- 3mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 2 of 10 trials. Failures cluster on floor_read_source_confluence (9/10 trials).
Common pattern
answer_approved_and_kyc_coverage, trace_all_connectors_and_trap and trace_ownersync_coverage_method failed in every unsuccessful trial; what else broke varies across the other 2 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Missed 4 of 12 checks — answer_approved_and_kyc_coverage, floor_read_source_confluence and trace_all_connectors_and_trap and 1 more.
- Trial 02Missed 5 of 12 checks — answer_approved_and_kyc_coverage, deliverable_slack_message_content and floor_read_source_confluence and 2 more.
- Trial 03Missed 1 of 12 checks — floor_read_source_confluence.
- Trial 04Missed 5 of 12 checks — answer_approved_and_kyc_coverage, deliverable_slack_message_content and floor_read_source_confluence and 2 more.
- Trial 05Missed 5 of 12 checks — answer_approved_and_kyc_coverage, deliverable_slack_message_content and floor_read_source_confluence and 2 more.
- Trial 06Missed 5 of 12 checks — answer_approved_and_kyc_coverage, deliverable_slack_message_content and floor_read_source_confluence and 2 more.
- Trial 07Missed 4 of 12 checks — answer_approved_and_kyc_coverage, deliverable_slack_message_content and trace_all_connectors_and_trap and 1 more.
- Trial 08Missed 2 of 12 checks — floor_read_source_confluence and trace_ownersync_coverage_method.
- Trial 09Missed 5 of 12 checks — answer_approved_and_kyc_coverage, deliverable_slack_message_content and floor_read_source_confluence and 2 more.
- Trial 10Missed 5 of 12 checks — answer_approved_and_kyc_coverage, deliverable_slack_message_content and floor_read_source_confluence and 2 more.
Gemini 3.1 Pro
Mean reward
50.8%All trials
Across 10 trials on this task, reward ran from 16.7% to 75% — 0 of 10 cleared every check.
- 6.1 / 12avg checks passed
- 49avg turns
- 59avg tool calls
- 12mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on floor_read_source_confluence (10/10 trials).
Common pattern
floor_read_source_confluence and trace_ownersync_coverage_method failed in every unsuccessful trial; what else broke varies across the other 8 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 6 of 12 checks — answer_approved_and_kyc_coverage, deliverable_jira_ticket_filed and deliverable_slack_message_content and 3 more.
- Trial 02Missed 5 of 12 checks — answer_approved_and_kyc_coverage, deliverable_slack_message_content and floor_read_source_confluence and 2 more.
- Trial 03Missed 6 of 12 checks — answer_approved_and_kyc_coverage, deliverable_jira_ticket_filed and deliverable_slack_message_content and 3 more.
- Trial 04Missed 5 of 12 checks — answer_approved_and_kyc_coverage, deliverable_slack_message_content and floor_read_source_confluence and 2 more.
- Trial 05Missed 3 of 12 checks — deliverable_jira_ticket_filed, floor_read_source_confluence and trace_ownersync_coverage_method.
- Trial 06Missed 6 of 12 checks — answer_approved_and_kyc_coverage, deliverable_jira_ticket_filed and deliverable_slack_message_content and 3 more.
- Trial 07Missed 10 of 12 checks — answer_approved_and_kyc_coverage, answer_new_owner_write_coverage and deliverable_jira_ticket_filed and 7 more.
- Trial 08Missed 6 of 12 checks — answer_approved_and_kyc_coverage, deliverable_jira_ticket_filed and deliverable_slack_message_content and 3 more.
- Trial 09Missed 6 of 12 checks — answer_approved_and_kyc_coverage, deliverable_jira_ticket_filed and deliverable_slack_message_content and 3 more.
- Trial 10Missed 6 of 12 checks — answer_approved_and_kyc_coverage, deliverable_jira_ticket_filed and deliverable_slack_message_content and 3 more.
Gemini 3.5 Flash
Mean reward
25.8%All trials
Across 10 trials on this task, reward ran from 8.3% to 41.7% — 0 of 10 cleared every check.
- 3.1 / 12avg checks passed
- 53avg turns
- 64avg tool calls
- 18mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on deliverable_jira_ticket_filed (10/10 trials).
Common pattern
deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 4 more failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 9 of 12 checks — answer_approved_and_kyc_coverage, answer_new_owner_write_coverage and deliverable_jira_ticket_filed and 6 more.
- Trial 02Missed 7 of 12 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 4 more.
- Trial 03Missed 9 of 12 checks — answer_approved_and_kyc_coverage, answer_new_owner_write_coverage and deliverable_jira_ticket_filed and 6 more.
- Trial 04Missed 9 of 12 checks — answer_approved_and_kyc_coverage, answer_owner_add_apps_total and deliverable_jira_ticket_filed and 6 more.
- Trial 05Missed 10 of 12 checks — answer_approved_and_kyc_coverage, answer_new_owner_write_coverage and deliverable_jira_ticket_filed and 7 more.
- Trial 06Missed 8 of 12 checks — answer_approved_and_kyc_coverage, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 5 more.
- Trial 07Missed 8 of 12 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 5 more.
- Trial 08Missed 11 of 12 checks — answer_approved_and_kyc_coverage, answer_new_owner_write_coverage and answer_owner_add_apps_total and 8 more.
- Trial 09Missed 9 of 12 checks — answer_approved_and_kyc_coverage, answer_new_owner_write_coverage and deliverable_jira_ticket_filed and 6 more.
- Trial 10Missed 9 of 12 checks — answer_approved_and_kyc_coverage, answer_new_owner_write_coverage and deliverable_jira_ticket_filed and 6 more.