Info-change review-control conformanceCompliance & Regulatory
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
75%All trials
Across 10 trials on this task, reward ran from 58.3% to 91.7% — 0 of 10 cleared every check.
- 9.0 / 12avg checks passed
- 23avg turns
- 28avg tool calls
- 3mavg time
Summary · GPT-5.5
GPT-5.5 passed none of its 10 trials. Failures cluster on answer_out_of_band_exception_count (8/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 5 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 12 checks — answer_out_of_band_exception_count and trace_all_connectors_and_trap.
- Trial 02Missed 4 of 12 checks — answer_out_of_band_exception_count, answer_state_machine_counts and trace_all_connectors_and_trap and 1 more.
- Trial 03Missed 5 of 12 checks — answer_out_of_band_exception_count, answer_state_machine_counts and deliverable_slack_message_content and 2 more.
- Trial 04Missed 2 of 12 checks — answer_out_of_band_exception_count and answer_state_machine_counts.
- Trial 05Missed 3 of 12 checks — answer_out_of_band_exception_count, answer_state_machine_counts and deliverable_slack_message_content.
- Trial 06Missed 5 of 12 checks — answer_out_of_band_exception_count, answer_state_machine_counts and deliverable_slack_message_content and 2 more.
- Trial 07Missed 4 of 12 checks — answer_out_of_band_exception_count, answer_state_machine_counts and trace_all_connectors_and_trap and 1 more.
- Trial 08Missed 1 of 12 checks — answer_state_machine_counts.
- Trial 09Missed 3 of 12 checks — answer_out_of_band_exception_count, trace_all_connectors_and_trap and trace_containment_dedup_method.
- Trial 10Missed 1 of 12 checks — answer_state_machine_counts.
Claude Opus 4.8
Mean reward
75.8%All trials
Across 10 trials on this task, reward ran from 75% to 83.3% — 0 of 10 cleared every check.
- 9.1 / 12avg checks passed
- 16avg turns
- 23avg tool calls
- 2mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed none of its 10 trials. Failures cluster on answer_out_of_band_exception_count (10/10 trials).
Common pattern
answer_out_of_band_exception_count and trace_all_connectors_and_trap failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 12 checks — answer_out_of_band_exception_count and trace_all_connectors_and_trap.
- Trial 02Missed 3 of 12 checks — answer_out_of_band_exception_count, trace_all_connectors_and_trap and trace_containment_dedup_method.
- Trial 03Missed 3 of 12 checks — answer_out_of_band_exception_count, trace_all_connectors_and_trap and trace_containment_dedup_method.
- Trial 04Missed 3 of 12 checks — answer_out_of_band_exception_count, trace_all_connectors_and_trap and trace_containment_dedup_method.
- Trial 05Missed 3 of 12 checks — answer_out_of_band_exception_count, trace_all_connectors_and_trap and trace_containment_dedup_method.
- Trial 06Missed 3 of 12 checks — answer_out_of_band_exception_count, trace_all_connectors_and_trap and trace_containment_dedup_method.
- Trial 07Missed 3 of 12 checks — answer_out_of_band_exception_count, trace_all_connectors_and_trap and trace_containment_dedup_method.
- Trial 08Missed 3 of 12 checks — answer_out_of_band_exception_count, trace_all_connectors_and_trap and trace_containment_dedup_method.
- Trial 09Missed 3 of 12 checks — answer_out_of_band_exception_count, trace_all_connectors_and_trap and trace_containment_dedup_method.
- Trial 10Missed 3 of 12 checks — answer_out_of_band_exception_count, trace_all_connectors_and_trap and trace_containment_dedup_method.
Gemini 3.1 Pro
Mean reward
75%All trials
Across 10 trials on this task, reward ran from 66.7% to 91.7% — 0 of 10 cleared every check.
- 9.0 / 12avg checks passed
- 34avg turns
- 34avg tool calls
- 6mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on answer_out_of_band_exception_count (8/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 4 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 12 checks — deliverable_jira_ticket_filed.
- Trial 02Missed 4 of 12 checks — answer_out_of_band_exception_count, deliverable_jira_ticket_filed and trace_all_connectors_and_trap and 1 more.
- Trial 03Missed 3 of 12 checks — answer_out_of_band_exception_count, trace_all_connectors_and_trap and trace_containment_dedup_method.
- Trial 04Missed 3 of 12 checks — answer_out_of_band_exception_count, trace_all_connectors_and_trap and trace_containment_dedup_method.
- Trial 05Missed 4 of 12 checks — answer_out_of_band_exception_count, deliverable_jira_ticket_filed and trace_all_connectors_and_trap and 1 more.
- Trial 06Missed 4 of 12 checks — answer_out_of_band_exception_count, deliverable_jira_ticket_filed and trace_all_connectors_and_trap and 1 more.
- Trial 07Missed 3 of 12 checks — answer_out_of_band_exception_count, trace_all_connectors_and_trap and trace_containment_dedup_method.
- Trial 08Missed 4 of 12 checks — answer_out_of_band_exception_count, deliverable_jira_ticket_filed and trace_all_connectors_and_trap and 1 more.
- Trial 09Missed 1 of 12 checks — deliverable_jira_ticket_filed.
- Trial 10Missed 3 of 12 checks — answer_out_of_band_exception_count, trace_all_connectors_and_trap and trace_containment_dedup_method.
Gemini 3.5 Flash
Mean reward
42.5%All trials
Across 10 trials on this task, reward ran from 25% to 100% — 1 of 10 cleared every check.
- 5.1 / 12avg checks passed
- 49avg turns
- 46avg tool calls
- 7mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed 1 of 10 trials. Failures cluster on answer_out_of_band_exception_count (9/10 trials).
Common pattern
answer_out_of_band_exception_count, trace_all_connectors_and_trap and trace_containment_dedup_method failed in every unsuccessful trial; what else broke varies across the other 6 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 9 of 12 checks — answer_out_of_band_exception_count, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 6 more.
- Trial 02Missed 3 of 12 checks — answer_out_of_band_exception_count, trace_all_connectors_and_trap and trace_containment_dedup_method.
- Trial 03Missed 3 of 12 checks — answer_out_of_band_exception_count, trace_all_connectors_and_trap and trace_containment_dedup_method.
- Trial 04Passed every verifier it was graded on.
- Trial 05Missed 9 of 12 checks — answer_out_of_band_exception_count, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 6 more.
- Trial 06Missed 9 of 12 checks — answer_out_of_band_exception_count, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 6 more.
- Trial 07Missed 9 of 12 checks — answer_out_of_band_exception_count, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 6 more.
- Trial 08Missed 9 of 12 checks — answer_out_of_band_exception_count, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 6 more.
- Trial 09Missed 9 of 12 checks — answer_out_of_band_exception_count, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 6 more.
- Trial 10Missed 9 of 12 checks — answer_out_of_band_exception_count, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 6 more.