Alloy would-approve flag vs human decisionsCompliance & Regulatory
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
93.8%All trials
Across 10 trials on this task, reward ran from 76.9% to 100% — 9 of 10 cleared every check.
- 12.2 / 13avg checks passed
- 16avg turns
- 22avg tool calls
- 3mavg time
Summary · GPT-5.5
GPT-5.5 passed 9 of 10 trials. Failures cluster on trace_alloy_vs_human_method (6/10 trials).
Common pattern
All 1 unsuccessful trials broke on exactly the same verifiers — answer_flagged_override_count, deliverable_slack_message_content and trace_alloy_vs_human_method failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 1 verifier across 1 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Missed 3 of 13 checks — answer_flagged_override_count, deliverable_slack_message_content and trace_alloy_vs_human_method.
- Trial 03Missed 1 of 13 checks — trace_alloy_vs_human_method.
- Trial 04Missed 1 of 13 checks — trace_alloy_vs_human_method.
- Trial 05Passed every verifier it was graded on.
- Trial 06Missed 1 of 13 checks — trace_alloy_vs_human_method.
- Trial 07Passed every verifier it was graded on.
- Trial 08Missed 1 of 13 checks — trace_alloy_vs_human_method.
- Trial 09Passed every verifier it was graded on.
- Trial 10Missed 1 of 13 checks — trace_alloy_vs_human_method.
Claude Opus 4.8
Mean reward
85.4%All trials
Across 10 trials on this task, reward ran from 61.5% to 100% — 6 of 10 cleared every check.
- 11.1 / 13avg checks passed
- 14avg turns
- 21avg tool calls
- 2mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 6 of 10 trials. Failures cluster on answer_flagged_override_count (4/10 trials).
Common pattern
answer_flagged_override_count and deliverable_slack_message_content failed in every unsuccessful trial; what else broke varies across the other 6 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 4 verifiers across 4 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Passed every verifier it was graded on.
- Trial 03Passed every verifier it was graded on.
- Trial 04Passed every verifier it was graded on.
- Trial 05Missed 5 of 13 checks — answer_flagged_approval_rate_pct, answer_flagged_override_count and answer_nonflagged_approval_rate_pct and 2 more.
- Trial 06Missed 3 of 13 checks — answer_flagged_override_count, deliverable_slack_message_content and trace_alloy_vs_human_method.
- Trial 07Missed 5 of 13 checks — answer_flagged_override_count, deliverable_slack_message_content and floor_all_connectors and 2 more.
- Trial 08Missed 1 of 13 checks — trace_alloy_vs_human_method.
- Trial 09Passed every verifier it was graded on.
- Trial 10Missed 5 of 13 checks — answer_flagged_approval_rate_pct, answer_flagged_override_count and answer_nonflagged_approval_rate_pct and 2 more.
Gemini 3.1 Pro
Mean reward
56.9%All trials
Across 10 trials on this task, reward ran from 30.8% to 84.6% — 2 of 10 cleared every check.
- 7.4 / 13avg checks passed
- 27avg turns
- 26avg tool calls
- 4mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 2 of 10 trials. Failures cluster on floor_all_connectors (9/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 9 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Missed 7 of 13 checks — answer_nonflagged_approval_rate_pct, deliverable_jira_ticket_filed and deliverable_slack_message_content and 4 more.
- Trial 02Missed 5 of 13 checks — answer_nonflagged_approval_rate_pct, deliverable_slack_message_content and floor_all_connectors and 2 more.
- Trial 03Missed 7 of 13 checks — answer_flagged_override_count, deliverable_jira_ticket_filed and deliverable_slack_message_content and 4 more.
- Trial 04Missed 7 of 13 checks — answer_flagged_override_count, deliverable_jira_ticket_filed and deliverable_slack_message_content and 4 more.
- Trial 05Missed 6 of 13 checks — answer_nonflagged_approval_rate_pct, deliverable_slack_message_content and floor_all_connectors and 3 more.
- Trial 06Missed 4 of 13 checks — floor_all_connectors, floor_read_source_ticket and trace_all_connectors_and_trap and 1 more.
- Trial 07Missed 5 of 13 checks — deliverable_jira_ticket_filed, floor_all_connectors and floor_read_source_ticket and 2 more.
- Trial 08Missed 4 of 13 checks — floor_all_connectors, floor_read_source_ticket and trace_all_connectors_and_trap and 1 more.
- Trial 09Missed 9 of 13 checks — answer_flagged_approval_rate_pct, answer_flagged_override_count and answer_nonflagged_approval_rate_pct and 6 more.
- Trial 10Missed 2 of 13 checks — deliverable_jira_ticket_filed and trace_alloy_vs_human_method.
Gemini 3.5 Flash
Mean reward
15.4%All trials
Across 10 trials on this task, reward ran from 15.4% to 15.4% — 0 of 10 cleared every check.
- 2.0 / 13avg checks passed
- 57avg turns
- 55avg tool calls
- 24mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on answer_flagged_approval_rate_pct (10/10 trials).
Common pattern
All 10 unsuccessful trials broke on exactly the same verifiers — answer_flagged_approval_rate_pct, answer_flagged_override_count and answer_nonflagged_approval_rate_pct and 8 more failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Missing field — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 11 of 13 checks — answer_flagged_approval_rate_pct, answer_flagged_override_count and answer_nonflagged_approval_rate_pct and 8 more.
- Trial 02Missed 11 of 13 checks — answer_flagged_approval_rate_pct, answer_flagged_override_count and answer_nonflagged_approval_rate_pct and 8 more.
- Trial 03Missed 11 of 13 checks — answer_flagged_approval_rate_pct, answer_flagged_override_count and answer_nonflagged_approval_rate_pct and 8 more.
- Trial 04Missed 11 of 13 checks — answer_flagged_approval_rate_pct, answer_flagged_override_count and answer_nonflagged_approval_rate_pct and 8 more.
- Trial 05Missed 11 of 13 checks — answer_flagged_approval_rate_pct, answer_flagged_override_count and answer_nonflagged_approval_rate_pct and 8 more.
- Trial 06Missed 11 of 13 checks — answer_flagged_approval_rate_pct, answer_flagged_override_count and answer_nonflagged_approval_rate_pct and 8 more.
- Trial 07Missed 11 of 13 checks — answer_flagged_approval_rate_pct, answer_flagged_override_count and answer_nonflagged_approval_rate_pct and 8 more.
- Trial 08Missed 11 of 13 checks — answer_flagged_approval_rate_pct, answer_flagged_override_count and answer_nonflagged_approval_rate_pct and 8 more.
- Trial 09Missed 11 of 13 checks — answer_flagged_approval_rate_pct, answer_flagged_override_count and answer_nonflagged_approval_rate_pct and 8 more.
- Trial 10Missed 11 of 13 checks — answer_flagged_approval_rate_pct, answer_flagged_override_count and answer_nonflagged_approval_rate_pct and 8 more.