Audit card authorizations after account lock timestampRisk & Fraud
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
88.2%All trials
Across 10 trials on this task, reward ran from 36.4% to 100% — 4 of 10 cleared every check.
- 9.7 / 11avg checks passed
- 63avg turns
- 294avg tool calls
- 16mavg time
Summary · GPT-5.5
GPT-5.5 passed 4 of 10 trials. Failures cluster on rubric_uses_all_connectors (4/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 10 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 4 verifiers across 4 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 11 checks — rubric_uses_all_connectors.
- Trial 02Missed 1 of 11 checks — rubric_uses_all_connectors.
- Trial 03Missed 1 of 11 checks — rubric_uses_all_connectors.
- Trial 04Missed 7 of 11 checks — gold_leaked_dollars, gold_offending_accounts and gold_offending_auths and 4 more.
- Trial 05Passed every verifier it was graded on.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Passed every verifier it was graded on.
- Trial 09Missed 2 of 11 checks — rubric_root_cause and rubric_uses_all_connectors.
- Trial 10Missed 1 of 11 checks — scopes_lock_leak_correctly.
Claude Opus 4.8
Mean reward
80.9%All trials
Across 10 trials on this task, reward ran from 36.4% to 100% — 3 of 10 cleared every check.
- 8.9 / 11avg checks passed
- 42avg turns
- 135avg tool calls
- 9mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 3 of 10 trials. Failures cluster on rubric_root_cause (4/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 8 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 11 checks — rubric_root_cause and scopes_lock_leak_correctly.
- Trial 02Passed every verifier it was graded on.
- Trial 03Missed 2 of 11 checks — scopes_lock_leak_correctly and gold_cards_never_propagated.
- Trial 04Missed 7 of 11 checks — gold_leaked_dollars, gold_offending_accounts and gold_offending_auths and 4 more.
- Trial 05Missed 6 of 11 checks — gold_leaked_dollars, gold_offending_accounts and gold_offending_auths and 3 more.
- Trial 06Missed 1 of 11 checks — rubric_root_cause.
- Trial 07Missed 1 of 11 checks — gold_cards_never_propagated.
- Trial 08Missed 1 of 11 checks — rubric_root_cause.
- Trial 09Passed every verifier it was graded on.
- Trial 10Missed 1 of 11 checks — rubric_root_cause.
Gemini 3.1 Pro
Mean reward
48.2%All trials
Across 10 trials on this task, reward ran from 27.3% to 81.8% — 0 of 10 cleared every check.
- 5.3 / 11avg checks passed
- 184avg turns
- 184avg tool calls
- 28mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on rubric_uses_all_connectors (9/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 8 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Exhausted iterations — 6 verifiers across 6 of 10 trials.
Per-trial notes
- Trial 01Missed 8 of 11 checks — gold_leaked_dollars, gold_offending_accounts and gold_offending_auths and 5 more.
- Trial 02Missed 7 of 11 checks — gold_leaked_dollars, gold_offending_accounts and gold_offending_auths and 4 more.
- Trial 03Missed 7 of 11 checks — gold_leaked_dollars, gold_offending_accounts and gold_offending_auths and 4 more.
- Trial 04Missed 2 of 11 checks — rubric_root_cause and rubric_uses_all_connectors.
- Trial 05Missed 5 of 11 checks — gold_leaked_dollars, gold_offending_accounts and gold_offending_auths and 2 more.
- Trial 06Missed 8 of 11 checks — gold_leaked_dollars, gold_offending_accounts and gold_offending_auths and 5 more.
- Trial 07Missed 2 of 11 checks — rubric_root_cause and rubric_uses_all_connectors.
- Trial 08Missed 8 of 11 checks — gold_leaked_dollars, gold_offending_accounts and gold_offending_auths and 5 more.
- Trial 09Missed 2 of 11 checks — rubric_root_cause and rubric_uses_all_connectors.
- Trial 10Missed 8 of 11 checks — gold_leaked_dollars, gold_offending_accounts and gold_offending_auths and 5 more.
Gemini 3.5 Flash
Mean reward
53.6%All trials
Across 10 trials on this task, reward ran from 9.1% to 100% — 1 of 10 cleared every check.
- 5.9 / 11avg checks passed
- 45avg turns
- 42avg tool calls
- 9mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed 1 of 10 trials. Failures cluster on rubric_root_cause (9/10 trials).
Common pattern
rubric_root_cause failed in every unsuccessful trial; what else broke varies across the other 9 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 6 verifiers across 6 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 11 checks — rubric_root_cause and rubric_uses_all_connectors.
- Trial 02Missed 9 of 11 checks — gold_leaked_dollars, gold_offending_accounts and gold_offending_auths and 6 more.
- Trial 03Missed 10 of 11 checks — gold_leaked_dollars, gold_offending_accounts and gold_offending_auths and 7 more.
- Trial 04Missed 2 of 11 checks — rubric_root_cause and rubric_uses_all_connectors.
- Trial 05Passed every verifier it was graded on.
- Trial 06Missed 5 of 11 checks — gold_leaked_dollars, gold_worst_account_dollars and rubric_root_cause and 2 more.
- Trial 07Missed 8 of 11 checks — gold_leaked_dollars, gold_offending_accounts and gold_offending_auths and 5 more.
- Trial 08Missed 3 of 11 checks — rubric_root_cause, rubric_uses_all_connectors and gold_cards_never_propagated.
- Trial 09Missed 3 of 11 checks — rubric_root_cause, rubric_uses_all_connectors and gold_cards_never_propagated.
- Trial 10Missed 9 of 11 checks — gold_leaked_dollars, gold_offending_accounts and gold_offending_auths and 6 more.