Duplicate-funding R01 lock cohort auditRisk & Fraud
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
95.3%All trials
Across 10 trials on this task, reward ran from 80% to 100% — 7 of 10 cleared every check.
- 14.3 / 15avg checks passed
- 11avg turns
- 12avg tool calls
- 2mavg time
Summary · GPT-5.5
GPT-5.5 passed 7 of 10 trials. Failures cluster on cp_bug_lockout_count (2/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 4 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 3 verifiers across 3 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Missed 3 of 15 checks — cp_bug_lockout_count, cp_bug_lockout_exposure_usd and cp_bug_lockout_account_ids.
- Trial 03Passed every verifier it was graded on.
- Trial 04Missed 1 of 15 checks — reconstructs_double_funding_lock_cohort.
- Trial 05Missed 3 of 15 checks — cp_bug_lockout_count, cp_bug_lockout_exposure_usd and cp_bug_lockout_account_ids.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Passed every verifier it was graded on.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Claude Opus 4.8
Mean reward
98.7%All trials
Across 10 trials on this task, reward ran from 93.3% to 100% — 8 of 10 cleared every check.
- 14.8 / 15avg checks passed
- 11avg turns
- 15avg tool calls
- 2mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 8 of 10 trials. Failures cluster on reconstructs_double_funding_lock_cohort (2/10 trials).
Common pattern
All 2 unsuccessful trials broke on exactly the same verifiers — reconstructs_double_funding_lock_cohort failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 2 verifiers across 2 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 15 checks — reconstructs_double_funding_lock_cohort.
- Trial 02Passed every verifier it was graded on.
- Trial 03Passed every verifier it was graded on.
- Trial 04Passed every verifier it was graded on.
- Trial 05Missed 1 of 15 checks — reconstructs_double_funding_lock_cohort.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Passed every verifier it was graded on.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Gemini 3.1 Pro
Mean reward
94.7%All trials
Across 10 trials on this task, reward ran from 93.3% to 100% — 2 of 10 cleared every check.
- 14.2 / 15avg checks passed
- 26avg turns
- 25avg tool calls
- 3mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 2 of 10 trials. Failures cluster on reconstructs_double_funding_lock_cohort (8/10 trials).
Common pattern
All 8 unsuccessful trials broke on exactly the same verifiers — reconstructs_double_funding_lock_cohort failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 15 checks — reconstructs_double_funding_lock_cohort.
- Trial 02Missed 1 of 15 checks — reconstructs_double_funding_lock_cohort.
- Trial 03Passed every verifier it was graded on.
- Trial 04Missed 1 of 15 checks — reconstructs_double_funding_lock_cohort.
- Trial 05Missed 1 of 15 checks — reconstructs_double_funding_lock_cohort.
- Trial 06Missed 1 of 15 checks — reconstructs_double_funding_lock_cohort.
- Trial 07Passed every verifier it was graded on.
- Trial 08Missed 1 of 15 checks — reconstructs_double_funding_lock_cohort.
- Trial 09Missed 1 of 15 checks — reconstructs_double_funding_lock_cohort.
- Trial 10Missed 1 of 15 checks — reconstructs_double_funding_lock_cohort.
Gemini 3.5 Flash
Mean reward
90.7%All trials
Across 10 trials on this task, reward ran from 60% to 100% — 6 of 10 cleared every check.
- 13.6 / 15avg checks passed
- 40avg turns
- 38avg tool calls
- 5mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed 6 of 10 trials. Failures cluster on reconstructs_double_funding_lock_cohort (4/10 trials).
Common pattern
reconstructs_double_funding_lock_cohort failed in every unsuccessful trial; what else broke varies across the other 5 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 4 verifiers across 4 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Missed 1 of 15 checks — reconstructs_double_funding_lock_cohort.
- Trial 03Passed every verifier it was graded on.
- Trial 04Passed every verifier it was graded on.
- Trial 05Passed every verifier it was graded on.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Missed 6 of 15 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 3 more.
- Trial 09Missed 1 of 15 checks — reconstructs_double_funding_lock_cohort.
- Trial 10Missed 6 of 15 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 3 more.