Manual-review reapplication repeat offendersRisk & Fraud
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
90.8%All trials
Across 10 trials on this task, reward ran from 61.5% to 100% — 9 of 10 cleared every check.
- 11.8 / 13avg checks passed
- 30avg turns
- 36avg tool calls
- 4mavg time
Summary · GPT-5.5
GPT-5.5 passed 9 of 10 trials. Failures cluster on trace_person_level_cohort_method (8/10 trials).
Common pattern
All 1 unsuccessful trials broke on exactly the same verifiers — answer_repeat_offender_count, answer_cohort_manual_review_apps and answer_total_outstanding_doc_requests and 2 more failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 1 verifier across 1 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 13 checks — trace_person_level_cohort_method.
- Trial 02Passed every verifier it was graded on.
- Trial 03Missed 1 of 13 checks — trace_person_level_cohort_method.
- Trial 04Passed every verifier it was graded on.
- Trial 05Missed 1 of 13 checks — trace_person_level_cohort_method.
- Trial 06Missed 1 of 13 checks — trace_person_level_cohort_method.
- Trial 07Missed 1 of 13 checks — trace_person_level_cohort_method.
- Trial 08Missed 1 of 13 checks — trace_person_level_cohort_method.
- Trial 09Missed 1 of 13 checks — trace_person_level_cohort_method.
- Trial 10Missed 5 of 13 checks — answer_repeat_offender_count, answer_cohort_manual_review_apps and answer_total_outstanding_doc_requests and 2 more.
Claude Opus 4.8
Mean reward
63.1%All trials
Across 10 trials on this task, reward ran from 46.2% to 92.3% — 1 of 10 cleared every check.
- 8.2 / 13avg checks passed
- 14avg turns
- 19avg tool calls
- 2mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 1 of 10 trials. Failures cluster on trace_person_level_cohort_method (10/10 trials).
Common pattern
answer_repeat_offender_count, answer_cohort_manual_review_apps and answer_total_outstanding_doc_requests and 2 more failed in every unsuccessful trial; what else broke varies across the other 2 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 5 of 13 checks — answer_repeat_offender_count, answer_cohort_manual_review_apps and answer_total_outstanding_doc_requests and 2 more.
- Trial 02Missed 5 of 13 checks — answer_repeat_offender_count, answer_cohort_manual_review_apps and answer_total_outstanding_doc_requests and 2 more.
- Trial 03Missed 5 of 13 checks — answer_repeat_offender_count, answer_cohort_manual_review_apps and answer_total_outstanding_doc_requests and 2 more.
- Trial 04Missed 7 of 13 checks — answer_cohort_severity_metrics, answer_repeat_offender_count and deliverable_slack_message_content and 4 more.
- Trial 05Missed 5 of 13 checks — answer_repeat_offender_count, answer_cohort_manual_review_apps and answer_total_outstanding_doc_requests and 2 more.
- Trial 06Missed 5 of 13 checks — answer_repeat_offender_count, answer_cohort_manual_review_apps and answer_total_outstanding_doc_requests and 2 more.
- Trial 07Missed 1 of 13 checks — trace_person_level_cohort_method.
- Trial 08Missed 5 of 13 checks — answer_repeat_offender_count, answer_cohort_manual_review_apps and answer_total_outstanding_doc_requests and 2 more.
- Trial 09Missed 5 of 13 checks — answer_repeat_offender_count, answer_cohort_manual_review_apps and answer_total_outstanding_doc_requests and 2 more.
- Trial 10Missed 5 of 13 checks — answer_repeat_offender_count, answer_cohort_manual_review_apps and answer_total_outstanding_doc_requests and 2 more.
Gemini 3.1 Pro
Mean reward
69.2%All trials
Across 10 trials on this task, reward ran from 53.8% to 92.3% — 2 of 10 cleared every check.
- 9.0 / 13avg checks passed
- 21avg turns
- 20avg tool calls
- 4mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 2 of 10 trials. Failures cluster on trace_person_level_cohort_method (10/10 trials).
Common pattern
trace_person_level_cohort_method failed in every unsuccessful trial; what else broke varies across the other 5 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 13 checks — trace_person_level_cohort_method.
- Trial 02Missed 5 of 13 checks — answer_repeat_offender_count, answer_cohort_manual_review_apps and answer_total_outstanding_doc_requests and 2 more.
- Trial 03Missed 6 of 13 checks — answer_repeat_offender_count, deliverable_jira_ticket_filed and answer_cohort_manual_review_apps and 3 more.
- Trial 04Missed 2 of 13 checks — deliverable_jira_ticket_filed and trace_person_level_cohort_method.
- Trial 05Missed 6 of 13 checks — answer_repeat_offender_count, deliverable_jira_ticket_filed and answer_cohort_manual_review_apps and 3 more.
- Trial 06Missed 6 of 13 checks — answer_repeat_offender_count, deliverable_jira_ticket_filed and answer_cohort_manual_review_apps and 3 more.
- Trial 07Missed 2 of 13 checks — deliverable_jira_ticket_filed and trace_person_level_cohort_method.
- Trial 08Missed 1 of 13 checks — trace_person_level_cohort_method.
- Trial 09Missed 5 of 13 checks — answer_repeat_offender_count, answer_cohort_manual_review_apps and answer_total_outstanding_doc_requests and 2 more.
- Trial 10Missed 6 of 13 checks — answer_repeat_offender_count, deliverable_jira_ticket_filed and answer_cohort_manual_review_apps and 3 more.
Gemini 3.5 Flash
Mean reward
40%All trials
Across 10 trials on this task, reward ran from 15.4% to 92.3% — 2 of 10 cleared every check.
- 5.2 / 13avg checks passed
- 47avg turns
- 45avg tool calls
- 23mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed 2 of 10 trials. Failures cluster on trace_person_level_cohort_method (10/10 trials).
Common pattern
deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 4 more failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 6 verifiers across 6 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 13 checks — trace_person_level_cohort_method.
- Trial 02Missed 11 of 13 checks — answer_cohort_severity_metrics, answer_repeat_offender_count and deliverable_jira_ticket_filed and 8 more.
- Trial 03Missed 7 of 13 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 4 more.
- Trial 04Missed 11 of 13 checks — answer_cohort_severity_metrics, answer_repeat_offender_count and deliverable_jira_ticket_filed and 8 more.
- Trial 05Missed 11 of 13 checks — answer_cohort_severity_metrics, answer_repeat_offender_count and deliverable_jira_ticket_filed and 8 more.
- Trial 06Missed 1 of 13 checks — trace_person_level_cohort_method.
- Trial 07Missed 7 of 13 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 4 more.
- Trial 08Missed 7 of 13 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 4 more.
- Trial 09Missed 11 of 13 checks — answer_cohort_severity_metrics, answer_repeat_offender_count and deliverable_jira_ticket_filed and 8 more.
- Trial 10Missed 11 of 13 checks — answer_cohort_severity_metrics, answer_repeat_offender_count and deliverable_jira_ticket_filed and 8 more.