Inbound-transfer point-in-time false-positive backtestRisk & Fraud
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
99.5%All trials
Across 10 trials on this task, reward ran from 94.7% to 100% — 9 of 10 cleared every check.
- 18.9 / 19avg checks passed
- 14avg turns
- 16avg tool calls
- 3mavg time
Summary · GPT-5.5
GPT-5.5 passed 9 of 10 trials. Failures cluster on answer_lookahead_correction (1/10 trials).
Common pattern
All 1 unsuccessful trials broke on exactly the same verifiers — answer_lookahead_correction failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 1 verifier across 1 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Passed every verifier it was graded on.
- Trial 03Passed every verifier it was graded on.
- Trial 04Passed every verifier it was graded on.
- Trial 05Passed every verifier it was graded on.
- Trial 06Missed 1 of 19 checks — answer_lookahead_correction.
- Trial 07Passed every verifier it was graded on.
- Trial 08Passed every verifier it was graded on.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Claude Opus 4.8
Mean reward
94.2%All trials
Across 10 trials on this task, reward ran from 89.5% to 94.7% — 9 of 10 cleared every check.
- 17.9 / 19avg checks passed
- 16avg turns
- 18avg tool calls
- 4mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 9 of 10 trials. Failures cluster on trace_pending_vs_posted_method (10/10 trials).
Common pattern
All 1 unsuccessful trials broke on exactly the same verifiers — deliverable_slack_message_content and trace_pending_vs_posted_method failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 1 verifier across 1 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 19 checks — trace_pending_vs_posted_method.
- Trial 02Missed 1 of 19 checks — trace_pending_vs_posted_method.
- Trial 03Missed 1 of 19 checks — trace_pending_vs_posted_method.
- Trial 04Missed 1 of 19 checks — trace_pending_vs_posted_method.
- Trial 05Missed 1 of 19 checks — trace_pending_vs_posted_method.
- Trial 06Missed 1 of 19 checks — trace_pending_vs_posted_method.
- Trial 07Missed 1 of 19 checks — trace_pending_vs_posted_method.
- Trial 08Missed 1 of 19 checks — trace_pending_vs_posted_method.
- Trial 09Missed 2 of 19 checks — deliverable_slack_message_content and trace_pending_vs_posted_method.
- Trial 10Missed 1 of 19 checks — trace_pending_vs_posted_method.
Gemini 3.1 Pro
Mean reward
65.8%All trials
Across 10 trials on this task, reward ran from 52.6% to 84.2% — 0 of 10 cleared every check.
- 12.5 / 19avg checks passed
- 22avg turns
- 20avg tool calls
- 4mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on answer_rollout_decision_flip (10/10 trials).
Common pattern
answer_rollout_decision_flip and deliverable_slack_message_content failed in every unsuccessful trial; what else broke varies across the other 7 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 7 of 19 checks — answer_current_row_view, answer_lookahead_correction and answer_rollout_decision_flip and 4 more.
- Trial 02Missed 7 of 19 checks — answer_current_row_view, answer_lookahead_correction and answer_rollout_decision_flip and 4 more.
- Trial 03Missed 8 of 19 checks — answer_current_row_view, answer_gate_margin and answer_lookahead_correction and 5 more.
- Trial 04Missed 6 of 19 checks — answer_current_row_view, answer_lookahead_correction and answer_rollout_decision_flip and 3 more.
- Trial 05Missed 9 of 19 checks — answer_current_row_view, answer_gate_margin and answer_lookahead_correction and 6 more.
- Trial 06Missed 4 of 19 checks — answer_current_row_view, answer_lookahead_correction and answer_rollout_decision_flip and 1 more.
- Trial 07Missed 4 of 19 checks — answer_current_row_view, answer_lookahead_correction and answer_rollout_decision_flip and 1 more.
- Trial 08Missed 3 of 19 checks — answer_gate_margin, answer_rollout_decision_flip and deliverable_slack_message_content.
- Trial 09Missed 8 of 19 checks — answer_current_row_view, answer_lookahead_correction and answer_rollout_decision_flip and 5 more.
- Trial 10Missed 9 of 19 checks — answer_current_row_view, answer_gate_margin and answer_lookahead_correction and 6 more.
Gemini 3.5 Flash
Mean reward
41.6%All trials
Across 10 trials on this task, reward ran from 10.5% to 89.5% — 0 of 10 cleared every check.
- 7.9 / 19avg checks passed
- 34avg turns
- 31avg tool calls
- 8mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on deliverable_slack_message_content (10/10 trials).
Common pattern
deliverable_slack_message_content failed in every unsuccessful trial; what else broke varies across the other 16 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 5 of 19 checks — answer_current_row_view, answer_lookahead_correction and answer_rollout_decision_flip and 2 more.
- Trial 02Missed 3 of 19 checks — deliverable_slack_message_content, trace_pending_vs_posted_method and trace_point_in_time_method.
- Trial 03Missed 17 of 19 checks — answer_blocked_cohort, answer_current_row_view and answer_excluded_attempt_exposure and 14 more.
- Trial 04Missed 17 of 19 checks — answer_blocked_cohort, answer_current_row_view and answer_excluded_attempt_exposure and 14 more.
- Trial 05Missed 2 of 19 checks — deliverable_slack_message_content and trace_pending_vs_posted_method.
- Trial 06Missed 16 of 19 checks — answer_blocked_cohort, answer_current_row_view and answer_excluded_attempt_exposure and 13 more.
- Trial 07Missed 14 of 19 checks — answer_blocked_cohort, answer_current_row_view and answer_lookahead_correction and 11 more.
- Trial 08Missed 16 of 19 checks — answer_blocked_cohort, answer_current_row_view and answer_excluded_attempt_exposure and 13 more.
- Trial 09Missed 4 of 19 checks — answer_blocked_cohort, answer_lookahead_correction and deliverable_slack_message_content and 1 more.
- Trial 10Missed 17 of 19 checks — answer_blocked_cohort, answer_current_row_view and answer_excluded_attempt_exposure and 14 more.