Flex Reserve funding: reconstruct in-red Overview balancesReconciliation & Settlement
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
75%All trials
Across 10 trials on this task, reward ran from 50% to 83.3% — 0 of 10 cleared every check.
- 9.0 / 12avg checks passed
- 26avg turns
- 36avg tool calls
- 3mavg time
Summary · GPT-5.5
GPT-5.5 passed none of its 10 trials. Failures cluster on trace_flex_reserve_recon_method (10/10 trials).
Common pattern
trace_flex_reserve_recon_method failed in every unsuccessful trial; what else broke varies across the other 5 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 12 checks — answer_in_red_population and trace_flex_reserve_recon_method.
- Trial 02Missed 4 of 12 checks — answer_in_red_population, answer_stale_target_accounts and deliverable_slack_message_content and 1 more.
- Trial 03Missed 4 of 12 checks — answer_in_red_population, answer_stale_target_accounts and deliverable_slack_message_content and 1 more.
- Trial 04Missed 2 of 12 checks — answer_in_red_population and trace_flex_reserve_recon_method.
- Trial 05Missed 2 of 12 checks — answer_in_red_population and trace_flex_reserve_recon_method.
- Trial 06Missed 3 of 12 checks — answer_stale_target_accounts, deliverable_slack_message_content and trace_flex_reserve_recon_method.
- Trial 07Missed 6 of 12 checks — answer_audited_and_recon_match, answer_in_red_population and answer_stale_target_accounts and 3 more.
- Trial 08Missed 3 of 12 checks — answer_stale_target_accounts, deliverable_slack_message_content and trace_flex_reserve_recon_method.
- Trial 09Missed 2 of 12 checks — answer_in_red_population and trace_flex_reserve_recon_method.
- Trial 10Missed 2 of 12 checks — answer_in_red_population and trace_flex_reserve_recon_method.
Claude Opus 4.8
Mean reward
79.2%All trials
Across 10 trials on this task, reward ran from 66.7% to 91.7% — 0 of 10 cleared every check.
- 9.5 / 12avg checks passed
- 19avg turns
- 27avg tool calls
- 3mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed none of its 10 trials. Failures cluster on answer_stale_target_accounts (8/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 4 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 3 of 12 checks — answer_stale_target_accounts, deliverable_slack_message_content and trace_flex_reserve_recon_method.
- Trial 02Missed 4 of 12 checks — answer_in_red_population, answer_stale_target_accounts and deliverable_slack_message_content and 1 more.
- Trial 03Missed 2 of 12 checks — answer_stale_target_accounts and deliverable_slack_message_content.
- Trial 04Missed 1 of 12 checks — answer_in_red_population.
- Trial 05Missed 2 of 12 checks — answer_stale_target_accounts and deliverable_slack_message_content.
- Trial 06Missed 3 of 12 checks — answer_in_red_population, answer_stale_target_accounts and deliverable_slack_message_content.
- Trial 07Missed 1 of 12 checks — answer_in_red_population.
- Trial 08Missed 3 of 12 checks — answer_stale_target_accounts, deliverable_slack_message_content and trace_flex_reserve_recon_method.
- Trial 09Missed 4 of 12 checks — answer_in_red_population, answer_stale_target_accounts and deliverable_slack_message_content and 1 more.
- Trial 10Missed 2 of 12 checks — answer_stale_target_accounts and deliverable_slack_message_content.
Gemini 3.1 Pro
Mean reward
78.3%All trials
Across 10 trials on this task, reward ran from 58.3% to 91.7% — 3 of 10 cleared every check.
- 9.4 / 12avg checks passed
- 53avg turns
- 51avg tool calls
- 14mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 3 of 10 trials. Failures cluster on trace_flex_reserve_recon_method (10/10 trials).
Common pattern
answer_stale_target_accounts and trace_flex_reserve_recon_method failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 6 verifiers across 6 of 10 trials.
Per-trial notes
- Trial 01Missed 3 of 12 checks — answer_stale_target_accounts, deliverable_slack_message_content and trace_flex_reserve_recon_method.
- Trial 02Missed 3 of 12 checks — answer_stale_target_accounts, deliverable_slack_message_content and trace_flex_reserve_recon_method.
- Trial 03Missed 1 of 12 checks — trace_flex_reserve_recon_method.
- Trial 04Missed 1 of 12 checks — trace_flex_reserve_recon_method.
- Trial 05Missed 3 of 12 checks — answer_stale_target_accounts, deliverable_slack_message_content and trace_flex_reserve_recon_method.
- Trial 06Missed 1 of 12 checks — trace_flex_reserve_recon_method.
- Trial 07Missed 3 of 12 checks — answer_stale_target_accounts, deliverable_slack_message_content and trace_flex_reserve_recon_method.
- Trial 08Missed 3 of 12 checks — answer_stale_target_accounts, deliverable_slack_message_content and trace_flex_reserve_recon_method.
- Trial 09Missed 3 of 12 checks — answer_stale_target_accounts, deliverable_slack_message_content and trace_flex_reserve_recon_method.
- Trial 10Missed 5 of 12 checks — answer_audited_and_recon_match, answer_in_red_population and answer_stale_target_accounts and 2 more.
Gemini 3.5 Flash
Mean reward
13.3%All trials
Across 10 trials on this task, reward ran from 8.3% to 16.7% — 0 of 10 cleared every check.
- 1.6 / 12avg checks passed
- 30avg turns
- 29avg tool calls
- 17mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on answer_audited_and_recon_match (10/10 trials).
Common pattern
answer_audited_and_recon_match, answer_in_red_population and answer_stale_target_accounts and 7 more failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Missed 11 of 12 checks — answer_audited_and_recon_match, answer_in_red_population and answer_stale_target_accounts and 8 more.
- Trial 02Missed 10 of 12 checks — answer_audited_and_recon_match, answer_in_red_population and answer_stale_target_accounts and 7 more.
- Trial 03Missed 10 of 12 checks — answer_audited_and_recon_match, answer_in_red_population and answer_stale_target_accounts and 7 more.
- Trial 04Missed 10 of 12 checks — answer_audited_and_recon_match, answer_in_red_population and answer_stale_target_accounts and 7 more.
- Trial 05Missed 11 of 12 checks — answer_audited_and_recon_match, answer_in_red_population and answer_stale_target_accounts and 8 more.
- Trial 06Missed 10 of 12 checks — answer_audited_and_recon_match, answer_in_red_population and answer_stale_target_accounts and 7 more.
- Trial 07Missed 11 of 12 checks — answer_audited_and_recon_match, answer_in_red_population and answer_stale_target_accounts and 8 more.
- Trial 08Missed 11 of 12 checks — answer_audited_and_recon_match, answer_in_red_population and answer_stale_target_accounts and 8 more.
- Trial 09Missed 10 of 12 checks — answer_audited_and_recon_match, answer_in_red_population and answer_stale_target_accounts and 7 more.
- Trial 10Missed 10 of 12 checks — answer_audited_and_recon_match, answer_in_red_population and answer_stale_target_accounts and 7 more.