Foreign transaction fee revenue closeoutRevenue & Fees
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
85%All trials
Across 10 trials on this task, reward ran from 83.3% to 100% — 1 of 10 cleared every check.
- 5.1 / 6avg checks passed
- 15avg turns
- 14avg tool calls
- 1mavg time
Summary · GPT-5.5
GPT-5.5 passed 1 of 10 trials. Failures cluster on scopes_status_netting_and_excludes_atm (9/10 trials).
Common pattern
All 9 unsuccessful trials broke on exactly the same verifiers — scopes_status_netting_and_excludes_atm failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 6 checks — scopes_status_netting_and_excludes_atm.
- Trial 02Missed 1 of 6 checks — scopes_status_netting_and_excludes_atm.
- Trial 03Missed 1 of 6 checks — scopes_status_netting_and_excludes_atm.
- Trial 04Missed 1 of 6 checks — scopes_status_netting_and_excludes_atm.
- Trial 05Missed 1 of 6 checks — scopes_status_netting_and_excludes_atm.
- Trial 06Missed 1 of 6 checks — scopes_status_netting_and_excludes_atm.
- Trial 07Passed every verifier it was graded on.
- Trial 08Missed 1 of 6 checks — scopes_status_netting_and_excludes_atm.
- Trial 09Missed 1 of 6 checks — scopes_status_netting_and_excludes_atm.
- Trial 10Missed 1 of 6 checks — scopes_status_netting_and_excludes_atm.
Claude Opus 4.8
Mean reward
81.7%All trials
Across 10 trials on this task, reward ran from 66.7% to 100% — 1 of 10 cleared every check.
- 4.9 / 6avg checks passed
- 10avg turns
- 9avg tool calls
- 1mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 1 of 10 trials. Failures cluster on scopes_status_netting_and_excludes_atm (9/10 trials).
Common pattern
scopes_status_netting_and_excludes_atm failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 6 checks — scopes_status_netting_and_excludes_atm.
- Trial 02Missed 1 of 6 checks — scopes_status_netting_and_excludes_atm.
- Trial 03Missed 1 of 6 checks — scopes_status_netting_and_excludes_atm.
- Trial 04Missed 2 of 6 checks — gold_net_foreign_tx_fee_collected and scopes_status_netting_and_excludes_atm.
- Trial 05Missed 1 of 6 checks — scopes_status_netting_and_excludes_atm.
- Trial 06Missed 1 of 6 checks — scopes_status_netting_and_excludes_atm.
- Trial 07Missed 1 of 6 checks — scopes_status_netting_and_excludes_atm.
- Trial 08Missed 1 of 6 checks — scopes_status_netting_and_excludes_atm.
- Trial 09Missed 2 of 6 checks — gold_net_foreign_tx_fee_collected and scopes_status_netting_and_excludes_atm.
- Trial 10Passed every verifier it was graded on.
Gemini 3.1 Pro
Mean reward
66.7%All trials
Across 10 trials on this task, reward ran from 66.7% to 66.7% — 0 of 10 cleared every check.
- 4.0 / 6avg checks passed
- 18avg turns
- 16avg tool calls
- 2mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on gold_net_foreign_tx_fee_collected (10/10 trials).
Common pattern
All 10 unsuccessful trials broke on exactly the same verifiers — gold_net_foreign_tx_fee_collected and scopes_status_netting_and_excludes_atm failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 6 checks — gold_net_foreign_tx_fee_collected and scopes_status_netting_and_excludes_atm.
- Trial 02Missed 2 of 6 checks — gold_net_foreign_tx_fee_collected and scopes_status_netting_and_excludes_atm.
- Trial 03Missed 2 of 6 checks — gold_net_foreign_tx_fee_collected and scopes_status_netting_and_excludes_atm.
- Trial 04Missed 2 of 6 checks — gold_net_foreign_tx_fee_collected and scopes_status_netting_and_excludes_atm.
- Trial 05Missed 2 of 6 checks — gold_net_foreign_tx_fee_collected and scopes_status_netting_and_excludes_atm.
- Trial 06Missed 2 of 6 checks — gold_net_foreign_tx_fee_collected and scopes_status_netting_and_excludes_atm.
- Trial 07Missed 2 of 6 checks — gold_net_foreign_tx_fee_collected and scopes_status_netting_and_excludes_atm.
- Trial 08Missed 2 of 6 checks — gold_net_foreign_tx_fee_collected and scopes_status_netting_and_excludes_atm.
- Trial 09Missed 2 of 6 checks — gold_net_foreign_tx_fee_collected and scopes_status_netting_and_excludes_atm.
- Trial 10Missed 2 of 6 checks — gold_net_foreign_tx_fee_collected and scopes_status_netting_and_excludes_atm.
Gemini 3.5 Flash
Mean reward
26.7%All trials
Across 10 trials on this task, reward ran from 16.7% to 50% — 0 of 10 cleared every check.
- 1.6 / 6avg checks passed
- 41avg turns
- 46avg tool calls
- 18mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on deliverable_slack_channel_exists (10/10 trials).
Common pattern
deliverable_slack_channel_exists, deliverable_slack_message_posted and scopes_status_netting_and_excludes_atm failed in every unsuccessful trial; what else broke varies across the other 2 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 5 of 6 checks — deliverable_slack_channel_exists, deliverable_slack_message_posted and gold_net_foreign_tx_fee_collected and 2 more.
- Trial 02Missed 5 of 6 checks — deliverable_slack_channel_exists, deliverable_slack_message_posted and gold_net_foreign_tx_fee_collected and 2 more.
- Trial 03Missed 4 of 6 checks — deliverable_slack_channel_exists, deliverable_slack_message_posted and gold_net_foreign_tx_fee_collected and 1 more.
- Trial 04Missed 5 of 6 checks — deliverable_slack_channel_exists, deliverable_slack_message_posted and gold_net_foreign_tx_fee_collected and 2 more.
- Trial 05Missed 4 of 6 checks — deliverable_slack_channel_exists, deliverable_slack_message_posted and gold_net_foreign_tx_fee_collected and 1 more.
- Trial 06Missed 5 of 6 checks — deliverable_slack_channel_exists, deliverable_slack_message_posted and gold_net_foreign_tx_fee_collected and 2 more.
- Trial 07Missed 5 of 6 checks — deliverable_slack_channel_exists, deliverable_slack_message_posted and gold_net_foreign_tx_fee_collected and 2 more.
- Trial 08Missed 4 of 6 checks — deliverable_slack_channel_exists, deliverable_slack_message_posted and gold_net_foreign_tx_fee_collected and 1 more.
- Trial 09Missed 3 of 6 checks — deliverable_slack_channel_exists, deliverable_slack_message_posted and scopes_status_netting_and_excludes_atm.
- Trial 10Missed 4 of 6 checks — deliverable_slack_channel_exists, deliverable_slack_message_posted and gold_net_foreign_tx_fee_collected and 1 more.