Net Institutional Fee RevenueRevenue & Fees
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
92.7%All trials
Across 10 trials on this task, reward ran from 84.1% to 100% — 3 of 10 cleared every check.
- 40.8 / 44avg checks passed
- 18avg turns
- 63avg tool calls
- 3mavg time
Summary · GPT-5.5
GPT-5.5 passed 3 of 10 trials. Failures cluster on jira_followup_ticket_has_evidence (4/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 11 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 44 checks — jira_followup_ticket_has_evidence and trace_reconciled_posted_amount.
- Trial 02Passed every verifier it was graded on.
- Trial 03Missed 3 of 44 checks.
- Trial 04Missed 6 of 44 checks — answer_fee_credits_and_refunds, answer_highest_refund_event_rate_pct and answer_net_fee_revenue and 3 more.
- Trial 05Passed every verifier it was graded on.
- Trial 06Missed 6 of 44 checks — answer_fee_credits_and_refunds, answer_net_fee_revenue and answer_wire_family_net and 3 more.
- Trial 07Missed 7 of 44 checks — answer_fee_credits_and_refunds, answer_highest_refund_event_rate_pct and answer_net_fee_revenue and 4 more.
- Trial 08Passed every verifier it was graded on.
- Trial 09Missed 5 of 44 checks.
- Trial 10Missed 3 of 44 checks — answer_highest_refund_event_rate_pct, jira_followup_ticket_has_evidence and trace_reconciled_posted_amount.
Claude Opus 4.8
Mean reward
84.8%All trials
Across 10 trials on this task, reward ran from 70.5% to 97.7% — 0 of 10 cleared every check.
- 37.3 / 44avg checks passed
- 27avg turns
- 49avg tool calls
- 5mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed none of its 10 trials. Failures cluster on jira_followup_ticket_has_evidence (10/10 trials).
Common pattern
jira_followup_ticket_has_evidence failed in every unsuccessful trial; what else broke varies across the other 13 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 44 checks — jira_followup_ticket_has_evidence.
- Trial 02Missed 10 of 44 checks — answer_fee_credits_and_refunds, answer_net_fee_revenue and answer_oon_atm_family_net and 7 more.
- Trial 03Missed 2 of 44 checks — jira_followup_ticket_has_evidence and slack_close_summary_has_required_figures.
- Trial 04Missed 5 of 44 checks.
- Trial 05Missed 5 of 44 checks.
- Trial 06Missed 13 of 44 checks.
- Trial 07Missed 10 of 44 checks — answer_fee_credits_and_refunds, answer_net_fee_revenue and answer_oon_atm_family_net and 7 more.
- Trial 08Missed 13 of 44 checks.
- Trial 09Missed 2 of 44 checks — jira_followup_ticket_has_evidence and slack_close_summary_has_required_figures.
- Trial 10Missed 6 of 44 checks.
Gemini 3.1 Pro
Mean reward
90.9%All trials
Across 10 trials on this task, reward ran from 90.9% to 90.9% — 0 of 10 cleared every check.
- 40.0 / 44avg checks passed
- 48avg turns
- 53avg tool calls
- 7mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on jira_followup_ticket_exists (10/10 trials).
Common pattern
All 10 unsuccessful trials broke on exactly the same verifiers — jira_followup_ticket_exists, jira_followup_ticket_has_evidence and slack_close_summary_has_required_figures and 1 more failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 4 of 44 checks — jira_followup_ticket_exists, jira_followup_ticket_has_evidence and slack_close_summary_has_required_figures and 1 more.
- Trial 02Missed 4 of 44 checks — jira_followup_ticket_exists, jira_followup_ticket_has_evidence and slack_close_summary_has_required_figures and 1 more.
- Trial 03Missed 4 of 44 checks — jira_followup_ticket_exists, jira_followup_ticket_has_evidence and slack_close_summary_has_required_figures and 1 more.
- Trial 04Missed 4 of 44 checks — jira_followup_ticket_exists, jira_followup_ticket_has_evidence and slack_close_summary_has_required_figures and 1 more.
- Trial 05Missed 4 of 44 checks — jira_followup_ticket_exists, jira_followup_ticket_has_evidence and slack_close_summary_has_required_figures and 1 more.
- Trial 06Missed 4 of 44 checks — jira_followup_ticket_exists, jira_followup_ticket_has_evidence and slack_close_summary_has_required_figures and 1 more.
- Trial 07Missed 4 of 44 checks — jira_followup_ticket_exists, jira_followup_ticket_has_evidence and slack_close_summary_has_required_figures and 1 more.
- Trial 08Missed 4 of 44 checks — jira_followup_ticket_exists, jira_followup_ticket_has_evidence and slack_close_summary_has_required_figures and 1 more.
- Trial 09Missed 4 of 44 checks — jira_followup_ticket_exists, jira_followup_ticket_has_evidence and slack_close_summary_has_required_figures and 1 more.
- Trial 10Missed 4 of 44 checks — jira_followup_ticket_exists, jira_followup_ticket_has_evidence and slack_close_summary_has_required_figures and 1 more.
Gemini 3.5 Flash
Mean reward
53.2%All trials
Across 10 trials on this task, reward ran from 11.4% to 75% — 0 of 10 cleared every check.
- 23.4 / 44avg checks passed
- 67avg turns
- 78avg tool calls
- 18mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on answer_fee_subject_busiest_requester (10/10 trials).
Common pattern
answer_fee_subject_busiest_requester, answer_fee_subject_busiest_requester_ticket_count and answer_fee_subject_distinct_requester_count and 8 more failed in every unsuccessful trial; what else broke varies across the other 30 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Missing field — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Missed 15 of 44 checks.
- Trial 02Missed 15 of 44 checks.
- Trial 03Missed 39 of 44 checks — answer_card_replacement_family_net, answer_check_issuance_family_net and answer_fee_credits_and_refunds and 36 more.
- Trial 04Missed 17 of 44 checks.
- Trial 05Missed 15 of 44 checks.
- Trial 06Missed 15 of 44 checks.
- Trial 07Missed 15 of 44 checks.
- Trial 08Missed 11 of 44 checks.
- Trial 09Missed 39 of 44 checks — answer_card_replacement_family_net, answer_check_issuance_family_net and answer_fee_credits_and_refunds and 36 more.
- Trial 10Missed 25 of 44 checks.