Institutional fee-revenue reconstructionRevenue & Fees
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
61.3%All trials
Across 10 trials on this task, reward ran from 50% to 75% — 0 of 10 cleared every check.
- 4.9 / 8avg checks passed
- 11avg turns
- 9avg tool calls
- 2mavg time
Summary · GPT-5.5
GPT-5.5 passed none of its 10 trials. Failures cluster on cp2_peak_year_net (10/10 trials).
Common pattern
cp2_peak_year_net and rubric_collected_not_scheduled failed in every unsuccessful trial; what else broke varies across the other 2 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 3 of 8 checks — cp2_peak_year_net, cp4_top_category_net and rubric_collected_not_scheduled.
- Trial 02Missed 4 of 8 checks — cp2_peak_year_net, cp4_top_category_net and rubric_collected_not_scheduled and 1 more.
- Trial 03Missed 4 of 8 checks — cp2_peak_year_net, cp4_top_category_net and rubric_collected_not_scheduled and 1 more.
- Trial 04Missed 4 of 8 checks — cp2_peak_year_net, cp4_top_category_net and rubric_collected_not_scheduled and 1 more.
- Trial 05Missed 3 of 8 checks — cp2_peak_year_net, cp4_top_category_net and rubric_collected_not_scheduled.
- Trial 06Missed 3 of 8 checks — cp2_peak_year_net, rubric_collected_not_scheduled and rubric_consulted_fee_policy.
- Trial 07Missed 2 of 8 checks — cp2_peak_year_net and rubric_collected_not_scheduled.
- Trial 08Missed 3 of 8 checks — cp2_peak_year_net, rubric_collected_not_scheduled and rubric_consulted_fee_policy.
- Trial 09Missed 2 of 8 checks — cp2_peak_year_net and rubric_collected_not_scheduled.
- Trial 10Missed 3 of 8 checks — cp2_peak_year_net, rubric_collected_not_scheduled and rubric_consulted_fee_policy.
Claude Opus 4.8
Mean reward
70%All trials
Across 10 trials on this task, reward ran from 62.5% to 100% — 1 of 10 cleared every check.
- 5.6 / 8avg checks passed
- 11avg turns
- 13avg tool calls
- 2mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 1 of 10 trials. Failures cluster on cp2_peak_year_net (9/10 trials).
Common pattern
cp2_peak_year_net and rubric_collected_not_scheduled failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 3 of 8 checks — cp2_peak_year_net, cp4_top_category_net and rubric_collected_not_scheduled.
- Trial 02Missed 2 of 8 checks — cp2_peak_year_net and rubric_collected_not_scheduled.
- Trial 03Missed 2 of 8 checks — cp2_peak_year_net and rubric_collected_not_scheduled.
- Trial 04Missed 3 of 8 checks — cp2_peak_year_net, cp4_top_category_net and rubric_collected_not_scheduled.
- Trial 05Missed 3 of 8 checks — cp2_peak_year_net, cp4_top_category_net and rubric_collected_not_scheduled.
- Trial 06Missed 2 of 8 checks — cp2_peak_year_net and rubric_collected_not_scheduled.
- Trial 07Missed 3 of 8 checks — cp2_peak_year_net, cp4_top_category_net and rubric_collected_not_scheduled.
- Trial 08Passed every verifier it was graded on.
- Trial 09Missed 3 of 8 checks — cp2_peak_year_net, cp4_top_category_net and rubric_collected_not_scheduled.
- Trial 10Missed 3 of 8 checks — cp2_peak_year_net, cp4_top_category_net and rubric_collected_not_scheduled.
Gemini 3.1 Pro
Mean reward
77.5%All trials
Across 10 trials on this task, reward ran from 62.5% to 100% — 4 of 10 cleared every check.
- 6.2 / 8avg checks passed
- 21avg turns
- 19avg tool calls
- 3mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 4 of 10 trials. Failures cluster on cp2_peak_year_net (6/10 trials).
Common pattern
cp2_peak_year_net and rubric_collected_not_scheduled failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 6 verifiers across 6 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 8 checks — cp2_peak_year_net and rubric_collected_not_scheduled.
- Trial 02Missed 2 of 8 checks — cp2_peak_year_net and rubric_collected_not_scheduled.
- Trial 03Missed 3 of 8 checks — cp2_peak_year_net, rubric_collected_not_scheduled and rubric_consulted_fee_policy.
- Trial 04Missed 3 of 8 checks — cp2_peak_year_net, rubric_collected_not_scheduled and rubric_consulted_fee_policy.
- Trial 05Passed every verifier it was graded on.
- Trial 06Missed 1 of 8 checks — rubric_consulted_fee_policy.
- Trial 07Missed 1 of 8 checks — rubric_consulted_fee_policy.
- Trial 08Missed 2 of 8 checks — cp2_peak_year_net and rubric_collected_not_scheduled.
- Trial 09Missed 3 of 8 checks — cp2_peak_year_net, rubric_collected_not_scheduled and rubric_consulted_fee_policy.
- Trial 10Missed 1 of 8 checks — rubric_consulted_fee_policy.
Gemini 3.5 Flash
Mean reward
27.5%All trials
Across 10 trials on this task, reward ran from 12.5% to 100% — 1 of 10 cleared every check.
- 2.2 / 8avg checks passed
- 61avg turns
- 96avg tool calls
- 8mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed 1 of 10 trials. Failures cluster on cp1_peak_year (9/10 trials).
Common pattern
cp1_peak_year, cp2_peak_year_net and cp4_top_category_net and 1 more failed in every unsuccessful trial; what else broke varies across the other 3 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Missing field — 6 verifiers across 6 of 10 trials.
Per-trial notes
- Trial 01Missed 7 of 8 checks — cp1_peak_year, cp2_peak_year_net and cp3_top_fee_category and 4 more.
- Trial 02Missed 7 of 8 checks — cp1_peak_year, cp2_peak_year_net and cp3_top_fee_category and 4 more.
- Trial 03Missed 7 of 8 checks — cp1_peak_year, cp2_peak_year_net and cp3_top_fee_category and 4 more.
- Trial 04Missed 7 of 8 checks — cp1_peak_year, cp2_peak_year_net and cp3_top_fee_category and 4 more.
- Trial 05Missed 6 of 8 checks — cp1_peak_year, cp2_peak_year_net and cp3_top_fee_category and 3 more.
- Trial 06Missed 4 of 8 checks — cp1_peak_year, cp2_peak_year_net and cp4_top_category_net and 1 more.
- Trial 07Missed 6 of 8 checks — cp1_peak_year, cp2_peak_year_net and cp3_top_fee_category and 3 more.
- Trial 08Missed 7 of 8 checks — cp1_peak_year, cp2_peak_year_net and cp3_top_fee_category and 4 more.
- Trial 09Missed 7 of 8 checks — cp1_peak_year, cp2_peak_year_net and cp3_top_fee_category and 4 more.
- Trial 10Passed every verifier it was graded on.