Long-horizon fee-revenue reviewRevenue & Fees
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
90%All trials
Across 10 trials on this task, reward ran from 77.8% to 100% — 3 of 10 cleared every check.
- 8.1 / 9avg checks passed
- 9avg turns
- 7avg tool calls
- 1mavg time
Summary · GPT-5.5
GPT-5.5 passed 3 of 10 trials. Failures cluster on cp6_peak_over_prior_increase (7/10 trials).
Common pattern
cp6_peak_over_prior_increase failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 9 checks — cp6_peak_over_prior_increase.
- Trial 02Missed 1 of 9 checks — cp6_peak_over_prior_increase.
- Trial 03Passed every verifier it was graded on.
- Trial 04Missed 1 of 9 checks — cp6_peak_over_prior_increase.
- Trial 05Passed every verifier it was graded on.
- Trial 06Missed 2 of 9 checks — cp6_peak_over_prior_increase and cp7_top_category_share.
- Trial 07Missed 1 of 9 checks — cp6_peak_over_prior_increase.
- Trial 08Passed every verifier it was graded on.
- Trial 09Missed 2 of 9 checks — cp6_peak_over_prior_increase and cp7_top_category_share.
- Trial 10Missed 1 of 9 checks — cp6_peak_over_prior_increase.
Claude Opus 4.8
Mean reward
87.8%All trials
Across 10 trials on this task, reward ran from 66.7% to 100% — 5 of 10 cleared every check.
- 7.9 / 9avg checks passed
- 11avg turns
- 9avg tool calls
- 1mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 5 of 10 trials. Failures cluster on cp6_peak_over_prior_increase (5/10 trials).
Common pattern
cp6_peak_over_prior_increase failed in every unsuccessful trial; what else broke varies across the other 2 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 5 verifiers across 5 of 10 trials.
Per-trial notes
- Trial 01Missed 3 of 9 checks — cp2_peak_year_net, cp6_peak_over_prior_increase and cp7_top_category_share.
- Trial 02Passed every verifier it was graded on.
- Trial 03Missed 3 of 9 checks — cp2_peak_year_net, cp6_peak_over_prior_increase and cp7_top_category_share.
- Trial 04Passed every verifier it was graded on.
- Trial 05Missed 3 of 9 checks — cp2_peak_year_net, cp6_peak_over_prior_increase and cp7_top_category_share.
- Trial 06Passed every verifier it was graded on.
- Trial 07Missed 1 of 9 checks — cp6_peak_over_prior_increase.
- Trial 08Passed every verifier it was graded on.
- Trial 09Missed 1 of 9 checks — cp6_peak_over_prior_increase.
- Trial 10Passed every verifier it was graded on.
Gemini 3.1 Pro
Mean reward
92.2%All trials
Across 10 trials on this task, reward ran from 77.8% to 100% — 5 of 10 cleared every check.
- 8.3 / 9avg checks passed
- 15avg turns
- 12avg tool calls
- 2mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 5 of 10 trials. Failures cluster on cp6_peak_over_prior_increase (5/10 trials).
Common pattern
cp6_peak_over_prior_increase failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 5 verifiers across 5 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 9 checks — cp6_peak_over_prior_increase and cp7_top_category_share.
- Trial 02Passed every verifier it was graded on.
- Trial 03Missed 2 of 9 checks — cp6_peak_over_prior_increase and cp7_top_category_share.
- Trial 04Passed every verifier it was graded on.
- Trial 05Missed 1 of 9 checks — cp6_peak_over_prior_increase.
- Trial 06Missed 1 of 9 checks — cp6_peak_over_prior_increase.
- Trial 07Missed 1 of 9 checks — cp6_peak_over_prior_increase.
- Trial 08Passed every verifier it was graded on.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Gemini 3.5 Flash
Mean reward
20%All trials
Across 10 trials on this task, reward ran from 0% to 100% — 2 of 10 cleared every check.
- 1.8 / 9avg checks passed
- 45avg turns
- 56avg tool calls
- 23mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed 2 of 10 trials. Failures cluster on cp1_peak_year (8/10 trials).
Common pattern
All 8 unsuccessful trials broke on exactly the same verifiers — cp1_peak_year, cp2_peak_year_net and cp3_top_category and 6 more failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 9 of 9 checks — cp1_peak_year, cp2_peak_year_net and cp3_top_category and 6 more.
- Trial 02Missed 9 of 9 checks — cp1_peak_year, cp2_peak_year_net and cp3_top_category and 6 more.
- Trial 03Missed 9 of 9 checks — cp1_peak_year, cp2_peak_year_net and cp3_top_category and 6 more.
- Trial 04Missed 9 of 9 checks — cp1_peak_year, cp2_peak_year_net and cp3_top_category and 6 more.
- Trial 05Passed every verifier it was graded on.
- Trial 06Passed every verifier it was graded on.
- Trial 07Missed 9 of 9 checks — cp1_peak_year, cp2_peak_year_net and cp3_top_category and 6 more.
- Trial 08Missed 9 of 9 checks — cp1_peak_year, cp2_peak_year_net and cp3_top_category and 6 more.
- Trial 09Missed 9 of 9 checks — cp1_peak_year, cp2_peak_year_net and cp3_top_category and 6 more.
- Trial 10Missed 9 of 9 checks — cp1_peak_year, cp2_peak_year_net and cp3_top_category and 6 more.