Subscription dead invoice rateSubscriptions & Billing
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
46.7%All trials
Across 10 trials on this task, reward ran from 33.3% to 66.7% — 4 of 10 cleared every check.
- 1.4 / 3avg checks passed
- 15avg turns
- 15avg tool calls
- 2mavg time
Summary · GPT-5.5
GPT-5.5 passed 4 of 10 trials. Failures cluster on process_confluence_read (10/10 trials).
Common pattern
All 6 unsuccessful trials broke on exactly the same verifiers — gold_billed_revenue_collected_share and process_confluence_read failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 6 verifiers across 6 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 3 checks — gold_billed_revenue_collected_share and process_confluence_read.
- Trial 02Missed 1 of 3 checks — process_confluence_read.
- Trial 03Missed 2 of 3 checks — gold_billed_revenue_collected_share and process_confluence_read.
- Trial 04Missed 2 of 3 checks — gold_billed_revenue_collected_share and process_confluence_read.
- Trial 05Missed 2 of 3 checks — gold_billed_revenue_collected_share and process_confluence_read.
- Trial 06Missed 1 of 3 checks — process_confluence_read.
- Trial 07Missed 1 of 3 checks — process_confluence_read.
- Trial 08Missed 1 of 3 checks — process_confluence_read.
- Trial 09Missed 2 of 3 checks — gold_billed_revenue_collected_share and process_confluence_read.
- Trial 10Missed 2 of 3 checks — gold_billed_revenue_collected_share and process_confluence_read.
Claude Opus 4.8
Mean reward
40%All trials
Across 10 trials on this task, reward ran from 33.3% to 66.7% — 2 of 10 cleared every check.
- 1.2 / 3avg checks passed
- 7avg turns
- 6avg tool calls
- 1mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 2 of 10 trials. Failures cluster on process_confluence_read (10/10 trials).
Common pattern
All 8 unsuccessful trials broke on exactly the same verifiers — gold_billed_revenue_collected_share and process_confluence_read failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 3 checks — gold_billed_revenue_collected_share and process_confluence_read.
- Trial 02Missed 1 of 3 checks — process_confluence_read.
- Trial 03Missed 1 of 3 checks — process_confluence_read.
- Trial 04Missed 2 of 3 checks — gold_billed_revenue_collected_share and process_confluence_read.
- Trial 05Missed 2 of 3 checks — gold_billed_revenue_collected_share and process_confluence_read.
- Trial 06Missed 2 of 3 checks — gold_billed_revenue_collected_share and process_confluence_read.
- Trial 07Missed 2 of 3 checks — gold_billed_revenue_collected_share and process_confluence_read.
- Trial 08Missed 2 of 3 checks — gold_billed_revenue_collected_share and process_confluence_read.
- Trial 09Missed 2 of 3 checks — gold_billed_revenue_collected_share and process_confluence_read.
- Trial 10Missed 2 of 3 checks — gold_billed_revenue_collected_share and process_confluence_read.
Gemini 3.1 Pro
Mean reward
86.7%All trials
Across 10 trials on this task, reward ran from 33.3% to 100% — 8 of 10 cleared every check.
- 2.6 / 3avg checks passed
- 45avg turns
- 43avg tool calls
- 4mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 8 of 10 trials. Failures cluster on gold_billed_revenue_collected_share (2/10 trials).
Common pattern
gold_billed_revenue_collected_share failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 2 verifiers across 2 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 3 checks — process_confluence_read.
- Trial 02Passed every verifier it was graded on.
- Trial 03Passed every verifier it was graded on.
- Trial 04Missed 2 of 3 checks — gold_billed_revenue_collected_share and process_confluence_read.
- Trial 05Passed every verifier it was graded on.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Passed every verifier it was graded on.
- Trial 09Missed 1 of 3 checks — gold_billed_revenue_collected_share.
- Trial 10Passed every verifier it was graded on.
Gemini 3.5 Flash
Mean reward
80%All trials
Across 10 trials on this task, reward ran from 66.7% to 100% — 5 of 10 cleared every check.
- 2.4 / 3avg checks passed
- 71avg turns
- 69avg tool calls
- 34mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed 5 of 10 trials. Failures cluster on gold_billed_revenue_collected_share (5/10 trials).
Common pattern
All 5 unsuccessful trials broke on exactly the same verifiers — gold_billed_revenue_collected_share failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Near miss — 4 verifiers across 4 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Missed 1 of 3 checks — gold_billed_revenue_collected_share.
- Trial 03Passed every verifier it was graded on.
- Trial 04Passed every verifier it was graded on.
- Trial 05Missed 1 of 3 checks — gold_billed_revenue_collected_share.
- Trial 06Missed 1 of 3 checks — process_confluence_read.
- Trial 07Passed every verifier it was graded on.
- Trial 08Missed 1 of 3 checks — gold_billed_revenue_collected_share.
- Trial 09Missed 1 of 3 checks — gold_billed_revenue_collected_share.
- Trial 10Missed 1 of 3 checks — gold_billed_revenue_collected_share.