Subscription grace exhaustionSubscriptions & Billing
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
60%All trials
Across 10 trials on this task, reward ran from 33.3% to 66.7% — 8 of 10 cleared every check.
- 1.8 / 3avg checks passed
- 11avg turns
- 11avg tool calls
- 1mavg time
Summary · GPT-5.5
GPT-5.5 passed 8 of 10 trials. Failures cluster on process_confluence_read (10/10 trials).
Common pattern
All 2 unsuccessful trials broke on exactly the same verifiers — gold_grace_exhaustion_cancellations and process_confluence_read failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 2 verifiers across 2 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 3 checks — process_confluence_read.
- Trial 02Missed 1 of 3 checks — process_confluence_read.
- Trial 03Missed 1 of 3 checks — process_confluence_read.
- Trial 04Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 05Missed 1 of 3 checks — process_confluence_read.
- Trial 06Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 07Missed 1 of 3 checks — process_confluence_read.
- Trial 08Missed 1 of 3 checks — process_confluence_read.
- Trial 09Missed 1 of 3 checks — process_confluence_read.
- Trial 10Missed 1 of 3 checks — process_confluence_read.
Claude Opus 4.8
Mean reward
33.3%All trials
Across 10 trials on this task, reward ran from 33.3% to 33.3% — 0 of 10 cleared every check.
- 1.0 / 3avg checks passed
- 6avg turns
- 6avg tool calls
- 0mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed none of its 10 trials. Failures cluster on gold_grace_exhaustion_cancellations (10/10 trials).
Common pattern
All 10 unsuccessful trials broke on exactly the same verifiers — gold_grace_exhaustion_cancellations and process_confluence_read failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 02Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 03Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 04Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 05Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 06Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 07Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 08Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 09Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 10Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
Gemini 3.1 Pro
Mean reward
33.3%All trials
Across 10 trials on this task, reward ran from 33.3% to 33.3% — 0 of 10 cleared every check.
- 1.0 / 3avg checks passed
- 10avg turns
- 8avg tool calls
- 1mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on gold_grace_exhaustion_cancellations (10/10 trials).
Common pattern
All 10 unsuccessful trials broke on exactly the same verifiers — gold_grace_exhaustion_cancellations and process_confluence_read failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 02Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 03Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 04Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 05Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 06Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 07Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 08Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 09Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 10Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
Gemini 3.5 Flash
Mean reward
76.7%All trials
Across 10 trials on this task, reward ran from 33.3% to 100% — 4 of 10 cleared every check.
- 2.3 / 3avg checks passed
- 74avg turns
- 73avg tool calls
- 35mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed 4 of 10 trials. Failures cluster on gold_grace_exhaustion_cancellations (6/10 trials).
Common pattern
gold_grace_exhaustion_cancellations failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 5 verifiers across 5 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 3 checks — gold_grace_exhaustion_cancellations.
- Trial 02Passed every verifier it was graded on.
- Trial 03Missed 2 of 3 checks — gold_grace_exhaustion_cancellations and process_confluence_read.
- Trial 04Missed 1 of 3 checks — gold_grace_exhaustion_cancellations.
- Trial 05Missed 1 of 3 checks — gold_grace_exhaustion_cancellations.
- Trial 06Missed 1 of 3 checks — gold_grace_exhaustion_cancellations.
- Trial 07Passed every verifier it was graded on.
- Trial 08Missed 1 of 3 checks — gold_grace_exhaustion_cancellations.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.