Billing Write-Off Pack + ACATS Transfer Compliance — Jira + Confluence + SQL + SlackCompliance & Regulatory
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
83.8%All trials
Across 10 trials on this task, reward ran from 69.2% to 92.3% — 3 of 10 cleared every check.
- 10.9 / 13avg checks passed
- 31avg turns
- 103avg tool calls
- 5mavg time
Summary · GPT-5.5
GPT-5.5 passed 3 of 10 trials. Failures cluster on floor_source_and_sql (10/10 trials).
Common pattern
floor_source_and_sql failed in every unsuccessful trial; what else broke varies across the other 5 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 13 checks — floor_source_and_sql.
- Trial 02Missed 1 of 13 checks — floor_source_and_sql.
- Trial 03Missed 2 of 13 checks — slack_value_forwarding_days and floor_source_and_sql.
- Trial 04Missed 3 of 13 checks — slack_value_billing_trigger_key, slack_value_billing_trigger_status and floor_source_and_sql.
- Trial 05Missed 3 of 13 checks — slack_value_grace_breach_count, slack_value_grace_breach_share and floor_source_and_sql.
- Trial 06Missed 4 of 13 checks — slack_value_forwarding_days, slack_value_grace_breach_count and slack_value_grace_breach_share and 1 more.
- Trial 07Missed 2 of 13 checks — slack_value_forwarding_days and floor_source_and_sql.
- Trial 08Missed 2 of 13 checks — slack_value_forwarding_days and floor_source_and_sql.
- Trial 09Missed 1 of 13 checks — floor_source_and_sql.
- Trial 10Missed 2 of 13 checks — slack_value_forwarding_days and floor_source_and_sql.
Claude Opus 4.8
Mean reward
62.3%All trials
Across 10 trials on this task, reward ran from 46.2% to 84.6% — 0 of 10 cleared every check.
- 8.1 / 13avg checks passed
- 37avg turns
- 63avg tool calls
- 5mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed none of its 10 trials. Failures cluster on slack_value_forwarding_days (10/10 trials).
Common pattern
slack_value_forwarding_days and floor_source_and_sql failed in every unsuccessful trial; what else broke varies across the other 5 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 7 of 13 checks — slack_value_billing_trigger_key, slack_value_billing_trigger_status and slack_value_forwarding_days and 4 more.
- Trial 02Missed 5 of 13 checks — slack_value_forwarding_days, slack_value_grace_breach_count and slack_value_grace_breach_share and 2 more.
- Trial 03Missed 4 of 13 checks — slack_value_billing_trigger_key, slack_value_billing_trigger_status and slack_value_forwarding_days and 1 more.
- Trial 04Missed 7 of 13 checks — slack_value_billing_trigger_key, slack_value_billing_trigger_status and slack_value_forwarding_days and 4 more.
- Trial 05Missed 4 of 13 checks — slack_value_billing_trigger_key, slack_value_billing_trigger_status and slack_value_forwarding_days and 1 more.
- Trial 06Missed 4 of 13 checks — slack_value_billing_trigger_key, slack_value_billing_trigger_status and slack_value_forwarding_days and 1 more.
- Trial 07Missed 4 of 13 checks — slack_value_billing_trigger_key, slack_value_billing_trigger_status and slack_value_forwarding_days and 1 more.
- Trial 08Missed 5 of 13 checks — slack_value_forwarding_days, slack_value_grace_breach_count and slack_value_grace_breach_share and 2 more.
- Trial 09Missed 2 of 13 checks — slack_value_forwarding_days and floor_source_and_sql.
- Trial 10Missed 7 of 13 checks — slack_value_billing_trigger_key, slack_value_billing_trigger_status and slack_value_forwarding_days and 4 more.
Gemini 3.1 Pro
Mean reward
81.5%All trials
Across 10 trials on this task, reward ran from 69.2% to 84.6% — 0 of 10 cleared every check.
- 10.6 / 13avg checks passed
- 77avg turns
- 79avg tool calls
- 12mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on slack_value_forwarding_days (10/10 trials).
Common pattern
slack_value_forwarding_days and floor_source_and_sql failed in every unsuccessful trial; what else broke varies across the other 2 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 13 checks — slack_value_forwarding_days and floor_source_and_sql.
- Trial 02Missed 2 of 13 checks — slack_value_forwarding_days and floor_source_and_sql.
- Trial 03Missed 4 of 13 checks — slack_value_billing_trigger_key, slack_value_billing_trigger_status and slack_value_forwarding_days and 1 more.
- Trial 04Missed 2 of 13 checks — slack_value_forwarding_days and floor_source_and_sql.
- Trial 05Missed 2 of 13 checks — slack_value_forwarding_days and floor_source_and_sql.
- Trial 06Missed 4 of 13 checks — slack_value_billing_trigger_key, slack_value_billing_trigger_status and slack_value_forwarding_days and 1 more.
- Trial 07Missed 2 of 13 checks — slack_value_forwarding_days and floor_source_and_sql.
- Trial 08Missed 2 of 13 checks — slack_value_forwarding_days and floor_source_and_sql.
- Trial 09Missed 2 of 13 checks — slack_value_forwarding_days and floor_source_and_sql.
- Trial 10Missed 2 of 13 checks — slack_value_forwarding_days and floor_source_and_sql.
Gemini 3.5 Flash
Mean reward
4.6%All trials
Across 10 trials on this task, reward ran from 0% to 7.7% — 0 of 10 cleared every check.
- 0.6 / 13avg checks passed
- 30avg turns
- 28avg tool calls
- 8mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on deliverable_slack_channel_exists (10/10 trials).
Common pattern
deliverable_slack_channel_exists, slack_value_billing_trigger_key and slack_value_billing_trigger_status and 9 more failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 13 of 13 checks — deliverable_slack_channel_exists, slack_value_billing_trigger_key and slack_value_billing_trigger_status and 10 more.
- Trial 02Missed 12 of 13 checks — deliverable_slack_channel_exists, slack_value_billing_trigger_key and slack_value_billing_trigger_status and 9 more.
- Trial 03Missed 12 of 13 checks — deliverable_slack_channel_exists, slack_value_billing_trigger_key and slack_value_billing_trigger_status and 9 more.
- Trial 04Missed 13 of 13 checks — deliverable_slack_channel_exists, slack_value_billing_trigger_key and slack_value_billing_trigger_status and 10 more.
- Trial 05Missed 12 of 13 checks — deliverable_slack_channel_exists, slack_value_billing_trigger_key and slack_value_billing_trigger_status and 9 more.
- Trial 06Missed 13 of 13 checks — deliverable_slack_channel_exists, slack_value_billing_trigger_key and slack_value_billing_trigger_status and 10 more.
- Trial 07Missed 12 of 13 checks — deliverable_slack_channel_exists, slack_value_billing_trigger_key and slack_value_billing_trigger_status and 9 more.
- Trial 08Missed 12 of 13 checks — deliverable_slack_channel_exists, slack_value_billing_trigger_key and slack_value_billing_trigger_status and 9 more.
- Trial 09Missed 12 of 13 checks — deliverable_slack_channel_exists, slack_value_billing_trigger_key and slack_value_billing_trigger_status and 9 more.
- Trial 10Missed 13 of 13 checks — deliverable_slack_channel_exists, slack_value_billing_trigger_key and slack_value_billing_trigger_status and 10 more.