Bill underfunded-alert conformance auditSubscriptions & Billing
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
71.7%All trials
Across 10 trials on this task, reward ran from 58.3% to 91.7% — 3 of 10 cleared every check.
- 8.6 / 12avg checks passed
- 29avg turns
- 38avg tool calls
- 5mavg time
Summary · GPT-5.5
GPT-5.5 passed 3 of 10 trials. Failures cluster on answer_spec_underfunded_population (7/10 trials).
Common pattern
answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_slack_message_content and 1 more failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 7 verifiers across 7 of 10 trials.
Per-trial notes
- Trial 01Missed 4 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_slack_message_content and 1 more.
- Trial 02Missed 1 of 12 checks — trace_date_gated_inflow_method.
- Trial 03Missed 5 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_slack_message_content and 2 more.
- Trial 04Missed 5 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_slack_message_content and 2 more.
- Trial 05Missed 5 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_slack_message_content and 2 more.
- Trial 06Missed 4 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_slack_message_content and 1 more.
- Trial 07Missed 1 of 12 checks — trace_date_gated_inflow_method.
- Trial 08Missed 1 of 12 checks — trace_date_gated_inflow_method.
- Trial 09Missed 4 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_slack_message_content and 1 more.
- Trial 10Missed 4 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_slack_message_content and 1 more.
Claude Opus 4.8
Mean reward
64.2%All trials
Across 10 trials on this task, reward ran from 58.3% to 91.7% — 1 of 10 cleared every check.
- 7.7 / 12avg checks passed
- 24avg turns
- 37avg tool calls
- 6mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed 1 of 10 trials. Failures cluster on answer_spec_underfunded_population (9/10 trials).
Common pattern
answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_slack_message_content and 1 more failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 5 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_slack_message_content and 2 more.
- Trial 02Missed 4 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_slack_message_content and 1 more.
- Trial 03Missed 4 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_slack_message_content and 1 more.
- Trial 04Missed 5 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_slack_message_content and 2 more.
- Trial 05Missed 5 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_slack_message_content and 2 more.
- Trial 06Missed 5 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_slack_message_content and 2 more.
- Trial 07Missed 5 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_slack_message_content and 2 more.
- Trial 08Missed 5 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_slack_message_content and 2 more.
- Trial 09Missed 4 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_slack_message_content and 1 more.
- Trial 10Missed 1 of 12 checks — trace_date_gated_inflow_method.
Gemini 3.1 Pro
Mean reward
36.7%All trials
Across 10 trials on this task, reward ran from 16.7% to 66.7% — 0 of 10 cleared every check.
- 4.4 / 12avg checks passed
- 75avg turns
- 102avg tool calls
- 14mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 10 trials. Failures cluster on trace_date_gated_inflow_method (10/10 trials).
Common pattern
trace_date_gated_inflow_method failed in every unsuccessful trial; what else broke varies across the other 9 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Missed 7 of 12 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 4 more.
- Trial 02Missed 4 of 12 checks — deliverable_jira_ticket_filed, floor_all_connectors and floor_write_back and 1 more.
- Trial 03Missed 10 of 12 checks — answer_false_funded_blast_radius, answer_spec_underfunded_population and answer_total_spec_shortfall_cents and 7 more.
- Trial 04Missed 10 of 12 checks — answer_false_funded_blast_radius, answer_spec_underfunded_population and answer_total_spec_shortfall_cents and 7 more.
- Trial 05Missed 9 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_jira_ticket_filed and 6 more.
- Trial 06Missed 9 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_jira_ticket_filed and 6 more.
- Trial 07Missed 5 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_slack_message_content and 2 more.
- Trial 08Missed 9 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_jira_ticket_filed and 6 more.
- Trial 09Missed 9 of 12 checks — answer_spec_underfunded_population, answer_total_spec_shortfall_cents and deliverable_jira_ticket_filed and 6 more.
- Trial 10Missed 4 of 12 checks — deliverable_jira_ticket_filed, floor_all_connectors and floor_write_back and 1 more.
Gemini 3.5 Flash
Mean reward
16.7%All trials
Across 10 trials on this task, reward ran from 16.7% to 16.7% — 0 of 10 cleared every check.
- 2.0 / 12avg checks passed
- 69avg turns
- 67avg tool calls
- 26mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on answer_false_funded_blast_radius (10/10 trials).
Common pattern
All 10 unsuccessful trials broke on exactly the same verifiers — answer_false_funded_blast_radius, answer_spec_underfunded_population and answer_total_spec_shortfall_cents and 7 more failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 10 of 12 checks — answer_false_funded_blast_radius, answer_spec_underfunded_population and answer_total_spec_shortfall_cents and 7 more.
- Trial 02Missed 10 of 12 checks — answer_false_funded_blast_radius, answer_spec_underfunded_population and answer_total_spec_shortfall_cents and 7 more.
- Trial 03Missed 10 of 12 checks — answer_false_funded_blast_radius, answer_spec_underfunded_population and answer_total_spec_shortfall_cents and 7 more.
- Trial 04Missed 10 of 12 checks — answer_false_funded_blast_radius, answer_spec_underfunded_population and answer_total_spec_shortfall_cents and 7 more.
- Trial 05Missed 10 of 12 checks — answer_false_funded_blast_radius, answer_spec_underfunded_population and answer_total_spec_shortfall_cents and 7 more.
- Trial 06Missed 10 of 12 checks — answer_false_funded_blast_radius, answer_spec_underfunded_population and answer_total_spec_shortfall_cents and 7 more.
- Trial 07Missed 10 of 12 checks — answer_false_funded_blast_radius, answer_spec_underfunded_population and answer_total_spec_shortfall_cents and 7 more.
- Trial 08Missed 10 of 12 checks — answer_false_funded_blast_radius, answer_spec_underfunded_population and answer_total_spec_shortfall_cents and 7 more.
- Trial 09Missed 10 of 12 checks — answer_false_funded_blast_radius, answer_spec_underfunded_population and answer_total_spec_shortfall_cents and 7 more.
- Trial 10Missed 10 of 12 checks — answer_false_funded_blast_radius, answer_spec_underfunded_population and answer_total_spec_shortfall_cents and 7 more.