Spend From rule application audit for account 1178Payments & Transfers
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
89.3%All trials
Across 10 trials on this task, reward ran from 78.6% to 100% — 5 of 10 cleared every check.
- 12.5 / 14avg checks passed
- 33avg turns
- 43avg tool calls
- 4mavg time
Summary · GPT-5.5
GPT-5.5 passed 5 of 10 trials. Failures cluster on answer_goal_draw_counts (5/10 trials).
Common pattern
All 5 unsuccessful trials broke on exactly the same verifiers — answer_goal_draw_counts, deliverable_slack_message_content and trace_spend_from_editor_id_method failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 5 verifiers across 5 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Missed 3 of 14 checks — answer_goal_draw_counts, deliverable_slack_message_content and trace_spend_from_editor_id_method.
- Trial 03Missed 3 of 14 checks — answer_goal_draw_counts, deliverable_slack_message_content and trace_spend_from_editor_id_method.
- Trial 04Passed every verifier it was graded on.
- Trial 05Passed every verifier it was graded on.
- Trial 06Passed every verifier it was graded on.
- Trial 07Missed 3 of 14 checks — answer_goal_draw_counts, deliverable_slack_message_content and trace_spend_from_editor_id_method.
- Trial 08Passed every verifier it was graded on.
- Trial 09Missed 3 of 14 checks — answer_goal_draw_counts, deliverable_slack_message_content and trace_spend_from_editor_id_method.
- Trial 10Missed 3 of 14 checks — answer_goal_draw_counts, deliverable_slack_message_content and trace_spend_from_editor_id_method.
Claude Opus 4.8
Mean reward
100%All trials
Across 10 trials on this task, reward ran from 100% to 100% — 10 of 10 cleared every check.
- 14.0 / 14avg checks passed
- 21avg turns
- 26avg tool calls
- 3mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 cleared every check in all 10 trials.
Common pattern
Every trial cleared the core tier, so there is no failure pattern to read.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Passed every verifier it was graded on.
- Trial 03Passed every verifier it was graded on.
- Trial 04Passed every verifier it was graded on.
- Trial 05Passed every verifier it was graded on.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Passed every verifier it was graded on.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Gemini 3.1 Pro
Mean reward
78.6%All trials
Across 10 trials on this task, reward ran from 14.3% to 100% — 4 of 10 cleared every check.
- 11.0 / 14avg checks passed
- 81avg turns
- 79avg tool calls
- 15mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 4 of 10 trials. Failures cluster on answer_goal_draw_counts (5/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 12 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 5 verifiers across 5 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Missed 1 of 14 checks — answer_goal_draw_counts.
- Trial 03Missed 3 of 14 checks — deliverable_slack_message_content, trace_all_connectors_and_verdict and trace_rule_history_grounded_in_rules.
- Trial 04Missed 1 of 14 checks — trace_rule_history_grounded_in_rules.
- Trial 05Missed 1 of 14 checks — trace_all_connectors_and_verdict.
- Trial 06Missed 4 of 14 checks — answer_goal_draw_counts, deliverable_slack_message_content and trace_all_connectors_and_verdict and 1 more.
- Trial 07Missed 12 of 14 checks — answer_goal_draw_counts, answer_grocery_goal_subaccount_id and answer_grocery_untagged_count and 9 more.
- Trial 08Missed 4 of 14 checks — answer_goal_draw_counts, deliverable_slack_message_content and trace_rule_history_grounded_in_rules and 1 more.
- Trial 09Missed 1 of 14 checks — trace_all_connectors_and_verdict.
- Trial 10Missed 3 of 14 checks — answer_goal_draw_counts, deliverable_slack_message_content and trace_spend_from_editor_id_method.
Gemini 3.5 Flash
Mean reward
25.7%All trials
Across 10 trials on this task, reward ran from 14.3% to 35.7% — 0 of 10 cleared every check.
- 3.6 / 14avg checks passed
- 60avg turns
- 59avg tool calls
- 29mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on answer_grocery_untagged_count (10/10 trials).
Common pattern
answer_grocery_untagged_count, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 4 more failed in every unsuccessful trial; what else broke varies across the other 5 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 6 verifiers across 6 of 10 trials.
Per-trial notes
- Trial 01Missed 11 of 14 checks — answer_goal_draw_counts, answer_grocery_goal_subaccount_id and answer_grocery_untagged_count and 8 more.
- Trial 02Missed 12 of 14 checks — answer_goal_draw_counts, answer_grocery_goal_subaccount_id and answer_grocery_untagged_count and 9 more.
- Trial 03Missed 9 of 14 checks — answer_grocery_untagged_count, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 6 more.
- Trial 04Missed 10 of 14 checks — answer_goal_draw_counts, answer_grocery_untagged_count and deliverable_jira_ticket_filed and 7 more.
- Trial 05Missed 9 of 14 checks — answer_grocery_untagged_count, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 6 more.
- Trial 06Missed 11 of 14 checks — answer_goal_draw_counts, answer_grocery_untagged_count and answer_spend_from_rule_history and 8 more.
- Trial 07Missed 11 of 14 checks — answer_goal_draw_counts, answer_grocery_goal_subaccount_id and answer_grocery_untagged_count and 8 more.
- Trial 08Missed 9 of 14 checks — answer_goal_draw_counts, answer_grocery_untagged_count and deliverable_jira_ticket_filed and 6 more.
- Trial 09Missed 10 of 14 checks — answer_goal_draw_counts, answer_grocery_untagged_count and deliverable_jira_ticket_filed and 7 more.
- Trial 10Missed 12 of 14 checks — answer_goal_draw_counts, answer_grocery_goal_subaccount_id and answer_grocery_untagged_count and 9 more.