Q4 Compliance Audit Review - Jira & SlackCompliance & Regulatory
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
97.9%All trials
Across 10 trials on this task, reward ran from 94.7% to 100% — 6 of 10 cleared every check.
- 18.6 / 19avg checks passed
- 10avg turns
- 36avg tool calls
- 2mavg time
Summary · GPT-5.5
GPT-5.5 passed 6 of 10 trials. Failures cluster on answer_piermont_count_and_ranking_json (4/10 trials).
Common pattern
All 4 unsuccessful trials broke on exactly the same verifiers — answer_piermont_count_and_ranking_json failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 4 verifiers across 4 of 10 trials.
Per-trial notes
- Trial 01Missed 1 of 19 checks — answer_piermont_count_and_ranking_json.
- Trial 02Passed every verifier it was graded on.
- Trial 03Passed every verifier it was graded on.
- Trial 04Missed 1 of 19 checks — answer_piermont_count_and_ranking_json.
- Trial 05Missed 1 of 19 checks — answer_piermont_count_and_ranking_json.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Missed 1 of 19 checks — answer_piermont_count_and_ranking_json.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Claude Opus 4.8
Mean reward
100%All trials
Across 10 trials on this task, reward ran from 100% to 100% — 10 of 10 cleared every check.
- 19.0 / 19avg checks passed
- 13avg turns
- 36avg tool calls
- 3mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 cleared every check in all 10 trials.
Common pattern
Every trial cleared the core tier, so there is no failure pattern to read.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Passed every verifier it was graded on.
- Trial 03Passed every verifier it was graded on.
- Trial 04Passed every verifier it was graded on.
- Trial 05Passed every verifier it was graded on.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Passed every verifier it was graded on.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Gemini 3.1 Pro
Mean reward
100%All trials
Across 10 trials on this task, reward ran from 100% to 100% — 10 of 10 cleared every check.
- 19.0 / 19avg checks passed
- 12avg turns
- 41avg tool calls
- 2mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro cleared every check in all 10 trials.
Common pattern
Every trial cleared the core tier, so there is no failure pattern to read.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Passed every verifier it was graded on.
- Trial 03Passed every verifier it was graded on.
- Trial 04Passed every verifier it was graded on.
- Trial 05Passed every verifier it was graded on.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Passed every verifier it was graded on.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Gemini 3.5 Flash
Mean reward
73.2%All trials
Across 10 trials on this task, reward ran from 52.6% to 100% — 1 of 10 cleared every check.
- 13.9 / 19avg checks passed
- 45avg turns
- 57avg tool calls
- 15mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed 1 of 10 trials. Failures cluster on answer_piermont_count_and_ranking_json (8/10 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 11 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 6 verifiers across 6 of 10 trials.
Per-trial notes
- Trial 01Missed 7 of 19 checks.
- Trial 02Missed 1 of 19 checks — answer_piermont_count_and_ranking_json.
- Trial 03Missed 3 of 19 checks — posted_slack_message, slack_message_in_database and slack_message_to_correct_channel.
- Trial 04Missed 1 of 19 checks — answer_piermont_count_and_ranking_json.
- Trial 05Missed 7 of 19 checks.
- Trial 06Missed 5 of 19 checks — answer_piermont_count_and_ranking_json, posted_slack_message and slack_message_in_database and 2 more.
- Trial 07Missed 9 of 19 checks — answer_audit_items_json, answer_in_progress_items_json and answer_piermont_count_and_ranking_json and 6 more.
- Trial 08Missed 9 of 19 checks.
- Trial 09Passed every verifier it was graded on.
- Trial 10Missed 9 of 19 checks — answer_audit_items_json, answer_in_progress_items_json and answer_piermont_count_and_ranking_json and 6 more.