1099-INT team-scoping gap vs annual-aggregate ruleCompliance & Regulatory
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
82.5%All trials
Across 10 trials on this task, reward ran from 25% to 100% — 7 of 10 cleared every check.
- 3.3 / 4avg checks passed
- 19avg turns
- 31avg tool calls
- 3mavg time
Summary · GPT-5.5
GPT-5.5 passed 7 of 10 trials. Failures cluster on cp1_team_definition_account_count (3/10 trials).
Common pattern
cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 3 verifiers across 3 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 02Passed every verifier it was graded on.
- Trial 03Missed 3 of 4 checks — cp1_team_definition_account_count, cp2_accounts_missed_by_team_definition and process_floor.
- Trial 04Passed every verifier it was graded on.
- Trial 05Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 06Passed every verifier it was graded on.
- Trial 07Passed every verifier it was graded on.
- Trial 08Passed every verifier it was graded on.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Claude Opus 4.8
Mean reward
50%All trials
Across 10 trials on this task, reward ran from 50% to 50% — 0 of 10 cleared every check.
- 2.0 / 4avg checks passed
- 17avg turns
- 25avg tool calls
- 2mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed none of its 10 trials. Failures cluster on cp1_team_definition_account_count (10/10 trials).
Common pattern
All 10 unsuccessful trials broke on exactly the same verifiers — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 02Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 03Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 04Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 05Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 06Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 07Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 08Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 09Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 10Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
Gemini 3.1 Pro
Mean reward
72.5%All trials
Across 10 trials on this task, reward ran from 25% to 100% — 5 of 10 cleared every check.
- 2.9 / 4avg checks passed
- 33avg turns
- 50avg tool calls
- 16mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 5 of 10 trials. Failures cluster on cp1_team_definition_account_count (5/10 trials).
Common pattern
cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 5 verifiers across 5 of 10 trials.
Per-trial notes
- Trial 01Passed every verifier it was graded on.
- Trial 02Passed every verifier it was graded on.
- Trial 03Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 04Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 05Passed every verifier it was graded on.
- Trial 06Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 07Passed every verifier it was graded on.
- Trial 08Passed every verifier it was graded on.
- Trial 09Missed 3 of 4 checks — cp1_team_definition_account_count, cp2_accounts_missed_by_team_definition and process_floor.
- Trial 10Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
Gemini 3.5 Flash
Mean reward
45%All trials
Across 10 trials on this task, reward ran from 25% to 50% — 0 of 10 cleared every check.
- 1.8 / 4avg checks passed
- 73avg turns
- 71avg tool calls
- 17mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on cp1_team_definition_account_count (10/10 trials).
Common pattern
cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 6 verifiers across 6 of 10 trials.
Per-trial notes
- Trial 01Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 02Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 03Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 04Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 05Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 06Missed 3 of 4 checks — cp1_team_definition_account_count, cp2_accounts_missed_by_team_definition and process_floor.
- Trial 07Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 08Missed 3 of 4 checks — cp1_team_definition_account_count, cp2_accounts_missed_by_team_definition and process_floor.
- Trial 09Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.
- Trial 10Missed 2 of 4 checks — cp1_team_definition_account_count and cp2_accounts_missed_by_team_definition.