Onboarding DOB timezone shift auditCustomer & Account Analytics
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
GPT-5.5
Mean reward
83.1%All trials
Across 10 trials on this task, reward ran from 61.5% to 100% — 4 of 10 cleared every check.
- 10.8 / 13avg checks passed
- 47avg turns
- 60avg tool calls
- 6mavg time
Summary · GPT-5.5
GPT-5.5 passed 4 of 10 trials. Failures cluster on answer_shift_direction_split (6/10 trials).
Common pattern
answer_shift_direction_split failed in every unsuccessful trial; what else broke varies across the other 4 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of GPT-5.5's failures on this task were classified as Wrong answer — 6 verifiers across 6 of 10 trials.
Per-trial notes
- Trial 01Missed 5 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 2 more.
- Trial 02Missed 1 of 13 checks — answer_shift_direction_split.
- Trial 03Missed 1 of 13 checks — answer_shift_direction_split.
- Trial 04Passed every verifier it was graded on.
- Trial 05Missed 5 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 2 more.
- Trial 06Missed 5 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 2 more.
- Trial 07Missed 5 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 2 more.
- Trial 08Passed every verifier it was graded on.
- Trial 09Passed every verifier it was graded on.
- Trial 10Passed every verifier it was graded on.
Claude Opus 4.8
Mean reward
60.8%All trials
Across 10 trials on this task, reward ran from 53.8% to 61.5% — 0 of 10 cleared every check.
- 7.9 / 13avg checks passed
- 60avg turns
- 77avg tool calls
- 9mavg time
Summary · Claude Opus 4.8
Claude Opus 4.8 passed none of its 10 trials. Failures cluster on answer_affected_accounts (10/10 trials).
Common pattern
answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 2 more failed in every unsuccessful trial; what else broke varies across the other 1 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Claude Opus 4.8's failures on this task were classified as Wrong answer — 10 verifiers across 10 of 10 trials.
Per-trial notes
- Trial 01Missed 5 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 2 more.
- Trial 02Missed 5 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 2 more.
- Trial 03Missed 5 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 2 more.
- Trial 04Missed 5 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 2 more.
- Trial 05Missed 5 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 2 more.
- Trial 06Missed 6 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 3 more.
- Trial 07Missed 5 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 2 more.
- Trial 08Missed 5 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 2 more.
- Trial 09Missed 5 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 2 more.
- Trial 10Missed 5 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 2 more.
Gemini 3.1 Pro
Mean reward
37.7%All trials
Across 10 trials on this task, reward ran from 15.4% to 92.3% — 1 of 10 cleared every check.
- 4.9 / 13avg checks passed
- 116avg turns
- 138avg tool calls
- 76mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed 1 of 10 trials. Failures cluster on deliverable_jira_ticket_filed (9/10 trials).
Common pattern
deliverable_jira_ticket_filed failed in every unsuccessful trial; what else broke varies across the other 10 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Wrong answer — 8 verifiers across 8 of 10 trials.
Per-trial notes
- Trial 01Missed 9 of 13 checks — answer_shift_direction_split, deliverable_jira_ticket_filed and deliverable_slack_channel_exists and 6 more.
- Trial 02Missed 11 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 8 more.
- Trial 03Missed 10 of 13 checks — answer_affected_accounts, answer_shift_direction_split and deliverable_jira_ticket_filed and 7 more.
- Trial 04Missed 11 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 8 more.
- Trial 05Missed 10 of 13 checks — answer_affected_accounts, answer_shift_direction_split and deliverable_jira_ticket_filed and 7 more.
- Trial 06Missed 7 of 13 checks — deliverable_jira_ticket_filed, deliverable_slack_channel_exists and deliverable_slack_message_content and 4 more.
- Trial 07Missed 10 of 13 checks — answer_affected_accounts, answer_shift_direction_split and deliverable_jira_ticket_filed and 7 more.
- Trial 08Missed 1 of 13 checks — trace_dob_shift_method.
- Trial 09Missed 10 of 13 checks — answer_affected_accounts, answer_shift_direction_split and deliverable_jira_ticket_filed and 7 more.
- Trial 10Missed 2 of 13 checks — deliverable_jira_ticket_filed and trace_dob_shift_method.
Gemini 3.5 Flash
Mean reward
14.6%All trials
Across 10 trials on this task, reward ran from 7.7% to 15.4% — 0 of 10 cleared every check.
- 1.9 / 13avg checks passed
- 62avg turns
- 72avg tool calls
- 24mavg time
Summary · Gemini 3.5 Flash
Gemini 3.5 Flash passed none of its 10 trials. Failures cluster on answer_affected_accounts (10/10 trials).
Common pattern
answer_affected_accounts, answer_shift_direction_split and deliverable_jira_ticket_filed and 7 more failed in every unsuccessful trial; what else broke varies across the other 2 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.5 Flash's failures on this task were classified as Wrong answer — 9 verifiers across 9 of 10 trials.
Per-trial notes
- Trial 01Missed 11 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 8 more.
- Trial 02Missed 11 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 8 more.
- Trial 03Missed 11 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 8 more.
- Trial 04Missed 11 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 8 more.
- Trial 05Missed 11 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 8 more.
- Trial 06Missed 11 of 13 checks — answer_affected_accounts, answer_shift_direction_split and deliverable_jira_ticket_filed and 8 more.
- Trial 07Missed 12 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 9 more.
- Trial 08Missed 11 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 8 more.
- Trial 09Missed 11 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 8 more.
- Trial 10Missed 11 of 13 checks — answer_affected_accounts, answer_affected_population and answer_shift_direction_split and 8 more.