task_53Health care and social assistance
Start with all trials for auto-written insights, or select a single trial to drill into its verifier results, trajectory, and workspace changes.
Claude Opus 5claude-code · max
Mean reward
94.3%All trials
Across 4 trials on this task, reward ran from 89% to 96% — 3 of 4 cleared every core verifier.
- 12.5 / 13avg core verifiers passed
- 3.3 / 4avg secondary passed
- 84avg turns
- 47avg tool calls
- $4.58avg cost
- 22mavg time
Summary · Claude Opus 5
Claude Opus 5 passed 3 of 4 trials. Failures cluster on side_effects (3/4 trials).
Common pattern
All 1 unsuccessful trials broke on exactly the same verifiers — memo_count_consistency and bridge_outreach failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of Claude Opus 5's failures on this task were classified as Off-Scope Advice Added — 3 verifiers across 3 of 4 trials.
Per-trial notes
GPT-5.6 Solcodex · max
Mean reward
92.8%All trials
Across 4 trials on this task, reward ran from 87% to 96% — 3 of 4 cleared every core verifier.
- 12.8 / 13avg core verifiers passed
- 2.8 / 4avg secondary passed
- 42avg turns
- 29avg tool calls
- $2.67avg cost
- 17mavg time
Summary · GPT-5.6 Sol
GPT-5.6 Sol passed 3 of 4 trials. Failures cluster on side_effects (4/4 trials).
Common pattern
All 1 unsuccessful trials broke on exactly the same verifiers — routine_waitlist_edge_cases and side_effects failed in every one, and nothing else did. That is one reproducible defect, not variance.
Why they broke
Most of GPT-5.6 Sol's failures on this task were classified as Off-Scope Advice Added — 4 verifiers across 4 of 4 trials.
Per-trial notes
GPT-5.6 Terracodex · max
Mean reward
89.1%All trials
Across 4 trials on this task, reward ran from 81% to 100% — 1 of 4 cleared every core verifier.
- 11.8 / 13avg core verifiers passed
- 3.5 / 4avg secondary passed
- 36avg turns
- 26avg tool calls
- $1.03avg cost
- 15mavg time
Summary · GPT-5.6 Terra
GPT-5.6 Terra passed 1 of 4 trials. Failures cluster on unresolved_priority (2/4 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 4 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of GPT-5.6 Terra's failures on this task were classified as Required Rationale or Authority Statement Omitted — 4 verifiers across 3 of 4 trials.
Per-trial notes
Claude Sonnet 5claude-code · max
Mean reward
91.9%All trials
Across 4 trials on this task, reward ran from 87% to 97.5% — 0 of 4 cleared every core verifier.
- 12.0 / 13avg core verifiers passed
- 3.5 / 4avg secondary passed
- 137avg turns
- 66avg tool calls
- $5.19avg cost
- 26mavg time
Summary · Claude Sonnet 5
Claude Sonnet 5 passed none of its 4 trials. Failures cluster on side_effects (2/4 trials).
Common pattern
No single verifier breaks every unsuccessful trial — failures move across 5 verifiers, so what varies between trials matters as much as the model.
Why they broke
Most of Claude Sonnet 5's failures on this task were classified as Required Rationale or Authority Statement Omitted — 3 verifiers across 3 of 4 trials.
Per-trial notes
Gemini 3.6 Flashgemini-cli · high
Mean reward
75.4%All trials
Across 4 trials on this task, reward ran from 71.5% to 77.5% — 0 of 4 cleared every core verifier.
- 10.5 / 13avg core verifiers passed
- 2.0 / 4avg secondary passed
- 77avg turns
- 38avg tool calls
- $0.90avg cost
- 17mavg time
Summary · Gemini 3.6 Flash
Gemini 3.6 Flash passed none of its 4 trials. Failures cluster on source_hierarchy (4/4 trials).
Common pattern
source_hierarchy and side_effects failed in every unsuccessful trial; what else broke varies across the other 7 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.6 Flash's failures on this task were classified as Required Rationale or Authority Statement Omitted — 10 verifiers across 4 of 4 trials.
Per-trial notes
- Trial 01Missed 2 of 13 core — slot_eligibility and routine_waitlist_edge_cases, plus 2 secondary.
- Trial 02Missed 4 of 13 core — workbook_outreach, workbook_protected_contact and routine_waitlist_edge_cases and 1 more, plus 2 secondary.
- Trial 03Missed 2 of 13 core — slot_eligibility and workbook_slot_pairs, plus 2 secondary.
- Trial 04Missed 2 of 13 core — bridge_outreach and routine_waitlist_edge_cases, plus 2 secondary.
Gemini 3.1 Progemini-cli · high
Mean reward
34%All trials
Across 4 trials on this task, reward ran from 17.5% to 47% — 0 of 4 cleared every core verifier.
- 3.5 / 13avg core verifiers passed
- 2.8 / 4avg secondary passed
- 21avg turns
- 16avg tool calls
- $0.39avg cost
- 6mavg time
Summary · Gemini 3.1 Pro
Gemini 3.1 Pro passed none of its 4 trials. Failures cluster on source_hierarchy (4/4 trials).
Common pattern
source_hierarchy, unresolved_priority and bridge_outreach and 6 more failed in every unsuccessful trial; what else broke varies across the other 5 verifiers. One reproducible defect, with noise around it.
Why they broke
Most of Gemini 3.1 Pro's failures on this task were classified as Required Rationale or Authority Statement Omitted — 24 verifiers across 4 of 4 trials.
Per-trial notes
- Trial 01Missed 12 of 13 core — memo_escalation, memo_count_consistency and unresolved_priority and 9 more, plus 1 secondary.
- Trial 02Missed 9 of 13 core — memo_escalation, unresolved_priority and bridge_outreach and 6 more, plus 2 secondary.
- Trial 03Missed 8 of 13 core — unresolved_priority, bridge_outreach and slot_eligibility and 5 more, plus 1 secondary.
- Trial 04Missed 9 of 13 core — memo_escalation, unresolved_priority and bridge_outreach and 6 more, plus 1 secondary.