task_53Health care and social assistance
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one trial to measure.
Core metrics
Core Pass %
Mean share of the core verifiers passing, averaged over runs. N = 4 trials per model.
98%
GPT-5.6 Solmax
96%
Claude Opus 5max
92%
Claude Sonnet 5max
90%
GPT-5.6 Terramax
81%
Gemini 3.6 Flashhigh
27%
Gemini 3.1 Prohigh
Verifier Pass %
Mean fraction of all verifiers passing, across trials. N = 4 trials per model.
93%
Claude Opus 5max
91%
GPT-5.6 Solmax
91%
Claude Sonnet 5max
90%
GPT-5.6 Terramax
74%
Gemini 3.6 Flashhigh
37%
Gemini 3.1 Prohigh
Avg Cost
Mean API spend across trials; the line under a value is its observed range. N = 4 trials per model.
$5.19$4.10–$6.79
Claude Sonnet 5max
$4.58$3.08–$5.31
Claude Opus 5max
$2.67$1.79–$3.07
GPT-5.6 Solmax
$1.03$0.87–$1.19
GPT-5.6 Terramax
$0.90$0.45–$1.50
Gemini 3.6 Flashhigh
$0.39$0.28–$0.51
Gemini 3.1 Prohigh
Extended metrics
Secondary Pass %
Mean fraction of the secondary verifiers passing. N = 4 trials per model.
88%
GPT-5.6 Terramax
88%
Claude Sonnet 5max
81%
Claude Opus 5max
69%
GPT-5.6 Solmax
69%
Gemini 3.1 Prohigh
50%
Gemini 3.6 Flashhigh
Median Time
Wall-clock minutes, median across trials. N = 4 trials per model.
25m
Claude Sonnet 5max
22m
Claude Opus 5max
18m
GPT-5.6 Solmax
17m
Gemini 3.6 Flashhigh
15m
GPT-5.6 Terramax
6m
Gemini 3.1 Prohigh
Tool Calls
Tool invocations, median across trials — counted from the trajectories. N = 4 trials per model.
61
Claude Sonnet 5max
50
Claude Opus 5max
32
Gemini 3.6 Flashhigh
30
GPT-5.6 Solmax
23
GPT-5.6 Terramax
16
Gemini 3.1 Prohigh
What the reward cost on this task
Reward vs Cost
Mean reward against mean spend per trial — top-left is more reward for less.
- Claude Opus 5
- GPT-5.6 Sol
- GPT-5.6 Terra
- Claude Sonnet 5
- Gemini 3.6 Flash
- Gemini 3.1 Pro
$0.39Cost / trial$5.19
Tokens Consumed vs Reward
Mean reward against cumulative input tokens per trial — not peak context.
- Claude Opus 5
- GPT-5.6 Sol
- GPT-5.6 Terra
- Claude Sonnet 5
- Gemini 3.6 Flash
- Gemini 3.1 Pro
377kInput tokens / trial9.3M
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5maxn=4 | GPT-5.6 Solmaxn=4 | GPT-5.6 Terramaxn=4 | Claude Sonnet 5maxn=4 | Gemini 3.6 Flashhighn=4 | Gemini 3.1 Prohighn=4 |
|---|---|---|---|---|---|---|
| core (13) | ||||||
| 100% | 100% | 100% | 100% | 100% | 25% | |
Why it mattersThe escalation queue is the most urgent operational outcome because crisis transfer and same-day review must precede ordinary scheduling. How to passChecks whether the memo presents the correct ranked escalation queue and crisis handling. | ||||||
| 75% | 100% | 100% | 100% | 100% | 75% | |
Why it mattersIncorrect headline counts can mislead the urgent-access lead about queue size and capacity pressure even when some client-level rows are correct. How to passChecks whether the memo's aggregate pathway counts are internally consistent with the detailed routing decisions. | ||||||
| 100% | 100% | 50% | 100% | 100% | 0% | |
Why it mattersUnresolved priority cases require follow-up capacity action and must not be hidden by ineligible appointments. How to passChecks whether the memo identifies the exact unresolved priority clients and why no eligible slot can be reserved. | ||||||
| 75% | 100% | 75% | 75% | 75% | 0% | |
Why it mattersBridge outreach is a safety-sensitive operational requirement for priority clients whose first clinical contact is delayed or unavailable. How to passChecks whether the memo applies the bridge-outreach rule to the correct clients. | ||||||
| 100% | 100% | 100% | 75% | 50% | 0% | |
Why it mattersSlot reservations are the primary capacity outcome, and the task tests whether the model respects specialty, payer, language, modality, and consent limits. How to passChecks whether the memo summarizes the correct high-acuity reservation set, total minutes, and excluded-capacity caveats without duplicating the workbook client-slot mapping verifier. | ||||||
| 100% | 100% | 100% | 100% | 100% | 100% | |
Why it mattersThe operations workbook must reconcile every referral exactly once, not only clients with reserved appointments. How to passChecks whether the workbook provides complete client-level coverage for the full batch. | ||||||
| 100% | 100% | 100% | 100% | 100% | 75% | |
Why it mattersClient-level pathway accuracy is the core operational reconciliation outcome across escalation, priority access, and routine/waitlist handling. How to passChecks whether the workbook records the correct routing pathway for every client. | ||||||
| 100% | 100% | 100% | 100% | 100% | 0% | |
Why it mattersOperations staff need a time-ordered reservation list and correct minute total to hold scarce access capacity safely. How to passChecks whether the workbook lists the exact high-acuity slot set and total minutes, with appointment-start ordering treated as an advisory presentation expectation rather than a standalone failure condition. | ||||||
| 100% | 100% | 100% | 100% | 75% | 0% | |
Why it mattersCorrect client-to-slot mapping is necessary because a right-looking slot list is still unsafe if assigned to the wrong client or program. How to passChecks whether the workbook-specific reservation table maps reserved slots to the correct clients and carries the key operational fields. | ||||||
| 100% | 100% | 100% | 100% | 75% | 75% | |
Why it mattersOperations staff need to distinguish clients with scheduled first clinical contacts from those requiring interim outreach or follow-up capacity action. How to passChecks whether the workbook uses client-level flags/tabs consistently for unresolved-priority and bridge-outreach operations, including avoiding routine/waitlist contradictions. | ||||||
| 100% | 100% | 100% | 100% | 75% | 0% | |
Why it mattersThe workbook is likely to drive follow-up calls, so an incorrect protected-contact flag can cause improper outreach or missed consent safeguards. How to passChecks whether the workbook consistently records protected-contact restrictions for the correct clients across protected-contact and client-level routing fields. | ||||||
| 100% | 75% | 50% | 75% | 25% | 0% | |
Why it mattersThese edge cases were a critical missing verifier dimension because each one looks superficially urgent unless the handoff correction and policy exclusion are applied. How to passChecks whether the workbook correctly resolves the four routine/waitlist edge cases. | ||||||
| 100% | 100% | 100% | 75% | 75% | 0% | |
Why it mattersThe memo is an operational handoff for the urgent-access lead, so omitting protected-contact restrictions could lead to inappropriate collateral outreach even if the workbook contains the correct flags. How to passChecks whether the memo carries forward the client-specific protected-contact restrictions required for outreach. | ||||||
| secondary (4) | ||||||
| 100% | 75% | 100% | 100% | 0% | 0% | |
Why it mattersThe source-hierarchy rule prevents unfinalized care-coordination text from overriding the referral export. How to passChecks whether the memo applies the signed-handoff hierarchy and rejects the unsigned draft note. | ||||||
| 100% | 100% | 100% | 100% | 100% | 100% | |
Why it mattersThe deliverable is for operational access reconciliation, so diagnoses, medication advice, and external treatment recommendations are outside scope. How to passChecks whether the memo stays within access-routing scope rather than giving clinical treatment advice. | ||||||
| 100% | 100% | 100% | 100% | 100% | 75% | |
Why it mattersA strong operations workbook should preserve exclusion reasons so staff do not accidentally schedule into ineligible capacity after the reconciliation. How to passChecks whether the workbook documents why tempting capacity was not usable for unresolved clients. | ||||||
| 25% | 0% | 50% | 50% | 0% | 100% | |
Why it mattersThe task asks for two reconciliation outputs only, so extra artifacts, policy changes, or capacity releases create unsafe ambiguity and violate the revised deliverable requirement. How to passChecks whether the memo avoids materially unsolicited operational side effects and extra artifact instructions. | ||||||
Failure modes on this task
85 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5max | GPT-5.6 Solmax | GPT-5.6 Terramax | Claude Sonnet 5max | Gemini 3.6 Flashhigh | Gemini 3.1 Prohigh |
|---|---|---|---|---|---|---|
| Presentation and Communication2 modes | 3 | 6 | 6 | 5 | 14 | 24 |
| Reasoning and Computation3 modes | 1 | — | 1 | — | 3 | 9 |
| Deliverable Completion and Structure1 mode | — | — | — | 1 | 1 | 10 |
| Grader Error1 mode · the grader | 1 | — | — | — | — | — |