A friend in the UK has a 2-year-oldFinance
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5.5
100%
Claude Opus 5
67%
GPT-6 sol
67%
GPT-5.6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
17m
Claude Opus 5
14m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
92
Claude Opus 5.5
84
Claude Opus 5
58
GPT-5.6 sol
41
GPT-6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (6) | ||||
| 100% | 100% | 100% | 100% | |
How to passUse Petplan's current official Covered For Life product page and linked policy wording to record the available annual vet-fee cover levels, how the limit refreshes, and whether dental illness and injury are covered, including any stated conditions or limits. Record the exact URLs and mark any requested detail that is not publicly stated as "not stated" or "quote-specific". | ||||
| 100% | 100% | 100% | 100% | |
How to passUse Waggel's current official lifetime-policy page, FAQ and linked policy wording to record the available annual vet-fee cover levels, how the limit refreshes, and whether dental illness and injury are covered, including any stated conditions or sub-limits. Record the exact URLs and mark any requested detail that is not publicly stated as "not stated" or "quote-specific". | ||||
| 100% | 100% | 100% | 100% | |
How to passCompare how each insurer applies its fixed excess or claim contribution, including whether it is charged per condition, per policy year or otherwise, and any age-related percentage co-payment. Use current official Petplan FAQ or policy terms and Waggel FAQ or policy documents. If an amount is not publicly stated without a quote, record it as "quote-specific" rather than inventing it. | ||||
| 100% | 100% | 100% | 100% | |
How to passRecord the publicly stated breed and age eligibility or exclusion rules relevant to a 2-year-old medium mixed-breed dog. Use Petplan's current Covered For Life product page or policy wording and Waggel's current excluded-breeds page or policy wording. Because the constituent breeds are unknown, state whether crosses or mixes with excluded breeds are affected and identify what must be confirmed before treating the dog as eligible. | ||||
| 100% | 0% | 0% | 100% | |
How to passCreate a side-by-side table covering plan type, annual vet-fee options, dental illness conditions or sub-limits, fixed excess or claim contribution, percentage co-payment, and breed or age eligibility. Distinguish publicly confirmed terms from quote-specific or breed-dependent details and include the official URLs used. | ||||
| 100% | 0% | 0% | 100% | |
How to passGive a conditional recommendation based on the requested coverage criteria and eligibility uncertainties. Explain which trade-offs drive the recommendation and identify what the owner must confirm in an individual quote and current policy documents before purchasing. | ||||
Failure modes on this task
4 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | — | 2 | 2 | — |