My family's health insurance needs sorting properly IFinance
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
GPT-5.6 sol
88%
GPT-6 sol
50%
Claude Opus 5.5
38%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
46m
Claude Opus 5
9m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
118
Claude Opus 5
100
Claude Opus 5.5
68
GPT-6 sol
16
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (8) | ||||
| 0% | 100% | 100% | 0% | |
How to passSet up the spreadsheet around the family-health-insurance criteria that materially affect coverage and practical value. Include columns for insurer, exact product and variant, policy type, family-floater structure, eligible family composition where stated, sum-insured options, hospitalization coverage, room-rent and ICU eligibility or caps, co-payments, deductibles, treatment/procedure sub-limits, restoration or reinstatement rules where applicable, cashless-network information, pre-existing disease waiting period, initial waiting period, specific disease/procedure waiting periods, major exclusions, relevant eligibility conditions, premium evidence, source documents, research date, and verification status. Separate headline product-page information from fine-print policy-document evidence so that a summary claim cannot silently override the actual policy wording. Output: Create the reusable spreadsheet columns and a short data dictionary defining each field. | ||||
| 100% | 0% | 100% | 0% | |
How to passIdentify at least six currently marketed family health-insurance or family-floater policies from multiple IRDAI-regulated insurers in India using insurers’ official public websites. Add each exact product and relevant plan variant as a row in the spreadsheet. For every candidate, record the insurer, exact policy/product name, relevant variant, official product page, current policy wording or equivalent official document where publicly available, and research date. Do not include a product merely because it appears in an aggregator or comparison website if the current policy cannot be verified on the insurer’s own site. Output: Build a traceable candidate set covering multiple insurers rather than multiple variants from only one insurer. | ||||
| 100% | 100% | 100% | 100% | |
How to passFor every shortlisted policy, use the current official policy wording, customer information sheet, prospectus, schedule template, or equivalent insurer document where available to record the core comparison fields: sum-insured options, hospitalization scope, room-rent and ICU limits or eligibility, restoration/reinstatement benefits where applicable, cashless-network information, pre-existing disease waiting period, initial waiting period, specific disease or procedure waiting periods, and major exclusions. Use the insurer’s product page as a secondary summary source, but do not rely on a marketing page alone for a fine-print term when a policy document is publicly available. If the product page and policy document materially differ, record the discrepancy and the document version/date rather than silently selecting the more favorable wording. Mark unavailable or ambiguous information as “not verified” rather than inferring a term. | ||||
| 100% | 100% | 100% | 100% | |
How to passFor each policy, inspect the official current policy documents for less-obvious cost-sharing and coverage restrictions. Record, where applicable: co-payment clauses; voluntary or mandatory deductibles; disease/procedure sub-limits; room-category restrictions and any associated proportionate deductions where documented; waiting periods for named conditions or procedures; maternity or newborn restrictions where relevant to the product; non-payable or excluded expense categories; restoration/reinstatement conditions and limitations; claim-notification or pre-authorization requirements that materially affect use; and other material limitations that could make two headline-similar policies behave differently. Distinguish standard exclusions from product-specific exclusions and clearly cite the exact official document/page supporting each material restriction. Output: Add a Fine Print / Material Restrictions section to the comparison rather than reducing the policy to headline benefits. | ||||
| 100% | 100% | 100% | 0% | |
How to passDo not declare a sum insured or policy universally adequate for “a family.” Create an editable Family Profile & Assumptions section containing the information needed for a real adequacy assessment, such as family composition and ages, city, desired sum insured, existing employer or other health cover, budget, and major coverage priorities. Leave real-user fields blank where information has not been supplied. For the research comparison, define one clearly hypothetical reference family profile solely to standardize premium illustrations where public insurer quote tools permit comparison without account creation or private application data. Record the assumed ages, city, family composition, sum insured, policy tenure, and any other quote-relevant assumption. Add fields showing: sum insured being compared; whether room-rent restrictions could materially reduce usable coverage; relevant sub-limits or co-payments; restoration rules; publicly shown premium for the standardized profile where obtainable; and whether the quoted premium is directly comparable across policies. Do not treat a headline cashless-network count as proof of useful access. Record the insurer’s stated network size where available and include a separate Local Network Check Required field for the real user’s city and preferred hospitals. If a premium or local-network comparison cannot be completed publicly and consistently, preserve that gap instead of inventing a value judgment. | ||||
| 0% | 100% | 100% | 100% | |
How to passBuild a comparison method that helps surface strengths and weaknesses without pretending that one numeric score can establish personalized insurance suitability. First define any mandatory gates, such as the desired minimum sum insured or unacceptable room-rent restriction, but leave those gates editable because the real family requirements have not been supplied. Then compare the policies using transparent criteria aligned with the stated priorities: sum-insured flexibility and usable hospitalization coverage; room-rent/ICU restrictions; cashless-network evidence and need for local verification; waiting periods; major exclusions and sub-limits; co-payments/deductibles; restoration conditions; premium/value evidence where genuinely comparable; and evidence completeness. If weights are used, state them explicitly and make them editable. Do not treat missing information as a favorable score or automatically as the worst score; preserve it as uncertainty. Keep Evidence Confidence / Completeness separate from Policy Fit. Output: Create a strengths/weaknesses or weighted-fit view that remains auditable and can be recalculated when the real family profile is entered. | ||||
| 0% | 100% | 100% | 0% | |
How to passStructure the spreadsheet so new policies can be added and compared as the research widens, rather than being a fixed one time snapshot. Building for reuse matters because insurance research unfolds over time as more options surface, and a spreadsheet that keeps extending stays useful throughout rather than dating the moment another policy worth considering appears. | ||||
| 0% | 100% | 100% | 0% | |
How to passUsing the completed spreadsheet, identify the policies that appear strongest under the clearly stated hypothetical comparison assumptions, not as personalized recommendations for the actual family. For each leading policy, record concise reasoning covering sum-insured flexibility, room-rent or hospitalization restrictions, cashless-network evidence, waiting periods, major exclusions/sub-limits, restoration or cost-sharing terms, premium evidence where comparable, and important uncertainties. Include: Why It Stands Out Main Limitation Evidence Gap What Real Family Information Could Change This Result If critical information is missing or premiums are not comparable, allow an outcome such as Requires Further Verification rather than forcing a winner. Finish with a checklist of real-family details that must be entered before using the spreadsheet for an actual policy decision. This is research and comparison only; do not buy, apply for, or pay for a policy. | ||||
Failure modes on this task
10 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | 4 | — | — | 4 |
| Standing up the evidence1 mode | — | — | — | 1 |
| No signal in the judge’s words1 mode · the grader | — | 1 | — | — |