I'm renting a flat in the UKFinance
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
80%
Claude Opus 5.5
80%
GPT-5.6 sol
60%
GPT-6 sol
60%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
22m
GPT-5.6 sol
10m
Claude Opus 5
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
37
Claude Opus 5
31
GPT-6 sol
27
Claude Opus 5.5
23
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (5) | ||||
| 100% | 100% | 100% | 0% | |
How to passOn https://www.moneyhelper.org.uk/en/everyday-money/insurance/contents-insurance-if-you-rent-and-have-a-low-income, record how coverage responsibility splits between tenant and landlord, plus the page's advice on comparing quotes. | ||||
| 100% | 100% | 100% | 100% | |
How to passOn https://www.aviva.co.uk/insurance/home-products/contents-insurance/, record at least three specific, named coverage limits exactly as published, such as cover for alternative accommodation, home office equipment, or garden contents. | ||||
| 0% | 0% | 100% | 100% | |
How to passOn the Aviva contents insurance page, confirm whether an exact premium price is publicly displayed. If no exact premium is shown, record that the price requires starting Aviva's quote process. Do not start or submit the quote. | ||||
| 100% | 100% | 100% | 0% | |
How to passCross-check the MoneyHelper and Aviva pages specifically. Confirm that contents insurance covers the tenant's belongings/contents rather than the building structure, and record how each source describes the landlord's responsibility for the building. | ||||
| 100% | 0% | 0% | 100% | |
How to passState directly: what a typical policy actually covers by named limit, and whether a real price is published or requires a live quote. | ||||
Failure modes on this task
6 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Reaching the page1 mode | — | — | — | 2 |
| Standing up the evidence1 mode | — | — | 1 | — |
| No signal in the judge’s words1 mode · the grader | 1 | 2 | — | — |