I may rent an ensuite room in EghamBusiness and Consumer Services
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5.5
100%
Claude Opus 5
40%
GPT-6 sol
20%
GPT-5.6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
22m
Claude Opus 5
15m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
106
Claude Opus 5
75
Claude Opus 5.5
32
GPT-6 sol
29
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (5) | ||||
| 100% | 100% | 100% | 100% | |
How to passIdentify the English private-renting regime current when the task runs and the scope of each official guide. Use the current GOV.UK tenant guidance, deposit-protection and permitted-payment pages, Royal Holloway private-housing and Students’ Union advice, and the council serving Egham at that time. Record access dates and conflicts, and use withdrawn guidance only if an agreement’s start date makes it applicable. | ||||
| 100% | 100% | 0% | 100% | |
How to passFor each worked offer, determine the likely agreement category and state which contract facts would confirm or overturn it. Compare exclusive possession, shared or joint liability, landlord or provider status, notice, renewal, repairs, deposit or advance-payment treatment and access to redress. List the documents that should be available before signing, including written terms, landlord identity, deposit information, gas and electrical safety evidence, alarms and energy certificate where applicable. Flag exclusions or special rules for purpose-built student accommodation. | ||||
| 100% | 0% | 0% | 100% | |
How to passCalculate each example’s stated total rent, contract length and known upfront cash. Test the holding and tenancy payments against the current caps only where those rules apply. Create separate all-student and mixed-household council-tax branches using the current local authority’s guidance. Compare included and excluded bills, guarantor exposure, joint-and-several liability and early-exit risk. Do not invent missing utility costs; provide transparent formula cells, the evidence needed to populate them and a sensitivity table for user-entered monthly bills. | ||||
| 100% | 0% | 0% | 100% | |
How to passCreate staged checklists for before paying, before signing, move-in day, the first week and move-out. Include identity and ownership checks, fee and deposit receipts, contract questions, safety certificates, inventory wording, timestamped room and common-area photos, meter readings, keys, existing damage, repair reporting, deposit-scheme confirmation and forwarding details. Define filenames and an evidence register without creating accounts or uploading files. Add red flags that require pausing and seeking Royal Holloway Students’ Union, council or professional advice. | ||||
| 100% | 0% | 0% | 100% | |
How to passDeliver a reusable tracker containing the source register, arrangement matrix, cost formulas, council-tax branches, document checklist, five-stage action list, evidence register and red-flag log. Give a conditional ranking, identify every unanswered fact that could change it, and provide a short decision tree for whether to pay, seek clarification or walk away. Include live links, access dates, assumptions and successor-source substitutions. | ||||
Failure modes on this task
7 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | — | 2 | 3 | — |
| Standing up the evidence1 mode | — | 1 | 1 | — |