I am about to rent my first apartmentLaw and Government
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
86%
Claude Opus 5.5
57%
GPT-6 sol
57%
GPT-5.6 sol
57%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
17m
Claude Opus 5
9m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
75
Claude Opus 5
20
GPT-5.6 sol
16
GPT-6 sol
14
Claude Opus 5.5
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (7) | ||||
| 100% | 100% | 100% | 100% | |
How to passNavigate to https://www.law.cornell.edu/wex/landlord-tenant_law and record the page title, the landlord-tenant topics it lists such as lease terms, repairs, and eviction, and the federal protection it names. | ||||
| 100% | 100% | 0% | 0% | |
How to passNavigate to https://www.usa.gov/housing and record the page title, the housing help topics it lists. | ||||
| 100% | 100% | 100% | 0% | |
How to passNavigate to https://www.nolo.com/legal-encyclopedia/renters-rights and record the page title, the renting categories it lists such as security deposits, repairs, and eviction, and one tenant right it describes. | ||||
| 0% | 0% | 100% | 100% | |
How to passNavigate to https://en.wikipedia.org/wiki/Landlord%E2%80%93tenant_law and record the page's definition of landlord-tenant law, the common lease terms it names, and the deposit or repair guidance it shows. | ||||
| 100% | 100% | 100% | 0% | |
How to passNavigate to https://www.amazon.com/ and find one document organizer, small fireproof safe, or renter's binder under $40. Open its product-detail page and record the product title, dimensions or capacity, current price, and star rating visibly shown there. | ||||
| 100% | 0% | 0% | 100% | |
How to passCompare the Cornell, USA.gov, Nolo, and Wikipedia guidance on deposits, leases, and repairs. Decide the single document a first-time renter should keep most carefully, such as the signed lease, move-in inspection photos, or deposit receipt, and record a one-sentence reason citing one of the sources. | ||||
| 100% | 0% | 0% | 100% | |
How to passProduce a written renter primer with the three source summaries, the common lease terms and deposit or repair guidance from Wikipedia, the selected organizer with price and rating, a rights and responsibilities checklist, a lease-review checklist, a deposit and repair workflow, and the key document verdict. | ||||
Failure modes on this task
10 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Standing up the evidence1 mode | 1 | 1 | 1 | — |
| Delivering the result1 mode | — | 1 | 1 | — |
| No signal in the judge’s words1 mode · the grader | — | 1 | 1 | 3 |