I'm looking for a 1-bedroom apartment to rentBusiness and Consumer Services
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
80%
Claude Opus 5.5
80%
GPT-6 sol
80%
Claude Opus 5
60%
GPT-5.6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
46m
GPT-5.6 sol
37m
Claude Opus 5
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
94
Claude Opus 5
66
GPT-5.6 sol
50
Claude Opus 5.5
40
GPT-6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (5) | ||||
| 100% | 100% | 100% | 100% | |
How to passSearch Zillow, Apartments.com, and Craigslist for 1-bedroom rentals in Austin, Texas, using a maximum monthly rent of $1,600, within roughly 15 minutes of Downtown Austin, with pet-friendly policies and in-unit laundry. Use Downtown Austin as the consistent reference point across all three platforms. Record up to 3 qualifying listings per site, aiming for at least 3 when available. If fewer than 3 meet all criteria, record all qualifying listings and note the shortfall. | ||||
| 100% | 100% | 0% | 100% | |
How to passOpen each individual listing page and confirm the pet policy, including any pet fee, deposit, breed or size restriction, the laundry type (in-unit versus shared or none), monthly rent, and any clearly stated lease terms. Remove any listing that fails to clearly meet a required criterion. If a required detail is unavailable or unclear, mark it as “not shown” and exclude the listing from the final qualifying set. | ||||
| 100% | 100% | 100% | 100% | |
How to passCompare the remaining listings using monthly rent, square footage, distance and estimated typical driving time from the same defined downtown reference point, and building amenities such as parking, a gym, or a pool. Use the same mapping method and destination point for all listings and distinguish straight-line distance from driving time. | ||||
| 0% | 0% | 0% | 0% | |
How to passCheck whether the same unit is listed on more than one of the three platforms, sometimes at a different price or with different fees disclosed, and note which platform offers the better deal on that same unit. Treat listings as duplicates only when the address and unit/property details indicate they are the same rental; do not assume two listings are duplicates based on property name alone. | ||||
| 100% | 100% | 100% | 100% | |
How to passChoose the top 3 qualifying listings overall and create one comparison document containing the listing URL, address, monthly rent, square footage, commute information, key amenities, pet policy, laundry type, and landlord or leasing-agent contact information when publicly listed. Do not contact any landlord or leasing agent and do not submit an application. | ||||
Failure modes on this task
5 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Reaching the page1 mode | — | — | 1 | — |
| Delivering the result1 mode | — | 1 | — | — |
| No signal in the judge’s words1 mode · the grader | 1 | — | 1 | 1 |