I'm helping a small residential property management companyHome and Garden
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5.5
100%
GPT-6 sol
100%
GPT-5.6 sol
50%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
87m
Claude Opus 5
20m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
117
Claude Opus 5
29
Claude Opus 5.5
29
GPT-6 sol
29
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (10) | ||||
| 100% | 100% | 100% | 100% | |
How to passState that the framework helps triage and route maintenance requests, not perform repairs or make legal habitability determinations. Build it around general habitability and maintenance-response principles rather than one jurisdiction's specific code, and flag where local law would set hard response-time requirements the company should confirm for its own location. | ||||
| 100% | 100% | 100% | 100% | |
How to passCreate structured working notes with sections for a source register, urgency-classification table, dispatch-decision rules, open questions, and scenario tests. Use these notes throughout the remaining steps so the final framework can be assembled consistently rather than reconstructed from memory. | ||||
| 100% | 100% | 100% | 100% | |
How to passResearch guidance on habitability standards, emergency-repair response-time expectations, and established property-management best practices for maintenance request handling and vendor dispatch. Record each principle and its source in the working notes' source register, distinguishing formal legal requirements from industry best practice, and flagging where requirements are jurisdiction-specific rather than universal. | ||||
| 100% | 100% | 100% | 100% | |
How to passDefine the major request categories (no heat, water leak, electrical issue, pest infestation, appliance failure, cosmetic repair) and, for each, determine the appropriate urgency tier (emergency, urgent, routine) and realistic response-time target. Explain why a water leak and a cosmetic repair require fundamentally different response timelines. Build the urgency-classification table in the working notes now, one row per request type. | ||||
| 100% | 100% | 100% | 100% | |
How to passCreate a structured intake process that captures the tenant's description of the issue, unit and location details, and an initial urgency classification. For each request type, define what evidence must be requested or gathered (photos, description of symptoms, access instructions), from whom, and within what reasonable timeframe, including rules for a tenant who is unreachable for follow-up or an issue reported outside business hours. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a consistent method for routing each urgency tier to the appropriate vendor or technician, including which requests can wait for a scheduled visit versus require immediate dispatch. Define how to handle a request that could plausibly affect multiple units in the same building, and what happens when the preferred vendor is unavailable and the response-time target is at risk. | ||||
| 100% | 100% | 100% | 0% | |
How to passDefine what must be communicated to the tenant at each stage (acknowledgment, expected response window, technician visit confirmation, resolution) in plain, respectful language. Specify what must be logged for every request regardless of outcome (urgency classification, evidence gathered, vendor assigned, resolution date) so decisions remain defensible and consistent if reviewed later. | ||||
| 100% | 100% | 100% | 0% | |
How to passRun the framework against at least five difficult scenarios and record each outcome in the scratchpad: a tenant who reports a request as an emergency that turns out on inspection to be routine, conflicting information between the tenant's description and a technician's on-site assessment, a request that could affect multiple units in the same building, a case where the assigned vendor is unavailable and the response-time target is at risk, and a tenant who has filed multiple requests in a short period. For each scenario, document what the framework recommends and whether any rule needs refinement. | ||||
| 100% | 100% | 100% | 0% | |
How to passStep back and review the entire framework as a skeptical external auditor would. Check for leading or suggestive intake questions, urgency tiers that are too broad or too narrow, a dispatch workload that isn't realistic for a small property management team, inconsistent application of the urgency-classification rules across similar cases, and documentation gaps that would make a decision hard to defend later. Also check for the opposite failure: rules so cautious that routine requests get treated as emergencies and overwhelm the vendor schedule. Record every issue found and how it was fixed in the scratchpad. | ||||
| 100% | 100% | 100% | 0% | |
How to passAssemble the scratchpad's source register, urgency-classification table, dispatch-decision rules log, and scenario-test log into one polished, reusable document a property management company could hand to a new staff member: intake steps, urgency classification by request type, dispatch and vendor-routing rules, the communication process, and the fixes identified during the adversarial audit. Note any point where the framework depends on jurisdiction-specific habitability law the company would need to confirm locally. | ||||
Failure modes on this task
5 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | — | — | — | 2 |
| No signal in the judge’s words1 mode · the grader | — | — | — | 3 |