A homeowner wants to stop making rushed decisionsHome and Garden
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5.5
100%
GPT-5.6 sol
7%
GPT-6 sol
0%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
105m
Claude Opus 5
14m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
115
Claude Opus 5
24
GPT-5.6 sol
16
Claude Opus 5.5
14
GPT-6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (15) | ||||
| 100% | 0% | 100% | 0% | |
How to passStart by defining what the homeowner is trying to accomplish with this system and what the system should deliberately avoid doing. Establish a common vocabulary for symptoms, diagnosis, maintenance, repair, replacement, monitoring, temporary measures, professional inspection, and emergency escalation. Research authoritative housing, building-safety, and consumer-protection sources to identify reliable principles for deciding when professional assistance is appropriate. Make a clear distinction between observing a problem and diagnosing its cause. For example, water staining, unusual noise, poor cooling, or a repeatedly tripping electrical device may be symptoms rather than diagnoses. Establish rules for situations where a homeowner can safely gather information and situations where they should stop investigating and contact an appropriately qualified professional. Because building regulations and safety requirements differ by location, identify which conclusions can be general and which must be verified locally. The framework should become a decision-support system, not a collection of DIY repair instructions. A stranger should understand immediately how to use the system without interpreting it as permission to perform hazardous work. | ||||
| 100% | 0% | 100% | 0% | |
How to passResearch government housing agencies, building authorities, consumer-protection organizations, safety regulators, manufacturers, recognized standards bodies, and professional trade organizations. Determine which sources should be consulted for different questions, such as safety, maintenance intervals, equipment specifications, building requirements, installation practices, and consumer rights. Create an evidence hierarchy that separates official requirements, manufacturer documentation, professional guidance, contractor claims, consumer advice, and informal online discussions. Explain why a manufacturer manual may be more appropriate for equipment-specific maintenance than a generic home-improvement article, while a local building authority may be more relevant to regulatory requirements. Establish a process for recording the source, applicable product or system, jurisdiction, date or version when available, and the exact issue supported. Include a procedure for handling inaccessible documentation, conflicting guidance, discontinued equipment, or advice that cannot be independently verified. The objective is to prevent the homeowner from making expensive decisions based on whichever search result appears first. The source system should remain useful even when websites, product models, and contractors change. | ||||
| 100% | 0% | 100% | 0% | |
How to passCreate a standardized way to document a household problem before seeking a repair recommendation. Include the observed symptom, location, frequency, timing, recent changes, affected equipment or building component, visible evidence, previous repairs, relevant manuals, and any known maintenance history. Explain why the homeowner should document observations without prematurely deciding what caused them. Develop a method for distinguishing direct observations from assumptions. For example, “water appeared on the ceiling after heavy rain” is different from “the roof is leaking.” Include appropriate photographs, measurements, labels, model numbers, error messages, dates, and other non-invasive information where useful and safe. Establish a rule for recording uncertainty rather than filling missing information with guesses. Create a fictional example where an initial symptom could plausibly have several causes and show how the intake process helps narrow the possibilities. The resulting intake record should be useful to the homeowner and to a professional who later receives the case. It should reduce repeated explanations and make contractor diagnosis more evidence-based without pretending that remote observations can replace an on-site inspection. | ||||
| 100% | 0% | 100% | 0% | |
How to passDevelop a method for deciding how urgently a household problem should be addressed. Research authoritative safety guidance covering issues such as electrical hazards, gas or combustion concerns, structural instability, significant water intrusion, indoor environmental hazards, fire risks, and other situations where delay could increase danger or damage. Do not give detailed instructions for performing hazardous repairs. Instead, establish categories such as immediate professional/emergency attention, prompt professional assessment, routine repair, maintenance follow-up, and monitor-and-document. Explain why the severity of the symptom alone may not determine urgency. Include fictional examples where a visually minor problem could have significant consequences and another where an alarming-looking cosmetic issue may not justify emergency action. Establish what evidence should be documented before escalation and what information should be communicated to the professional. Clearly identify when the homeowner should stop investigating. The system should prioritize safety and prevent the decision framework from encouraging unnecessary experimentation. The result should be a practical triage method that can be reused across plumbing, electrical, appliances, building envelope, HVAC, and other household systems. | ||||
| 100% | 0% | 100% | 0% | |
How to passDesign a method for identifying and verifying the correct manufacturer information for household equipment before making maintenance or replacement decisions. Cover appliances, heating and cooling systems, water-related equipment, electrical devices, windows, doors, and other products where model-specific documentation may matter. Explain how model numbers, serial numbers, installation dates, manuals, service bulletins, warranty documents, and manufacturer support pages can help distinguish one product from another. Establish a process for avoiding generic instructions when the manufacturer provides more specific information. Include a fictional situation where two visually similar products have different maintenance requirements. Explain how the reviewer should handle discontinued models, missing manuals, unclear model numbers, or third-party replacement components. Distinguish manufacturer recommendations from legally required inspections or local code requirements. The system should also prevent a contractor's statement about a product from being accepted as fact without verification when authoritative documentation is available. The goal is not to teach the homeowner to service complex equipment personally, but to make sure that repair and replacement decisions are based on the correct equipment information. | ||||
| 100% | 0% | 100% | 0% | |
How to passCreate a decision framework for choosing among four broad responses: repair now, replace, continue monitoring, or investigate further before deciding. Compare the approaches using evidence such as age and condition, recurring failures, safety concerns, manufacturer support, repairability, compatibility, remaining useful life, consequences of failure, and uncertainty about the underlying problem. Do not rely on arbitrary rules such as replacing an item after a particular number of years. Instead, explain how evidence and context should influence the decision. Include fictional cases where repair appears attractive but the diagnosis is uncertain, replacement seems convenient but the existing equipment may still be viable, and monitoring is reasonable because the consequence of failure is low. Make clear that a low-cost repair is not automatically the best choice if it leaves the underlying cause unresolved. Conversely, avoid treating replacement as automatically superior because it is newer. Create a decision record showing the evidence considered, unresolved questions, chosen action, and reason for rejecting alternatives. This should help the homeowner explain and revisit the decision later. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a process for deciding what information should be verified before hiring a contractor or specialist. Research authoritative consumer-protection agencies, licensing authorities, trade organizations, and relevant regulatory bodies for general guidance on qualifications, licensing where applicable, insurance, references, written scopes, warranties, permits, and dispute procedures. Because requirements vary by jurisdiction, identify which items must be checked locally instead of presenting them as universal requirements. Establish a distinction between verifying that a contractor is appropriately qualified and assuming that qualification guarantees good workmanship. Create questions that help determine whether the contractor has understood the actual problem rather than simply proposing a familiar repair. Include a fictional example where two contractors give substantially different explanations and explain what additional evidence should be requested before choosing between them. Avoid turning the system into a ranking of contractors or recommending a specific company. The result should give a homeowner a consistent method for gathering enough information to make a responsible hiring decision while recognizing that some technical questions require independent professional evaluation. | ||||
| 100% | 0% | 100% | 0% | |
How to passDesign a structured way to compare contractor proposals without focusing only on the final price. Identify important elements such as diagnosis, materials or equipment specified, work included, work excluded, preparation, removal, disposal, testing, cleanup, permits where applicable, warranties, assumptions, follow-up work, and conditions that could change the scope. Explain why two proposals can have different prices because they are actually offering different levels of work. Create a method for identifying vague phrases such as “repair as needed” or “replace damaged material” and turning them into clarification questions. Include a fictional comparison between two proposals where one is cheaper but leaves important parts of the problem unresolved. Establish how the homeowner should document unanswered questions before accepting a proposal. Do not attempt to generate current market prices. Instead, focus on scope completeness and comparability. The final method should help a stranger determine whether proposals are genuinely comparable and whether an apparently inexpensive quote may simply omit work that another contractor included. | ||||
| 100% | 0% | 100% | 0% | |
How to passCreate a practical documentation process for recording the condition of a household component before, during, and after professional work. Research authoritative guidance where relevant on maintenance records, warranties, equipment documentation, and property records. Identify useful evidence such as photographs, model numbers, invoices, written scopes, inspection findings, test results supplied by professionals, permits where applicable, warranty information, and replaced-component details. Establish what information should be retained permanently, what can be retained for a defined period, and what is unnecessary. Consider privacy and security when storing photographs or documents containing personal information. Include a fictional case where a recurring problem returns months after a repair and show how good documentation could help determine what was previously diagnosed and changed. Create a standardized repair record that distinguishes what the homeowner observed, what the professional reported, what work was authorized, and what was actually completed. The goal is to create useful history rather than paperwork for its own sake. A future contractor should be able to understand the property's repair history without relying entirely on memory. | ||||
| 100% | 0% | 100% | 0% | |
How to passDevelop a method for recognizing when repeated repairs indicate that the underlying problem may not have been correctly identified. Research authoritative maintenance and diagnostic principles relevant to recurring household failures. Establish indicators such as the same symptom returning, temporary improvement followed by recurrence, multiple components failing together, unexplained secondary damage, or repeated replacement of parts without resolution. Avoid assuming that recurrence automatically proves a contractor made a mistake; identify alternative explanations such as new contributing conditions, multiple independent failures, or incomplete information. Create a root-cause review process that records the original symptom, previous diagnosis, work performed, outcome, new evidence, and questions for the next professional assessment. Include fictional cases involving recurring moisture, repeated appliance faults, persistent temperature problems, and repeated drainage issues. Compare how the framework handles each case. Establish when the homeowner should seek a second opinion or specialist assessment. The system should help prevent a cycle in which the same visible symptom is repeatedly treated without examining why it keeps returning. | ||||
| 100% | 0% | 100% | 0% | |
How to passCreate a reusable method for prioritizing routine household maintenance without producing an overwhelming list of chores. Research authoritative manufacturer, housing, safety, and building-maintenance guidance. Categorize maintenance according to consequences of failure, likelihood of deterioration, ease of verification, cost of neglect, and dependency on specialist inspection. Avoid blindly copying manufacturer intervals or generic checklists; instead, establish a process for verifying whether a recommendation actually applies to the specific equipment or building component. Include fictional examples where a low-effort maintenance action prevents a larger problem and another where an apparently minor task is less important than addressing an existing unresolved defect. Create a priority system that separates safety-critical, damage-prevention, performance-related, cosmetic, and optional improvements. Include a method for recording completed work and outstanding concerns. The framework should remain flexible enough to adapt to different homes, equipment, climates, construction types, and occupant needs. The objective is to help the homeowner spend attention where it has the greatest practical value rather than treating every maintenance task as equally urgent. | ||||
| 100% | 0% | 100% | 0% | |
How to passTest the complete framework using several fictional cases where professionals disagree about the cause or recommended solution. Include situations involving water intrusion, HVAC performance, appliance failure, window or door problems, and an electrical or safety-related concern. For each case, identify what the homeowner can verify independently, what requires professional inspection, what evidence supports each proposed diagnosis, and what additional information would reduce uncertainty. Include one case where the more expensive proposal is justified by stronger evidence and another where the cheaper approach is reasonable because the additional work is not supported. Include a situation where both contractors may be partially correct because the property has more than one underlying issue. Examine whether the framework unfairly favors the first diagnosis received or the contractor who provides the most confident explanation. Record the decision path and identify where escalation is appropriate. Use the results to refine the diagnostic and proposal-comparison procedures. The objective is to make the system robust against uncertainty and conflicting professional opinions rather than simply choosing whichever recommendation sounds most convincing. | ||||
| 100% | 0% | 100% | 0% | |
How to passReview the system as if you were trying to find ways it could produce a poor repair decision. Look for assumptions such as “the cheapest quote is best,” “newer always means better,” “a visible symptom identifies the cause,” “a contractor's diagnosis is automatically correct,” or “a manufacturer recommendation applies to every model.” Check whether the framework could encourage unnecessary work, delay an urgent issue, overlook a safety concern, or create false confidence from incomplete evidence. Examine whether the documentation requirements are realistic enough that a homeowner would actually maintain them. Check whether local regulations, permits, qualifications, warranties, and manufacturer requirements are clearly separated from general advice. Identify points where the system depends on information that may become unavailable. For every weakness, propose a practical safeguard and explain why it is proportionate. Include at least one test where the framework should conclude “insufficient evidence — obtain professional diagnosis” rather than forcing a decision. The audit should make the system more reliable without turning routine household maintenance into an unnecessarily complicated administrative process. | ||||
| 100% | 0% | 100% | 0% | |
How to passTurn the complete framework into a practical system that a homeowner can introduce gradually. Define how to start with the property's most important systems, how to organize existing manuals and repair records, how to document current unresolved problems, and how to establish future maintenance records. Create reusable templates for problem intake, evidence collection, professional questions, contractor comparison, proposal review, repair records, recurring-problem analysis, and maintenance priorities. Establish event-based triggers for reviewing a system, such as repeated failure, major renovation, equipment replacement, significant water damage, change in household use, discovery of missing documentation, or conflicting professional advice. Avoid requiring a fixed annual schedule for every decision. Explain which records should follow the property or equipment and which can be discarded after their useful purpose ends. Include a lightweight process for reviewing whether the framework itself is working. The roadmap should be realistic for someone who has limited time and does not want to become a building expert. The objective is a sustainable decision process, not an enormous home-management bureaucracy. | ||||
| 100% | 0% | 100% | 0% | |
How to passCombine the research, source hierarchy, diagnostic workflow, risk triage, manufacturer verification, repair-versus-replacement framework, contractor qualification process, proposal comparison method, documentation system, recurring-problem analysis, maintenance prioritization, scenario testing, and adversarial audit into one polished handbook. Include concise decision trees, checklists, templates, and worked fictional examples where they genuinely improve usability. Clearly distinguish homeowner observations, professional findings, manufacturer information, regulatory requirements, assumptions, and unresolved questions. Mark location-specific requirements instead of presenting them as universal rules. Include an explicit escalation section covering situations where the homeowner should stop investigating and seek qualified professional or emergency assistance. The handbook should help a stranger take a new household problem, document what is known, identify what is uncertain, locate appropriate authoritative information, determine the next safe step, compare professional proposals, document the completed work, and decide how the issue should be monitored afterward. It should also provide a repeatable method for learning from recurring failures and maintaining a useful property history. The final guide should remain practical even when contractors, equipment models, manufacturers, and household circumstances change. | ||||
Failure modes on this task
29 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | — | 12 | — | 13 |
| Reaching the page1 mode | — | — | — | 1 |
| Standing up the evidence1 mode | — | — | — | 1 |
| No signal in the judge’s words1 mode · the grader | — | 2 | — | — |