I am evaluating whether buying a home isBusiness and Consumer Services
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5.5
100%
GPT-6 sol
90%
GPT-5.6 sol
80%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
90m
Claude Opus 5
85m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
119
Claude Opus 5
57
Claude Opus 5.5
37
GPT-6 sol
18
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (10) | ||||
| 100% | 100% | 100% | 100% | |
How to passDefine a reusable rent-versus-buy framework that separates documented facts, assumptions, calculations, and scenario-dependent variables. Identify the jurisdiction and property market used for any jurisdiction-specific mortgage, tax, insurance, lending, or transaction-cost evidence rather than assuming universal rules. Establish measurable inputs including purchase price, rent, down payment, mortgage structure and rate, loan term, property taxes, insurance, maintenance, HOA/association costs, closing and selling costs, expected holding period, home-price growth, investment returns on otherwise-invested capital, refinancing assumptions, and affordability constraints. Identify which assumptions are mandatory inputs, which can be varied, and which are jurisdiction- or property-specific. | ||||
| 100% | 100% | 100% | 100% | |
How to passResearch realistic mortgage structures relevant to the comparison, including fixed-rate and variable/adjustable-rate structures where publicly documented. Use authoritative government, regulatory, lender, and consumer-finance sources to establish how interest rates, amortization, principal repayment, rate adjustments, fees, refinancing, prepayment provisions, and other material financing conditions work. Compare the financial consequences of the financing structures rather than merely describing them. Identify which assumptions must be included in later modeling and which terms can materially change the result. Cross-check important mechanics using independent authoritative sources where practical. | ||||
| 100% | 100% | 100% | 100% | |
How to passResearch and structure the costs associated with owning a property beyond mortgage principal and interest. Include property taxes, homeowners insurance, HOA charges and assessments where applicable, routine maintenance, major repairs, transaction and closing costs, financing fees, refinancing costs, and other documented ownership expenses that can materially affect long-term affordability. Investigate which costs are commonly omitted from simplified rent-versus-buy analyses and verify them using appropriate government, regulatory, housing, financial, or independent sources. Distinguish predictable costs from uncertain or scenario-dependent costs and record evidence for each material assumption. | ||||
| 100% | 100% | 100% | 100% | |
How to passBuild the corresponding economic model for continuing to rent, including the opportunity cost of the down payment and other upfront cash that would otherwise remain invested, differences in recurring housing costs, and the potential investment return on capital not tied up in the property. Research credible evidence or methodology for handling investment-return assumptions without presenting an uncertain return as guaranteed. Ensure the model does not compare only monthly mortgage payments with monthly rent; account for the different timing and destination of cash flows under both choices. | ||||
| 100% | 100% | 100% | 100% | |
How to passUse the framework and research from the earlier steps to construct several realistic financing and ownership scenarios. Vary meaningful factors such as down payment, mortgage structure, interest rate, home-price change, rent growth, investment return, maintenance burden, property-tax and insurance changes, HOA costs, and expected holding period. Use scenarios that expose genuine trade-offs rather than arbitrary combinations. For each scenario, document the assumptions, evidence basis, calculation method, and major uncertainties. Ensure that earlier research findings determine which variables need sensitivity testing. | ||||
| 100% | 100% | 100% | 100% | |
How to passCalculate the financial outcomes of the scenarios over multiple holding periods and identify the conditions under which buying becomes financially stronger, financially weaker, or highly assumption-dependent relative to renting. Analyze mortgage amortization, accumulated home equity, ownership costs, investment opportunity cost, rental costs, transaction costs, and estimated terminal value together rather than evaluating isolated components. Identify break-even or crossover conditions where they can be meaningfully calculated. Clearly distinguish nominal figures from assumptions about future growth or investment returns and avoid treating uncertain forecasts as facts. | ||||
| 100% | 100% | 100% | 0% | |
How to passChallenge the preliminary conclusions using finite, realistic failure scenarios. Test at least: a higher-than-expected mortgage rate or rate reset, slower home-price growth, higher maintenance or major repair costs, unexpectedly higher property taxes or insurance, significant HOA costs or assessments, a shorter-than-planned holding period that increases the impact of transaction costs, lower investment returns on the renter's retained capital, and refinancing that is unavailable or less favorable than assumed. Determine whether the preferred outcome changes under each stress case and identify which assumptions have the greatest influence on the result. Preserve uncertainty where the evidence cannot justify a precise estimate. | ||||
| 100% | 100% | 100% | 0% | |
How to passPerform a structured audit of the completed analysis. Verify the most consequential mortgage, tax, insurance, transaction-cost, maintenance, and refinancing claims against appropriate authoritative sources. Check that assumptions are not being presented as facts, that one-time and recurring costs are not mixed incorrectly, that opportunity cost is included consistently, and that equity accumulation is not incorrectly treated as a pure expense or pure profit. Specifically audit for omitted costs commonly excluded from basic rent-versus-buy calculators. Where credible sources disagree, investigate whether the difference arises from jurisdiction, loan type, property type, timing, or methodology and preserve unresolved uncertainty when necessary. | ||||
| 100% | 100% | 0% | 100% | |
How to passProduce a reusable decision model that can be applied to another property without rebuilding the methodology. Include clearly defined input fields, source-backed assumptions, financing scenarios, ownership-cost categories, rental-cost assumptions, opportunity-cost calculations, holding-period analysis, sensitivity analysis, break-even logic where appropriate, and a clear interpretation framework. Separate user-entered facts from assumptions and calculated results. Include instructions explaining how to update the model when mortgage terms, taxes, insurance, maintenance expectations, rent, investment assumptions, or expected holding period change. Preserve the evidence and methodology behind the calculations so another reviewer can reproduce the analysis. | ||||
| 100% | 100% | 100% | 100% | |
How to passPresent the completed research as a practical decision brief accompanying the reusable model. Summarize the major evidence-backed findings, the financing structures evaluated, the ownership costs that materially affect the result, the opportunity-cost treatment, the scenario outcomes, the most important sensitivity drivers, and the circumstances under which buying or renting becomes more financially attractive. Explain the strongest arguments for each side rather than assuming one is superior. Identify the key limitations and what additional property- or borrower-specific information would be needed for a real purchase decision. Finish with a concise quality review confirming that the model accounts for the major documented costs, distinguishes facts from assumptions, preserves uncertainty, tests meaningful adverse scenarios, and can be reused for a different property without changing the underlying methodology. | ||||
Failure modes on this task
3 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Standing up the evidence2 modes | — | — | 1 | 1 |
| No signal in the judge’s words1 mode · the grader | — | — | — | 1 |