Create a comprehensive financial planning and investing r…Finance
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
93%
Claude Opus 5.5
80%
Claude Opus 5
27%
GPT-6 sol
20%
GPT-5.6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
43m
Claude Opus 5
7m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
101
Claude Opus 5
19
Claude Opus 5.5
17
GPT-6 sol
13
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (15) | ||||
| 100% | 100% | 0% | 100% | |
How to passEstablish a generic profile for the research without using a specific individual's personal information. Assume: A person has recently started earning a regular salary. They have little or no previous investing experience. They want to begin investing early. They have ordinary living expenses. Their financial goals may include building wealth, purchasing a home, supporting family, and retirement. Their exact income, expenses, dependents, debt, and risk tolerance are not predetermined. Explain why the financial plan must be based on cash flow, financial obligations, goals, time horizon, and risk tolerance, rather than simply choosing investments first. Create a basic financial hierarchy: Income Essential expenses Emergency reserve Insurance/risk protection High-cost debt Short-term goals Long-term investing Retirement planning Periodic review Clearly explain that this is a framework for research and decision-making, not a universal percentage allocation. | ||||
| 100% | 0% | 0% | 100% | |
How to passResearch how a new earner can structure their first few months of income. Create a framework for dividing income into categories such as: Essential expenses Lifestyle/discretionary expenses Emergency savings Insurance Debt repayment Short-term goals Long-term investments Retirement Create several illustrative scenarios rather than declaring one allocation universally correct. For example: Scenario A — No debt and low expenses Scenario B — Significant education/personal debt Scenario C — Family responsibilities Scenario D — High living costs Explain how the allocation should change between these scenarios. Do not use unrealistic assumptions or imply that a person should invest aggressively regardless of their financial obligations. | ||||
| 100% | 0% | 0% | 100% | |
How to passUse RBI and other authoritative financial-education sources to research emergency savings. Determine: Why an emergency fund exists. What expenses should be considered essential. How liquidity affects where emergency money should be held. How employment stability affects the required reserve. How dependents and debt can change the requirement. How to build the reserve gradually. Compare suitable categories of liquid savings vehicles without presenting a particular bank or product as universally best. Create an Emergency Fund Builder showing: Starting point. Target reserve. Monthly contribution. Conditions under which the target should be increased. Situations in which the emergency reserve can legitimately be used. How to replenish it after withdrawal. | ||||
| 100% | 0% | 0% | 100% | |
How to passResearch the role of insurance using https://irdai.gov.in/ and other authoritative sources. Cover: Health insurance. Life insurance. Personal accident coverage. Employer-provided insurance. Family/dependent considerations. Explain the difference between: Insurance for protection and Investment products marketed as insurance/investment combinations. Research the basic concepts of: Sum insured. Premium. Deductible. Waiting period. Exclusions. Claim process. Nominee. Policy tenure. Coverage limitations. Do not recommend a specific insurance company or policy without a separately defined comparison task. Create a Protection Checklist identifying what a new earner should investigate before committing a large portion of income to investments. | ||||
| 100% | 0% | 0% | 0% | |
How to passCreate a debt-assessment framework. Research: Credit-card debt. Personal loans. Education loans. Vehicle loans. Home loans. Other consumer debt. For each category, explain: Why interest cost matters. How repayment priority can affect investing decisions. When maintaining investments while repaying debt might make sense. Why high-cost debt generally deserves special attention. Create a simple Debt Decision Tree: No debt → manageable debt → high-cost debt → financial distress The purpose is to help the reader determine what information they need before deciding how aggressively to invest. Do not claim that investing is always better than repaying debt or vice versa. | ||||
| 100% | 0% | 0% | 100% | |
How to passCreate a goal-based framework. Separate goals into: Short term Examples: Emergency reserve. Near-term purchases. Travel. Professional certifications. Medium term Examples: Vehicle. Higher education. Home down payment. Marriage/family-related goals. Long term Examples: Retirement. Financial independence. Long-term wealth creation. For each time horizon, research how the following factors change: Acceptable volatility. Liquidity requirement. Investment duration. Capital-preservation requirement. Appropriate asset categories. The goal is to demonstrate why the same investment strategy should not automatically be used for every financial goal. | ||||
| 100% | 0% | 0% | 0% | |
How to passUsing SEBI, RBI, PFRDA, AMFI, and other authoritative sources, create a comparative overview of major investment categories. Include, where relevant: Savings/deposit products. Fixed deposits. Recurring deposits. Government securities. Bonds. Mutual funds. Exchange-traded funds. Direct equities. Gold. REITs/InvITs. NPS. Other relevant regulated investment categories. For every category, record: Factor Information What it is Basic explanation Main purpose Typical financial use Risk Relative risk characteristics Liquidity Ease of accessing money Time horizon Suitable horizon to investigate Return characteristics Fixed/market-linked/etc. Diversification Whether diversification is built in Costs Major cost considerations Tax Tax treatment to verify Complexity Beginner/intermediate/advanced Key risks Major risks to understand Do not rank the instruments as “best” without considering the investor's objective. | ||||
| 0% | 100% | 0% | 100% | |
How to passUse https://investor.sebi.gov.in/ and https://www.amfiindia.com/investor to research mutual funds. Explain: What a mutual fund is. Equity, debt, hybrid, and other major categories. Direct vs regular plans. Expense ratio. NAV. Riskometer. SIP. Lump-sum investing. Exit load where applicable. Portfolio diversification. Active vs passive approaches. Index funds and ETFs. Tracking error where relevant. AMFI describes SIP as a method of investing a fixed amount periodically rather than making a single lump-sum investment, while SEBI explains the distinction between direct and regular mutual-fund plans and their different cost structures. Create a Mutual Fund Evaluation Checklist containing the questions a beginner should ask before selecting a scheme. Do not simply produce a list of currently popular mutual funds. | ||||
| 100% | 0% | 0% | 0% | |
How to passUse SEBI investor resources to explain what changes when someone moves from diversified investment products to selecting individual companies. Cover: Equity ownership. Market volatility. Business risk. Valuation. Diversification. Concentration risk. Fundamental research. Investor behavior. Long-term vs short-term speculation. Clearly distinguish: Long-term equity investing from short-term trading/speculation. Create a Beginner Stock-Research Checklist covering the minimum information that should be researched before considering an individual company. Do not recommend specific stocks. | ||||
| 100% | 100% | 100% | 100% | |
How to passUse https://www.pfrda.org.in/ and official government sources to investigate retirement planning. Compare the role of: Employer retirement benefits. EPF where applicable. NPS. Long-term mutual-fund/equity investing. Other eligible retirement-oriented instruments. For NPS, research: How the system works. Eligibility. Contributions. Investment choices. Withdrawal/exit framework. Costs. Tax-related considerations. Long-term limitations. Do not assume NPS is suitable for everyone; present it as one retirement-planning option that needs to be evaluated. | ||||
| 100% | 100% | 100% | 100% | |
How to passUse the https://www.incometax.gov.in/ and official financial-product sources. Research the tax considerations associated with major investment categories. Cover, where applicable: Interest income. Dividend income. Capital gains. Equity investments. Mutual funds. Fixed-income investments. NPS. Tax-saving investment products. Investment-related deductions/exemptions where currently applicable. Do not hard-code tax rates, thresholds, or rules that may become outdated. Instead, instruct the researcher to verify the current financial-year rules from the Income Tax Department before making an investment decision. Also record major investment costs such as: Expense ratios. Brokerage. Transaction charges. Exit loads. Account/platform charges. Taxes and statutory charges. | ||||
| 100% | 0% | 100% | 100% | |
How to passResearch investor-protection resources from SEBI, AMFI, RBI, and other official bodies. Create a Red Flag List covering: Guaranteed-return claims. Unregulated investment schemes. Unverified social-media advice. “Hot stock” recommendations. Excessive trading. Concentrating all money in one asset. Investing emergency money in volatile assets. Ignoring debt. Ignoring insurance. Chasing past returns. Selecting funds solely because they recently performed well. Panic-selling during market declines. Borrowing money to invest. Sharing sensitive financial credentials. Fake investment advisers or fraudulent platforms. Explain how an investor can verify whether a financial intermediary or entity is appropriately regulated where applicable. | ||||
| 100% | 0% | 0% | 100% | |
How to passCombine all research into a practical timeline. Create: Month 1 — Financial foundation Understand take-home income. Track expenses. Identify debt. Review employer benefits. Begin emergency savings. Research insurance requirements. Months 2–3 — Stabilization Establish a sustainable budget. Continue emergency-fund building. Address high-cost debt. Define financial goals. Learn basic investment concepts. Months 4–6 — Begin structured investing Select investment categories based on goals. Establish an investment process. Consider systematic investing where appropriate. Keep short-term money separate from long-term investments. Months 7–12 — Review and improve Review savings rate. Review investment behavior. Increase contributions when income rises. Reassess insurance and emergency reserves. Review portfolio diversification and goal alignment. The timeline should be treated as an example framework, not a mandatory sequence for every individual. | ||||
| 100% | 0% | 0% | 100% | |
How to passBuild a decision tree that starts with: “I have received my salary.” Then ask: Are essential expenses covered? Do I have an emergency reserve? Do I have high-cost debt? Do I have appropriate insurance/protection? What is this money's goal? When will I need the money? How much volatility can the goal tolerate? Which investment categories fit the time horizon? What are the costs and taxes? Is the product regulated and understandable? Can I continue the investment consistently? When should I review the decision? The decision tree should lead to researching an appropriate category, not automatically to a specific investment product. | ||||
| 100% | 0% | 0% | 100% | |
How to passDeliver a comprehensive First-Earning Financial Playbook containing: First-salary financial framework. Budget and cash-flow worksheet. Emergency-fund framework. Insurance checklist. Debt-assessment framework. Financial-goal planner. Investment-category comparison. Mutual-fund/SIP research guide. Direct-equity research guide. Retirement-planning section. Tax and cost verification checklist. Investment-scam/red-flag checklist. First-year financial roadmap. Investment decision tree. Monthly financial-review checklist. Official sources used for verification. The final document should clearly distinguish between: Facts verified from regulators/government sources General financial-planning principles Illustrative examples Items that depend on an individual's circumstances Rules/tax information that must be rechecked before acting Do not provide a one-size-fits-all portfolio, promise returns, recommend a specific stock as a guaranteed investment, or imply that investing should begin before essential financial protection is established. | ||||
Failure modes on this task
27 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | — | 8 | 10 | 1 |
| Standing up the evidence1 mode | — | — | — | 2 |
| No signal in the judge’s words1 mode · the grader | 1 | 3 | 2 | — |