I want to choose accounting softwareBusiness and Consumer Services
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
67%
Claude Opus 5.5
56%
Claude Opus 5
44%
GPT-5.6 sol
33%
GPT-6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
16m
Claude Opus 5
4m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
69
Claude Opus 5
33
GPT-6 sol
15
Claude Opus 5.5
15
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (9) | ||||
| 0% | 0% | 0% | 0% | |
How to passUse the Canada Revenue Agency at https://www.canada.ca/en/revenue-agency.html and relevant small-business guidance to identify bookkeeping, invoicing, records, tax, and data-retention considerations that software may need to support. | ||||
| 100% | 100% | 100% | 100% | |
How to passResearch at least seven products using official sites such as https://quickbooks.intuit.com/ca/, https://www.xero.com/ca/, https://www.waveapps.com/, https://www.freshbooks.com/en-ca/, and other public providers. | ||||
| 100% | 100% | 100% | 100% | |
How to passRecord invoicing, expense capture, automatic bank feeds, bank reconciliation, reporting, tax features, project or time tracking, recurring invoices, receipt handling, and integrations. | ||||
| 100% | 100% | 100% | 100% | |
How to passVerify Canadian tax-related functionality and currency support using official documentation. Do not assume US functionality automatically applies to Canada. | ||||
| 0% | 0% | 100% | 0% | |
How to passResearch how each product handles data export, backups, account closure, and migration. Record file formats and documented limitations. | ||||
| 100% | 0% | 0% | 100% | |
How to passRecord public pricing tiers and key differences, distinguishing standard pricing from time-limited promotions or region-specific offers. | ||||
| 0% | 0% | 0% | 0% | |
How to passUse at least three reputable independent accounting or small-business publications to identify recurring usability, support, or migration concerns. | ||||
| 100% | 0% | 0% | 0% | |
How to passCompare the finalists for monthly invoicing, expense-heavy project work, and year-end record preparation. Identify which product fits each scenario best. | ||||
| 100% | 0% | 0% | 100% | |
How to passCreate a weighted scorecard, a primary choice, a low-cost alternative, and a more scalable alternative (as three distinct roles, not simply an overall 1-2-3 ranking), exact source URLs, migration considerations, and a checklist for verifying suitability with an accountant before committing to a platform. | ||||
Failure modes on this task
18 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | — | 2 | 2 | — |
| Standing up the evidence2 modes | — | — | 1 | 1 |
| Reaching the page1 mode | — | — | — | 1 |
| No signal in the judge’s words1 mode · the grader | 3 | 4 | 2 | 2 |