Grace Nakamura runs a small home-based bakeryComputers Electronics and Technology
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
88%
Claude Opus 5.5
75%
Claude Opus 5
25%
GPT-6 sol
25%
GPT-5.6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
11m
GPT-5.6 sol
4m
Claude Opus 5
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
22
GPT-6 sol
20
GPT-5.6 sol
19
Claude Opus 5.5
18
Claude Opus 5
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (8) | ||||
| 100% | 100% | 100% | 100% | |
How to passConfirm her use cases (social media graphics, simple logo, printable menus), skill level (no design background), and budget ($15/month). | ||||
| 100% | 0% | 0% | 100% | |
How to passGo to https://www.adobe.com/creativecloud.html, record relevant plan pricing (e.g., a single-app plan) and note the learning curve for a beginner. | ||||
| 100% | 100% | 0% | 100% | |
How to passGo to https://www.canva.com/, record Pro subscription pricing and confirm template availability for social posts, logos, and menus. | ||||
| 100% | 0% | 100% | 100% | |
How to passGo to https://affinity.serif.com/, record Affinity’s current pricing or access model, and note whether it is beginner-friendly or more suited to experienced designers. | ||||
| 100% | 0% | 0% | 100% | |
How to passFor each option, confirm the specific availability of bakery/food-business-relevant templates (menus, social posts). | ||||
| 100% | 0% | 0% | 0% | |
How to passLook up beginner reviews or tutorials to gauge how steep the learning curve is for each tool. | ||||
| 0% | 0% | 0% | 0% | |
How to passConfirm each tool can export print-ready files (for menus) and correctly sized social media graphics. | ||||
| 100% | 0% | 0% | 100% | |
How to passProduce a table with columns: Tool, Cost, Beginner-friendliness, Template availability, Print export quality — then state the top pick with reasoning. | ||||
Failure modes on this task
15 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Standing up the evidence1 mode | 1 | 1 | 1 | — |
| Delivering the result1 mode | — | 1 | — | — |
| No signal in the judge’s words1 mode · the grader | — | 4 | 5 | 2 |