I help run a small volunteer nonprofitBusiness and Consumer Services
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5.5
86%
GPT-6 sol
71%
GPT-5.6 sol
43%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
30m
Claude Opus 5
11m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
113
Claude Opus 5
79
Claude Opus 5.5
48
GPT-5.6 sol
32
GPT-6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (7) | ||||
| 100% | 100% | 100% | 100% | |
How to passSelect exactly three well-known email newsletter platforms that offer a publicly documented free or lowest-cost entry tier and are reasonably suitable for a small nonprofit. Name the three platforms before beginning the comparison. For each, open the official pricing page, free/lowest-tier documentation, and relevant official feature/documentation pages. Record the exact source URL, page title, access date, and any pricing or feature caveats. Do not create an account or sign up. | ||||
| 100% | 100% | 100% | 0% | |
How to passFor each platform, document the free or lowest-cost tier using the official pricing and documentation pages. Record maximum contacts/subscribers, monthly email-send allowance, number of users/seats if applicable, available templates/editor, automation restrictions, branding requirements, reporting/analytics, signup forms, imports, unsubscribe management, and any other limitation that materially affects a small nonprofit sending simple supporter updates. Record whether the limit is per month, per account, or based on contacts/subscribers. Distinguish confirmed facts from information that is not publicly stated. | ||||
| 100% | 100% | 100% | 100% | |
How to passFor each platform, trace how the subscriber limits and cost increase as a list grows, so the comparison accounts for where we will be in future rather than only where we are now. Looking at how each scales matters because our list will grow, and a platform that is generous at a small size but becomes expensive as it grows could trap us into a painful migration later, so understanding the whole curve rather than just the starting point is essential to a sensible long term choice. | ||||
| 100% | 0% | 0% | 0% | |
How to passEvaluate usability using publicly accessible official documentation, setup guides, editor documentation, help articles, tutorials, screenshots, and workflow descriptions. Assess the number and complexity of steps required for the core nonprofit workflow: importing/managing subscribers, creating a simple newsletter, editing content, sending a test, scheduling/sending an update, and managing unsubscribes. Do not claim that a platform is objectively “easy” unless the official evidence supports the conclusion. Where public evidence is insufficient, mark usability as uncertain rather than assuming it is easy or difficult. | ||||
| 100% | 100% | 100% | 0% | |
How to passEvaluate whether each platform publicly documents the basic functions required for the nonprofit: subscriber/list management, composing and editing a simple email, importing contacts, sending/scheduling newsletters, unsubscribe handling, bounce handling, basic delivery/reporting information, and account/list administration. Record evidence for each function from official documentation. Do not infer reliability merely because a feature exists; distinguish documented, partially documented, and not publicly verified. | ||||
| 100% | 100% | 100% | 100% | |
How to passCheck whether the platforms publicly mention anything relevant to nonprofits, such as any nonprofit provisions or relevant terms, recording what is found on their official pages without assuming anything not stated. Checking this matters because such provisions can meaningfully change the value of a platform for an organisation like ours, and capturing whatever the official pages genuinely say, while being careful not to overstate it, keeps the comparison both useful and accurate. | ||||
| 100% | 100% | 0% | 0% | |
How to passBring together the entry tier value, the way pricing scales, ease of use, reliability of the basics, and any nonprofit considerations, and produce a clear recommendation with reasoning and honest trade offs, since the cheapest option may be harder to use and the easiest may cost more as we grow. Making the trade offs explicit matters because the committee needs a recommendation they can understand and stand behind, and a comparison that shows its reasoning gives them something considered to decide from rather than a bare verdict. Do not create any account or sign up. | ||||
Failure modes on this task
7 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | — | — | — | 2 |
| Standing up the evidence1 mode | — | — | 1 | — |
| No signal in the judge’s words1 mode · the grader | — | 1 | 1 | 2 |