A colleague's family is formalizing their annual charitableCommunity and Society
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
86%
GPT-5.6 sol
29%
Claude Opus 5.5
14%
Claude Opus 5
0%
GPT-6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
51m
Claude Opus 5
38m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
117
Claude Opus 5
100
Claude Opus 5.5
47
GPT-5.6 sol
2
GPT-6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (7) | ||||
| 100% | 0% | 0% | 0% | |
How to passBrowse Charity Navigator's ocean/environment cause category (charitynavigator.org) to build a 15–20 organization longlist spanning both large and small/regional groups. | ||||
| 100% | 0% | 100% | 0% | |
How to passprojects.propublica.org/nonprofits pull the most recent Form 990 for each candidate; record program expense ratio and check top leadership compensation against total budget. | ||||
| 0% | 0% | 100% | 100% | |
How to passapps.irs.gov/app/eos confirm current 501(c)(3) status is active (not revoked) and filings are current for each candidate. | ||||
| 0% | 0% | 100% | 0% | |
How to passSearch each organization's name + "investigation" / "controversy" / "lawsuit"; review coverage from the last 2 years and note anything unresolved. | ||||
| 0% | 0% | 100% | 0% | |
How to passConfirm Charity Navigator star rating and note the visible Candid/GuideStar Seal tier (Bronze/Silver/Gold/Platinum) where publicly displayed. If the profile view limit is reached, note that the seal tier couldn't be verified for that organization rather than guessing. | ||||
| 0% | 0% | 100% | 0% | |
How to passNarrow to the final list meeting the size-mix requirement; propose the tiered $50,000 split with a short reasoning line per organization. | ||||
| 0% | 0% | 100% | 0% | |
How to passBuild the tracker with the five tabs specified, using a spreadsheet tool that doesn't require login (such as CryptPad), fully populated, plus the final allocation with reasoning. Leave the sheet open for review. | ||||
Failure modes on this task
19 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | 1 | 2 | — | 3 |
| Reaching the page1 mode | — | — | — | 1 |
| Standing up the evidence1 mode | — | — | — | 1 |
| No signal in the judge’s words1 mode · the grader | 4 | 5 | 1 | 1 |