A community sustainability group is trying to understandCommunity and Society
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5
90%
Claude Opus 5.5
90%
GPT-5.6 sol
70%
GPT-6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
33m
Claude Opus 5
7m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
113
Claude Opus 5
100
Claude Opus 5.5
57
GPT-6 sol
27
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (10) | ||||
| 100% | 0% | 100% | 100% | |
How to passEstablish the evaluation framework before comparing services. Define the difference between a programme being publicly promoted, publicly funded, publicly accessible, eligible to ordinary residents, and practically usable. Establish the criteria that determine whether a resident can realistically use a service, including eligibility, membership, booking, equipment access, expertise, safety requirements, fees, opening arrangements, accepted items, accessibility, training, and supervision. Also define how official information and participant experience should be treated differently in the evidence base. | ||||
| 100% | 100% | 100% | 100% | |
How to passResearch a diverse set of repair-and-reuse services across different regions, including repair cafés, tool libraries, community workshops, electronics-repair programmes, bicycle workshops, and comparable services. Use official programme, municipal, library, community-organisation, or operator pages to establish what each service actually offers and how residents are expected to access it. Identify meaningful differences in operating models rather than simply collecting examples, and retain enough variation to test whether the evaluation framework works across different types of service. | ||||
| 100% | 100% | 100% | 100% | |
How to passFor the strongest examples, verify the practical requirements for using each service. Investigate eligibility, residency requirements, membership, booking, waiting lists, fees, opening hours, accepted items, equipment availability, tool access, supervision, volunteer or staff expertise, safety rules, training, accessibility provisions, and any restrictions that could prevent an otherwise interested resident from using the service. Prefer official sources for these factual conditions and record explicitly when a detail is not stated. | ||||
| 100% | 100% | 100% | 100% | |
How to passExamine the practical scope of each service rather than relying on broad descriptions such as "repair anything" or "learn to fix it yourself." Determine which categories of items are accepted, what equipment is available, what repairs users are expected to perform themselves, what requires volunteer or staff expertise, and what is excluded. Where possible, identify whether limitations are caused by equipment, safety, expertise, capacity, booking arrangements, or programme policy. | ||||
| 100% | 100% | 100% | 100% | |
How to passSearch publicly accessible resident, participant, volunteer, community, and user discussions for practical experiences with the shortlisted services. Look specifically for recurring evidence concerning waiting lists, booking problems, unavailable equipment, insufficient expertise, repairs being refused, overcrowding, accessibility barriers, unclear fees, limited opening arrangements, or differences between advertised and experienced service levels. Record the context and date of relevant reports, distinguish individual anecdotes from recurring patterns, and do not treat review volume or popularity as proof of quality. | ||||
| 100% | 100% | 100% | 100% | |
How to passExamine how selected programmes describe their sustainability, waste-reduction, reuse, community, or skills-development goals and compare those claims with whatever public evidence exists about actual participation, repair activity, reuse outcomes, training, or community use. Do not assume that a sustainability claim is false merely because outcome data is absent. Instead, distinguish documented outcomes, stated intentions, indirect evidence, and evidence gaps, and identify where the public record is insufficient to judge effectiveness. | ||||
| 100% | 100% | 100% | 100% | |
How to passInvestigate the most meaningful conflicts between official information and user experience found in the research. Determine whether the apparent conflict results from outdated information, site-specific conditions, capacity limits, temporary circumstances, differences between eligibility and actual availability, or genuinely inconsistent service. Give greater weight to authoritative sources for formal rules and to appropriately contextualised participant evidence for lived experience. Preserve uncertainty where the available evidence cannot establish the reason for a discrepancy. | ||||
| 100% | 100% | 100% | 100% | |
How to passTest the emerging framework against a finite set of difficult cases drawn from issues found in the research, such as a service that is technically free but difficult to book, a tool library with extensive equipment but restrictive membership, a repair café with strong sustainability messaging but limited repair expertise, a bicycle workshop with excellent facilities but accessibility barriers, and a programme with negative user reports that conflict with current official information. Check whether the framework can distinguish apparent availability from practical usability and whether it avoids unfairly penalising services for limitations that are reasonable for their operating model. | ||||
| 0% | 0% | 0% | 100% | |
How to passSynthesize the findings into a reusable framework for evaluating repair-and-reuse services in other regions. Define the evaluation criteria, evidence hierarchy, method for comparing official information with participant experiences, treatment of missing information, approach to conflicting evidence, and a practical way to identify strengths, limitations, and access barriers without pretending that every service can be reduced to a single objective ranking. | ||||
| 100% | 0% | 100% | 100% | |
How to passPresent the selected services using the framework and explain what each type of service is realistically useful for, who can use it, what barriers may arise, and what evidence supports those conclusions. Clearly distinguish verified programme facts from participant reports, interpretation, and evidence gaps. explain the trade-offs between convenience, expertise, equipment access, cost, inclusiveness, and sustainability value, and provide practical guidance for someone evaluating a similar service in a new region. | ||||
Failure modes on this task
5 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | 1 | 3 | — | — |
| Standing up the evidence1 mode | — | — | 1 | — |