I’m interested in understanding how local news organisationsNews & Media Publishers
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5.5
100%
GPT-5.6 sol
75%
GPT-6 sol
25%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
35m
Claude Opus 5
23m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
111
Claude Opus 5
86
Claude Opus 5.5
53
GPT-6 sol
42
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (8) | ||||
| 100% | 0% | 100% | 0% | |
How to passEstablish the criteria that will be used consistently when comparing Kenyan news sources: organisation, publication type, geographic coverage, reporting format, accessibility, byline information, publication date and update information, use of named sources, links to primary documents where provided, distinction between reporting and opinion, correction or update information, editorial or complaints information where publicly documented, ownership or organisational information where relevant, and source URLs. Clearly distinguish a reported fact, an attributed claim, analysis, opinion, editorial content, and unverified information. Do not infer reliability from popularity, headline style, or political assumptions. | ||||
| 100% | 100% | 100% | 100% | |
How to passFind publicly accessible authoritative Kenyan sources explaining journalism, media standards, public information, corrections, complaints, or media literacy. Prefer the Media Council of Kenya, Communications Authority of Kenya, Kenyan government institutions, recognised journalism organisations, universities, and established media-literacy or professional organisations. Open the actual guidance or institutional pages and record what they explicitly state about journalistic standards, verification, attribution, corrections, public complaints, and responsible reporting where addressed. Use "not stated" for information that the sources do not explicitly provide. | ||||
| 100% | 100% | 100% | 100% | |
How to passSelect several established Kenyan news organisations with publicly accessible reporting. Include a range of formats such as national newspapers, digital-first outlets, broadcasters, or other established news publishers where reliable public information is available. For each organisation, verify its official website, reporting categories, publicly documented editorial information, correction or complaints mechanisms where available, and accessibility of articles without requiring a paid subscription. Do not assign an accuracy score or political label unless supported by a reliable source. | ||||
| 100% | 100% | 100% | 0% | |
How to passSelect a significant Kenyan public event or development for which multiple selected news organisations published publicly accessible coverage. Compare how the outlets report the same event, including headline wording, publication timing, named or anonymous sources, official statements, eyewitness information, background context, updates, and distinctions between news reporting and commentary. Identify agreements and meaningful differences without assuming that the most detailed or earliest article is automatically the most accurate. If an important claim cannot be independently verified from the available public evidence, record "not verified." | ||||
| 100% | 100% | 100% | 0% | |
How to passFor the selected events, identify important factual claims that can be checked against Kenyan government publications, court documents, official statistics, public statements, regulatory information, or other appropriate primary sources. Compare the news reports with those primary materials and record whether the claim is directly supported, attributed to another source, incomplete, disputed, or not verifiable from the available evidence. Do not treat an official statement as automatically proving every claim contained within it; distinguish the fact that an official made a statement from the truth of the statement itself. | ||||
| 100% | 100% | 100% | 0% | |
How to passExamine publicly accessible correction, update, editorial-policy, or complaints information from the selected news organisations where available. Record whether the organisation publicly explains how corrections, updates, reader complaints, or editorial concerns are handled. Where an organisation does not publish a requested policy, record "not stated" rather than assuming that it has no correction process. Clearly distinguish an organisation's stated policy from independently observed corrections or community claims. | ||||
| 100% | 100% | 100% | 0% | |
How to passSearch publicly readable Kenyan media, journalism, social-media, or relevant community discussions where multiple distinct readers or journalists discuss news consumption and verification. Look for recurring practical observations about sensational headlines, rapidly changing stories, copied reports, social-media reposts, corrections, inaccessible articles, paywalls, source attribution, or difficulty distinguishing opinion from reporting. Open the actual discussions and record their exact URLs. Summarise these as "community-reported observations rather than authoritative media-quality evidence." Do not use individual comments to establish that a news organisation is inaccurate, biased, or unreliable. | ||||
| 100% | 0% | 100% | 0% | |
How to passCreate a practical Kenyan news-verification and source-comparison guide covering established news organisations and cross-outlet event comparisons. Add a reusable verification checklist covering the original publisher, date and updates, author or byline, source attribution, primary evidence, distinction between reporting and opinion, corroboration by independent outlets, and correction or update information. Clearly separate verified facts, attributed claims, analysis, opinion, and community observations. Include exact URLs for every source used. | ||||
Failure modes on this task
8 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | — | 2 | — | 4 |
| Standing up the evidence2 modes | — | — | — | 2 |