I am researching how major international news organizationsNews & Media Publishers
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5.5
100%
GPT-6 sol
100%
GPT-5.6 sol
40%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
82m
Claude Opus 5
28m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
119
Claude Opus 5
100
Claude Opus 5.5
35
GPT-6 sol
34
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (10) | ||||
| 100% | 100% | 100% | 100% | |
How to passDefine and document the three-month study period, publisher-selection criteria, search strategy, inclusion and exclusion rules, and coding categories so that another researcher could reproduce the sampling process. | ||||
| 100% | 100% | 100% | 100% | |
How to passIdentify at least 12 established international news publishers with accessible public archives or search pages, and document the selection rationale and source used to verify each publisher's eligibility. | ||||
| 100% | 100% | 100% | 100% | |
How to passCollect the article sample using the documented publisher search strategy and Google, applying the defined inclusion rules and sample-size method consistently across publishers. Record each article's source URL and publication date. | ||||
| 100% | 100% | 100% | 0% | |
How to passCheck each article's title, publisher, date, author where available, and accessibility. | ||||
| 100% | 100% | 100% | 100% | |
How to passDefine consistent categories for event type, geography, scientific evidence, framing, attribution, solutions, and uncertainty. | ||||
| 100% | 100% | 100% | 0% | |
How to passApply the coding scheme systematically and record evidence supporting ambiguous classifications. | ||||
| 100% | 100% | 100% | 0% | |
How to passReview the dataset for duplicate articles, inconsistent categories, and sampling errors. | ||||
| 100% | 100% | 100% | 0% | |
How to passCalculate descriptive patterns across publishers, regions, event types, and framing categories without overgeneralizing beyond the sample. | ||||
| 100% | 100% | 100% | 0% | |
How to passIdentify search-engine, archive, language, sampling, paywall, and survivorship limitations. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a structured dataset and research report containing the documented methodology, publisher sample, coded results, findings, limitations, and source URLs for the sampled articles. | ||||
Failure modes on this task
6 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Reaching the page1 mode | — | — | — | 1 |
| No signal in the judge’s words1 mode · the grader | — | — | — | 5 |