I require a deep evidence based investigationNews & Media Publishers
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5.5
100%
GPT-6 sol
90%
GPT-5.6 sol
40%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
32m
Claude Opus 5
21m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
119
Claude Opus 5
65
GPT-5.6 sol
56
Claude Opus 5.5
34
GPT-6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (10) | ||||
| 100% | 100% | 100% | 100% | |
How to passBegin by defining precisely what qualifies as an independent, non-partisan political fact-checking organisation for this investigation. Search several potential candidates, then select PolitiFact as the organisation for the detailed investigation. Confirm that it regularly evaluates factual claims, has a publicly identifiable editorial methodology, and publishes sufficient material and source trails for independent examination. Explain why PolitiFact is appropriate for the investigation and identify important limitations such as geographic focus, language restrictions, or the types of political claims it normally evaluates. | ||||
| 100% | 100% | 100% | 100% | |
How to passOnce a candidate has been selected verify its International Fact Checking Network status directly through the IFCN's own public signatory database. Record the organisation's name country assessment date and current status and determine whether its verification is active or expired. Do not rely on the fact checker's own statement that it follows the IFCN Code. The IFCN explains that signatory organisations are assessed against multiple criteria covering areas such as non partisanship fairness sourcing funding transparency and organisational independence. Explain what the organisation's signatory status actually demonstrates and what it does not demonstrate. A verified signatory status should be treated as evidence that the organisation has undergone the IFCN assessment process rather than as an absolute guarantee that every published verdict is free from error. | ||||
| 100% | 100% | 100% | 100% | |
How to passResearch PolitiFact's financial structure using its official funding disclosures and other credible supporting records. Identify the major categories of income, significant funders, grants, donations, advertising revenue, and content partnerships. Record any publicly disclosed amounts or funding purposes where available. Determine whether PolitiFact publicly explains safeguards intended to prevent funders from influencing editorial conclusions. The IFCN Code requires transparency about funding and editorial independence. Do not treat a financial relationship with a politically interested organisation as proof of bias; distinguish an identifiable funding relationship from evidence that funding influenced a particular fact check. Assess both potential perceptions of bias and the available evidence concerning actual editorial independence. | ||||
| 100% | 100% | 100% | 0% | |
How to passFind a political claim fact checked by the selected organisation within the previous three months. Confirm the publication date of the fact check and ensure that the underlying political claim remains relevant at the time of research. Locate the earliest reliable version of the politician's statement whenever possible rather than relying entirely on the fact checker's quotation. Determine who made the claim where it was made what question the speaker was addressing and whether the statement was part of a speech political advertisement interview debate campaign event government announcement or social media communication. Reconstruct the surrounding context because a short quotation can sometimes produce a misleading impression when separated from the larger statement. Clearly identify the exact proposition that the fact checker evaluated and distinguish it from related political arguments that were not actually examined. | ||||
| 100% | 100% | 100% | 0% | |
How to passRead the complete fact check carefully and reconstruct the sequence of investigation used by the organisation. Identify the evidence that the fact checker considered most important and explain how that evidence was connected to the final verdict. Separate primary sources from secondary sources throughout the analysis. Primary sources should include government datasets official records legislation court documents election records transcripts direct statements government reports budget documents regulatory filings or original research. Secondary sources should include newspaper articles television reports academic commentary other fact checks or analysis produced by third parties. The IFCN's principles specifically emphasise identifying significant evidence and using the best available primary sources where suitable primary evidence exists. Determine whether the selected fact check appears to follow this standard and identify any important evidence that the fact checker relied upon without providing enough detail for an outside reader to independently reproduce the conclusion. | ||||
| 100% | 100% | 100% | 0% | |
How to passDo not simply repeat the fact checker's conclusion. Open and examine the most important primary sources cited in the investigation and determine whether they actually support the interpretation presented by the fact checker. Where the fact check cites government statistics compare the relevant figures directly. Where legislation or official policy documents are involved examine the relevant language and context. Where a politician's direct statement is central to the dispute locate the original transcript recording or complete quotation whenever available. Determine whether the evidence supports the verdict completely or whether the truth is more complicated. Identify any assumptions calculations definitions or contextual choices that materially affect the conclusion. The objective of this stage is to determine whether the verdict can be independently reconstructed from the evidence rather than merely accepted because an established fact checker published it. | ||||
| 100% | 100% | 100% | 0% | |
How to passCompare the selected political claim with coverage from two mainstream news organisations: CNN and Fox News. Examine their original reporting and determine how they described the underlying claim, evidence, and political context. Compare their language and factual conclusions with PolitiFact's verdict. If they agree, determine whether they independently examined the evidence or merely reported the fact check. If they disagree, identify exactly where the disagreement occurs. Note any additional context that strengthens, weakens, or complicates PolitiFact's conclusion. | ||||
| 100% | 100% | 0% | 0% | |
How to passGo beyond comparing headline wording and conduct a detailed framing analysis. Examine which facts each organisation placed near the beginning of the story which evidence received the greatest attention and which contextual details were emphasised or omitted. Determine whether differences arise from factual disagreement methodological differences political context article length or editorial priorities. Avoid using the word bias as a substitute for analysis. Instead identify observable differences in evidence selection language emphasis sourcing and interpretation. If one news outlet reports the politician's statement without discussing the fact checker's conclusion explain the significance of that omission. If another outlet prominently features the fact checker's verdict but provides little independent evidence explain that distinction as well. The goal is to show how the same factual dispute can produce different journalistic narratives even when the underlying evidence is largely shared. | ||||
| 100% | 100% | 100% | 100% | |
How to passBring together the institutional evidence and the individual fact check analysis to evaluate the organisation's credibility. Consider its IFCN status funding transparency source quality methodology consistency and ability to provide evidence that readers can independently inspect. Determine whether the selected fact check applies the same evidentiary standard regardless of which political side made the claim by examining comparable recent fact checks involving opposing political actors where possible. Do not assume that a fact checker is non partisan simply because it labels itself non partisan. Look for observable evidence concerning claim selection verdict distribution sourcing standards and methodology. At the same time do not infer political bias merely from the fact that a particular party receives more negative verdicts because the underlying claims and their factual characteristics may differ. The evaluation should remain evidence driven and should clearly distinguish institutional risk from demonstrated editorial misconduct. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a final evidence-based research brief containing PolitiFact's mission, IFCN verification status, funding structure, major funding sources, and safeguards concerning editorial independence. Present the selected recent political claim, including the speaker, original context, date, fact-check publication date, final verdict, and evidence trail. Include a structured comparison distinguishing primary evidence from secondary reporting. Then compare CNN and Fox News and classify their treatment of the claim as matching PolitiFact's conclusion, contradicting it, adding important nuance, or independently reaching a similar conclusion. Clearly cite the sources used and distinguish verified facts from analytical conclusions. | ||||
Failure modes on this task
7 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Standing up the evidence2 modes | — | — | 1 | 3 |
| Delivering the result1 mode | — | — | — | 2 |
| No signal in the judge’s words1 mode · the grader | — | — | — | 1 |