Build a reusable Media-Claim Verification Correction & Ed…News & Media Publishers
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5.5
100%
GPT-6 sol
100%
GPT-5.6 sol
0%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
72m
Claude Opus 5
38m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
107
Claude Opus 5
62
GPT-6 sol
57
GPT-5.6 sol
28
Claude Opus 5.5
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (15) | ||||
| 100% | 100% | 100% | 0% | |
How to passEstablish exactly what the tracker evaluates: pre-publication claims, published claims, corrections, updates, or all four. Define what counts as a factual claim and distinguish factual assertions from opinion, analysis, prediction, attribution, interpretation, and rhetorical language. Establish statuses such as Verified, Partially Supported, Unsupported, Contradicted, Unresolved, or Context-Dependent. Define when a claim should be escalated for additional review rather than receiving a definitive status. Separate the truth of a claim from the quality of its wording: a technically accurate statement can still create a misleading impression through omission, framing, or excessive certainty. Establish which evidence types are appropriate for different claim categories, such as public records, official statistics, original documents, direct statements, expert research, or independently corroborated reporting. The framework should remain usable months later when an article is revisited. Preserve enough information to reconstruct why a claim was accepted, challenged, corrected, or left unresolved. The objective is to make verification a repeatable editorial process rather than an informal judgment based on how convincing a source initially appears. | ||||
| 100% | 100% | 100% | 0% | |
How to passResearch and define an evidence hierarchy for editorial fact verification. Prioritize primary documents, official records, authoritative datasets, original research, direct statements, professional standards, and reputable institutional sources where appropriate. Use established journalism organizations, university journalism resources, academic research, books, and professional publications to explain verification practices. Distinguish between evidence that directly establishes a fact and sources that merely repeat another source's claim. Record source identity, publication date, author or institution, original-document status, accessibility, relevant passage or data, and limitations. Do not automatically treat a government or institutional source as infallible; identify the precise claim it supports and investigate material inconsistencies. Likewise, do not dismiss secondary reporting when it provides original reporting that cannot reasonably be obtained elsewhere. Define how social posts, aggregators, anonymous accounts, search snippets, and copied articles should be handled. The hierarchy should remain useful months later when the original source changes, disappears, or is superseded. Every tracker decision should make clear what evidence supports the conclusion and how directly that evidence establishes the claim. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a master register for every claim under review. Record claim ID, exact wording, article or publication context, publication status, author or source, publication information, related claims, and verification state. Preserve the exact original wording before any editorial correction. Record whether the claim is a number, quotation, causal assertion, attribution, historical statement, comparison, prediction, or other claim type. Identify whether the claim depends on a specific time, definition, population, dataset, or source. Avoid combining similar claims merely because they concern the same topic. A small wording change can materially change what is being asserted. This register should become the authoritative record for the verification process. Months later, if the article is updated or challenged, the team should be able to see exactly what was originally published and what evidence supported it at that time. Record relevant versions and timestamps as metadata where available, without treating publication time alone as proof of truth. The objective is to establish a precise claim identity before evidence is collected, preventing the review from accidentally evaluating a weaker or altered version of the original assertion. | ||||
| 100% | 100% | 100% | 0% | |
How to passDevelop a method for breaking compound statements into individual propositions that can be independently evaluated. Identify who, what, when, where, how much, compared with what, according to whom, and under what conditions. Separate factual components from implied causal or interpretive components. For example, a sentence may contain an accurate numerical fact but also imply a causal explanation that is not supported by the same evidence. Record each subclaim and map it to the evidence required to verify it. Research professional verification guidance to determine how complex claims should be handled without losing important context. Avoid simplifying the claim so much that its original meaning disappears. Months later, this decomposition should help reviewers determine exactly which part of an article became outdated or unsupported. The objective is to prevent a single source from being treated as evidence for an entire sentence when it actually supports only one component. A claim should not receive a fully verified status merely because one part of it is demonstrably true. The tracker should expose mixed claims and allow each meaningful proposition to receive its own evidence assessment. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a structured method for determining whether the evidence being cited actually originates from the claimed source. Verify document identity, publisher, author or institution, date, version, original location, relevant passage, dataset, and provenance where possible. Determine whether a source is original evidence or merely reproducing another report. For quotations, verify the original statement and surrounding context rather than relying on a quotation appearing in a secondary article. For statistics, confirm definitions, population, methodology, units, and relevant reporting period. For documents, determine whether an authentic original or authoritative copy is available. Record the exact relationship between source and claim: directly establishes, indirectly supports, contextualizes, or merely repeats. Months later, if the source is revised or removed, the provenance record should preserve enough information to reconstruct the original verification. The objective is to prevent citation chains from creating false confidence. A claim repeated by ten websites may still depend on one unsupported source. Conversely, a single primary document may provide stronger evidence than many secondary repetitions. The tracker should therefore evaluate evidence quality based on provenance and relevance, not source count alone. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a framework for checking whether the wording of a claim accurately reflects the strength of the evidence. Compare language such as “confirmed,” “reported,” “suggests,” “is associated with,” “caused,” “may,” “according to,” and “is disputed” against what the sources actually establish. Identify cases where an accurate underlying fact becomes misleading because the article uses stronger causal, numerical, temporal, or certainty language than the evidence supports. Research journalism standards and academic guidance on attribution, uncertainty, causal claims, and responsible reporting. Record the original wording, evidence-supported interpretation, unsupported implication, and recommended minimal adjustment where appropriate. Do not rewrite a claim simply to make it appear safer if the original wording is adequately supported. Likewise, do not treat hedging as automatically accurate if the underlying assertion remains unsupported. Months later, this framework can be reused when editorial teams review old articles for updates. The objective is to ensure that verification examines not only whether something happened, but also whether the publication communicates the evidence with an appropriate level of certainty. | ||||
| 100% | 100% | 100% | 0% | |
How to passDevelop a formal process for situations where credible sources disagree. Record claim, source A, source B, methodological differences, publication timing, definitions, population or scope differences, evidence quality, and unresolved issues. Investigate whether the apparent contradiction reflects different measurement methods, reporting periods, definitions, revisions, or genuinely incompatible findings. Do not resolve conflicts simply by choosing the source that supports the preferred narrative. Research professional journalism and scholarly methods for handling conflicting evidence. Where the disagreement cannot responsibly be resolved, allow the tracker to mark the claim Unresolved or Contested and specify what additional evidence would be needed. Months later, new official records or revised research should be attachable to the existing dispute rather than creating a completely new review. The objective is to make uncertainty visible rather than converting disagreement into artificial certainty. A credible editorial system should be capable of concluding that the evidence is genuinely mixed. It should also explain why one source may carry more weight for a specific question without claiming that every other credible source is simply incorrect. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a framework for claims whose validity depends on time. Identify whether the claim concerns a current status, historical condition, evolving event, statistic, policy, estimate, forecast, or temporary circumstance. Record the relevant time period and determine whether the source remains applicable. Research appropriate journalistic and data-verification guidance for handling revisions, updated statistics, changing policies, and evolving events. Distinguish between a claim that was accurate when published and one that was always inaccurate. Record the original status, later evidence, date of change, and required editorial action. Do not silently rewrite history when a previously accurate statement becomes outdated. Instead, preserve the original claim and document the update. Months later, this allows editors to determine whether an article needs a correction, an update note, or no change. The objective is to prevent time-sensitive claims from being evaluated as though they were timeless facts. A strong tracker should make it immediately visible which claims require periodic review because their truth conditions can change. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a correction-analysis framework that examines whether changing one sentence actually resolves the underlying problem. Record original claim, identified error, corrected wording, evidence supporting correction, publication location, related statements, and remaining contextual risk. Check whether neighboring sentences repeat the same false implication or whether the correction creates a contradiction elsewhere in the article. Research correction policies from reputable publishers and professional journalism organizations. Distinguish between a minor factual correction, clarification, update, attribution correction, and a material error that changes the reader's understanding. Do not assume that adding a correction note automatically fixes a misleading article. Months later, when an article is revisited, the tracker should show the complete correction history and whether subsequent evidence requires another review. The objective is to treat corrections as evidence-management events rather than isolated text edits. A correction should restore an accurate understanding of the underlying claim, not merely replace one incorrect number. This section should also record whether the original source itself later corrected its information and whether that change affects the publication's responsibility to update its reporting. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate the central decision methodology for determining claim status. Evaluate source authority, directness, provenance, corroboration quality, methodological soundness, temporal relevance, wording accuracy, contextual fit, contradiction level, and remaining uncertainty. Establish mandatory gates for serious unresolved problems rather than allowing a high-quality source elsewhere in the record to compensate automatically. For each claim, provide Status, Evidence Summary, Confidence, Key Limitation, Wording Assessment, and Required Action. Avoid false numerical precision when evidence quality cannot reasonably be expressed as an exact percentage. Where evidence is insufficient, the correct result should be Unresolved, not an artificially confident conclusion. Where a claim is partly supported, identify the supported and unsupported portions separately. Months later, new evidence should be added to the existing decision and trigger re-evaluation where material. The objective is to create an auditable editorial decision rather than a simple true/false label. The final system should explain not just what status was assigned, but why that status was appropriate given the evidence available at the time. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a long-horizon workflow covering the complete lifecycle of a claim: initial review, publication, challenge, evidence update, correction, follow-up, and closure. Define triggers such as new official records, source corrections, credible challenges, revised statistics, major policy changes, newly available research, or evidence that materially changes the original conclusion. Preserve the original decision and create a new review version rather than overwriting historical evidence. Record who reviewed the claim, what changed, which evidence triggered review, and what editorial action followed. Distinguish between no action, clarification, update, correction, retraction, or further investigation based on the evidence. Do not create a monitoring requirement for claims whose truth conditions are stable and unlikely to change. Months later, the system should allow an editor to identify which claims require renewed verification and why. The objective is to make the tracker useful beyond one publication cycle. It should function as a living evidence record that helps editorial teams respond consistently when information changes, while preserving the historical record needed to understand earlier decisions. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate exactly 8 fictional stress-test scenarios and apply the complete tracker to each. Clearly label every scenario as fictional. Include: (1) a numerically accurate statement whose wording creates an unsupported causal impression; (2) two credible sources that report conflicting figures because they use different definitions; (3) a secondary article that accurately quotes a statement but misrepresents the surrounding context; (4) a historically accurate claim that becomes misleading after circumstances change; (5) a viral claim repeated by many websites but traceable to one weak source; (6) an official document containing an apparent inconsistency that requires contextual investigation; (7) a correction that fixes a sentence while leaving the article's broader implication misleading; and (8) an important claim that cannot yet be verified despite strong pressure to publish it. For each fictional scenario, demonstrate claim decomposition, source provenance, evidence evaluation, wording analysis, contradiction handling, time sensitivity, correction logic, decision status, and re-verification requirements. These are fictional methodology tests only. | ||||
| 100% | 100% | 100% | 0% | |
How to passAttempt to break the completed system using common verification failures. Present ten websites repeating the same unsupported claim and test whether repetition is incorrectly treated as corroboration. Introduce an authoritative source that supports only one component of a compound sentence and test whether the whole sentence is marked verified. Add a credible source with a newer publication date but weaker methodological relevance and determine whether recency improperly dominates. Present an accurate statistic with a misleading headline and test whether wording strength is evaluated separately. Introduce a correction that fixes a number while leaving the causal implication untouched. Test whether the tracker distinguishes a claim that was once accurate from one that was always false. Present two credible sources using different definitions and determine whether the system forces a false consensus. Audit the evidence hierarchy, claim register, decomposition process, provenance checks, wording analysis, conflict workflow, time-sensitivity review, correction process, decision model, and re-verification workflow. Add safeguards against source-count bias, headline bias, recency bias, citation-chain bias, and false-certainty bias. | ||||
| 100% | 100% | 100% | 0% | |
How to passTest the completed tracker as though it were handed to an unfamiliar editorial reviewer. The stranger should be able to register a claim, preserve its exact wording, classify it, identify appropriate evidence, verify provenance, decompose complex assertions, evaluate wording strength, resolve or document conflicts, assess time sensitivity, record correction history, and determine whether re-verification is necessary. Add a concise How to Use This Tracker workflow covering Claim → Context → Source → Provenance → Decomposition → Evidence → Wording → Conflict → Time → Decision → Correction → Re-Review. Remove unexplained terminology and hidden assumptions. Confirm that a new claim can be added without changing the methodology. The Stranger Test is satisfied only when an unfamiliar reviewer can complete the process without asking follow-up questions. Verify that the tracker remains useful weeks or months later when a source changes, a correction is issued, a claim is challenged, or an evolving story develops. The future reviewer should be able to reconstruct what was known when the original decision was made and identify exactly what new evidence would justify changing it. | ||||
| 100% | 100% | 100% | 0% | |
How to passAssemble the final Media-Claim Verification, Correction & Editorial Traceability Tracker with clearly separated sections for Instructions, Claim Register, Publication Context, Source Register, Evidence & Provenance, Claim Decomposition, Wording Assessment, Conflicting Evidence, Time Sensitivity, Correction History, Decision Model, Re-Verification Queue, and Review History. Ensure every major decision is traceable to evidence appropriate to the claim. Verify that repeated secondary reporting is not mistaken for independent corroboration and that primary-source support is evaluated for relevance rather than authority alone. Confirm that factual accuracy and wording accuracy are assessed separately. Ensure corrections preserve the original claim and explain what changed. Confirm that the eight scenarios remain clearly fictional and that the adversarial audit produces concrete safeguards. Review status definitions, evidence fields, temporal logic, correction categories, re-verification triggers, and decision criteria for contradictions or ambiguity. Final Quality Check: a stranger must be able to evaluate an unfamiliar media claim, identify appropriate evidence, verify provenance, distinguish supported facts from unsupported implications, document uncertainty, manage corrections, preserve decision history, and know when the claim should be reviewed again—without asking follow-up questions. | ||||
Failure modes on this task
15 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Standing up the evidence1 mode | — | — | — | 10 |
| Delivering the result1 mode | — | — | — | 4 |
| Reaching the page1 mode | — | — | — | 1 |