I am writing a short reference pieceReference Materials
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5.5
91%
Claude Opus 5
55%
GPT-5.6 sol
36%
GPT-6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
34m
Claude Opus 5
31m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
87
Claude Opus 5
40
GPT-5.6 sol
39
Claude Opus 5.5
31
GPT-6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (11) | ||||
| 100% | 0% | 100% | 100% | |
How to passOpen https://whc.unesco.org/en/criteria/ and record how many selection criteria exist, how many a property must meet, and the document in which they are explained. Summarise each of the ten criteria in a few words each, keeping the roman numeral against each one. | ||||
| 100% | 0% | 0% | 100% | |
How to passFrom the same page, record what changed at the end of 2004 and reproduce the table showing which numerals counted as cultural and which as natural before and after. Then state what a pre-2005 source meant by natural criterion (ii) and which numeral now carries that meaning. This is the single most likely source of a published error. | ||||
| 100% | 0% | 100% | 0% | |
How to passFrom the official criteria page, record its warning that meeting a selection criterion is not the only consideration for inscription. Name the additional considerations stated there so the briefing does not imply that a criterion alone guarantees inscription. | ||||
| 100% | 100% | 100% | 100% | |
How to passOpen https://whc.unesco.org/en/list/stat and record the table of properties by region, with cultural, natural, mixed, total, share and number of States Parties for every region, plus the overall totals. Record the sentence about what the regions represent. | ||||
| 100% | 100% | 100% | 100% | |
How to passUsing only the regional table from the previous step, verify that the regional totals sum to the published overall total and that the cultural, natural and mixed columns reconcile with that total. Show the arithmetic and investigate any discrepancy through the page's counting notes rather than assuming an error. | ||||
| 100% | 0% | 100% | 100% | |
How to passFrom the same page, record the footnotes about trans-regional properties, naming each property involved, the countries it spans, and the single region it is counted in. State in one sentence what this means for anyone quoting a number of sites in Europe. | ||||
| 100% | 0% | 100% | 100% | |
How to passOpen https://whc.unesco.org/en/list/ and record the headline counts displayed above the listing: total properties, transboundary or transnational properties, and States Parties with properties out of the total number of States Parties. Record the page-access date because these counts can change. | ||||
| 100% | 100% | 0% | 100% | |
How to passOn the same page some property names carry symbols after them. State whether the page defines them anywhere in readable text. Then determine what at least one symbol means by cross-checking a marked property, for example the entry for Mosi-oa-Tunya / Victoria Falls at https://whc.unesco.org/en/list/509 , against the danger list in the next step. If you cannot establish a symbol's meaning from published text, say so rather than guessing. | ||||
| 100% | 100% | 0% | 100% | |
How to passOpen https://whc.unesco.org/en/danger-list/ and record the headline number of properties currently stated to be on the List of World Heritage in Danger. Preserve enough identifying information from the country-grouped entries to support the duplicate-count check in the next step, and record the page-access date. | ||||
| 100% | 0% | 0% | 100% | |
How to passCount the entries listed under country headings on the danger list and compare that number with the total the page states. If they differ, identify the property responsible, name both countries it is listed under, and explain in one sentence why the published total is the correct one. | ||||
| 100% | 0% | 0% | 100% | |
How to passProduce a briefing note with a table of every verified figure, its official source page, access date where the figure can change, and the counting rule that applies. Follow it with a short list of traps explaining how a careless writer could publish a wrong total or use obsolete criterion numbering, and how to avoid each error. | ||||
Failure modes on this task
13 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Standing up the evidence1 mode | — | 1 | 2 | — |
| Delivering the result1 mode | — | 1 | 2 | — |
| No signal in the judge’s words1 mode · the grader | — | 5 | 1 | 1 |