Locate and document the principal publicly available refe…Reference Materials
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
88%
Claude Opus 5.5
63%
GPT-6 sol
63%
Claude Opus 5
50%
GPT-5.6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
37m
GPT-5.6 sol
13m
Claude Opus 5
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
55
Claude Opus 5
42
GPT-6 sol
34
GPT-5.6 sol
28
Claude Opus 5.5
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (8) | ||||
| 100% | 100% | 100% | 100% | |
How to passNavigate to https://scar.org/library-data/maps. Record the listed SCAR map and geographic information products, including the Antarctic Digital Database, the Antarctic Map Catalogue, GeoMAP, BEDMAP and any other named datasets. Note the exact page URL and the short description provided for each product. | ||||
| 100% | 100% | 100% | 0% | |
How to passNavigate to https://www.bas.ac.uk/project/add/. Record the stated purpose of the SCAR Antarctic Digital Database, the geographic coverage south of 60°S, the types of datasets included such as coastline, contours and rock outcrop, and the access methods described. Capture the full page URL and the concrete dataset list. | ||||
| 100% | 0% | 0% | 0% | |
How to passNavigate to https://data.bas.ac.uk/items/e74543c0-4c4e-4b41-aa33-5bb2f67df389/. Record the current edition or version indicator, the list of available datasets, the projection details if stated, and any notes on download formats or previous versions. Note the exact page URL and the key metadata points. | ||||
| 0% | 0% | 100% | 100% | |
How to passNavigate to https://data.bas.ac.uk/collections/bas-published-maps. Record the description of the collection, examples of topographic or thematic maps listed, and any statements about open-licence or freely viewable items. Capture the page URL and the sample map titles or series mentioned. | ||||
| 100% | 100% | 0% | 0% | |
How to passNavigate to https://www.gns.cri.nz/data-and-resources/geomap-geological-mapping-of-antarctica/ (GNS Science, which leads the GeoMAP project). Record the purpose of GeoMAP, the types of geological information compiled, the scale or framework described, and any access or citation notes. Note the exact page URL and the main content summary. | ||||
| 100% | 100% | 0% | 100% | |
How to passNavigate to https://www.bas.ac.uk/project/bedmap/. Record the aim of the Bedmap project, the types of gridded products covering ice thickness, bed and surface elevation, and the portal or data-access methods indicated. Capture the full page URL and the key product descriptions. | ||||
| 100% | 100% | 100% | 100% | |
How to passNavigate to https://scar.org/library-data/maps/cga-composite-gazetteer-of-place-names (or the linked placenames.aq site). Record the purpose of the SCAR Composite Gazetteer of Antarctica, the number of names/features covered where stated, the countries contributing national gazetteers, and the search or access methods described. Note the exact page URL and the key summary content. | ||||
| 100% | 0% | 0% | 100% | |
How to passUsing only the data recorded in the prior steps, create a single written document organised under headings for Topographic Digital Databases, Map Catalogues and Published Sheets, Geological and Bedrock Datasets, Place-Name and Supporting Resources, and Access Methods. Place the exact source URL beside every factual claim and ensure all version or coverage statements remain relative to the latest published releases, leaving the document ready for review. | ||||
Failure modes on this task
11 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Standing up the evidence1 mode | — | — | — | 2 |
| Delivering the result1 mode | — | 1 | 1 | — |
| No signal in the judge’s words1 mode · the grader | 1 | 2 | 3 | 1 |