Build a reusable geographic data and map-verification systemReference Materials
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5.5
100%
GPT-6 sol
100%
GPT-5.6 sol
73%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
103m
Claude Opus 5
16m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
115
Claude Opus 5
53
GPT-6 sol
33
Claude Opus 5.5
24
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (15) | ||||
| 100% | 100% | 100% | 100% | |
How to passStart by defining what the system considers a geographic claim and what types of claims require verification. Research authoritative geospatial and mapping guidance to distinguish between a place name, physical location, coordinate, address, boundary, route, facility, landmark, and geographic relationship. Establish the main questions: Does the place exist? Is the identified entity the correct place? Is the location accurate enough for the intended purpose? Which administrative area does it belong to? Is the information current? Do multiple sources agree? Create an intake structure recording the claim, purpose, source, date, geographic precision required, and decision consequence. Include fictional examples where a coordinate is sufficient for general research but insufficient for property-level analysis, and another where an approximate location is perfectly adequate. Explain why verification requirements should depend on the intended use rather than applying one universal accuracy standard. This creates the foundation for the tracker and prevents unnecessary verification work while ensuring high-consequence geographic claims receive appropriate scrutiny. | ||||
| 100% | 100% | 100% | 100% | |
How to passIdentify authoritative sources for different types of geographic information. Prioritize official mapping agencies, government geographic datasets, municipal records, transportation authorities, environmental agencies, recognized geospatial standards bodies, and documented open-data sources. Distinguish authoritative source status from geographic accuracy for a particular claim. A government dataset may be authoritative for administrative boundaries but not necessarily the best source for current commercial facilities. Create an evidence register containing source, dataset or page, geographic scope, update information, methodology where available, intended use, and limitations. Include a fictional case where a crowdsourced map has more recent information than an older official dataset. Explain how both should be evaluated rather than automatically choosing one based only on institutional status. Record whether a source is primary, derived, aggregated, or user-contributed. The goal is to establish a source hierarchy that is claim-specific. This should prevent the tracker from treating all map sources as equivalent while also preventing the reviewer from assuming that official status eliminates the need for verification. | ||||
| 100% | 100% | 100% | 100% | |
How to passCreate the central tracker for recording geographic claims. Capture entity name, alternate names, address, coordinates, administrative area, feature type, source, observation date, verification status, intended use, and required confidence level. Include fields for exactness, missing information, conflicting information, and evidence links. Create categories such as verified, partially verified, conflicting, outdated, insufficient evidence, and unable to verify. Include fictional examples involving a landmark with multiple names, a facility listed under an abbreviated name, and a location with incomplete address information. Make sure the system distinguishes between missing evidence and evidence that contradicts the claim. Do not infer missing geographic details merely because another nearby place appears plausible. Record exactly what the source establishes. The intake sheet should be usable by someone who has never worked with the project before. It should also preserve the original claim so later corrections do not erase what was initially being verified. | ||||
| 100% | 100% | 100% | 100% | |
How to passDevelop a process for determining whether two geographic records actually refer to the same physical entity. Research authoritative geographic-data and entity-resolution practices where available. Compare names, alternate names, addresses, coordinates, feature types, surrounding geographic context, and official identifiers. Include fictional examples where two facilities have almost identical names, one organization has moved locations, and one place is commonly known by a nickname different from its official name. Create fields for matching evidence and confidence. Avoid treating identical names as proof of identity. Similarly, avoid assuming different names represent different places. Where official identifiers exist, record them as stronger evidence than name similarity alone. Identify ambiguous cases requiring additional verification. The purpose is to prevent the tracker from accidentally merging separate places or treating one place as several entities. This becomes especially important when data from multiple map sources is being compared. The final workflow should produce a documented identity decision rather than simply selecting whichever record looks most similar. | ||||
| 100% | 100% | 100% | 100% | |
How to passCreate a method for comparing coordinates from different sources. Research authoritative geospatial guidance concerning coordinate systems, positional accuracy, precision, and geographic representation. Record latitude, longitude, coordinate reference information where available, source, precision, and stated or estimated accuracy. Avoid assuming that more decimal places automatically mean greater accuracy. Include fictional cases where two coordinates differ slightly because of normal representation differences and another where the coordinates identify genuinely different physical locations. Define tolerance rules appropriate to the intended use rather than one universal distance threshold. Record whether a difference is insignificant, requires investigation, or materially changes the geographic conclusion. Where technical transformation between coordinate systems is necessary, document it rather than silently altering the data. The tracker should distinguish coordinate precision from actual positional confidence. This section should help users understand whether two map records meaningfully disagree or merely represent the same place at different levels of precision. | ||||
| 100% | 100% | 100% | 100% | |
How to passCreate a structured method for checking addresses and relationships between places. Research authoritative addressing and geographic-reference guidance where available. Verify components such as street or locality names, administrative areas, postal information where applicable, and relationships to nearby geographic features. Include fictional cases where a search result associates a business with a nearby street but the official address differs, and another where a location lies close to an administrative boundary. Record exactly which components are verified and which remain uncertain. Do not assume that an address automatically proves the physical coordinate is correct. Similarly, proximity to a landmark does not prove that the place is located within the landmark's administrative or functional area. Create fields for verified relationship, source, evidence, and confidence. This allows the tracker to handle claims such as “located within,” “adjacent to,” “served by,” or “nearest to” without treating them as interchangeable. The objective is to make geographic relationships explicit and evidence-based. | ||||
| 100% | 100% | 100% | 100% | |
How to passDevelop a method for resolving boundary-related discrepancies. Research official boundary datasets and recognized geographic standards. Identify the relevant boundary type, such as administrative, statistical, service, electoral, planning, environmental, or other functional boundaries where applicable. Include fictional examples where a place appears to belong to one area on a commercial map but another on an official administrative dataset. Add a case where boundaries have changed over time. Record boundary source, version or date, geographic precision, and interpretation. Avoid treating all boundaries as interchangeable merely because they share a similar name. Explain how the intended purpose determines which boundary matters. For example, a statistical region may not correspond to a service-delivery area. The tracker should allow multiple boundary relationships to be recorded simultaneously when necessary. This section should help prevent common geographic errors caused by using the wrong boundary definition. It should also make temporal changes visible rather than silently applying today's boundary to historical claims. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a temporal-verification process for geographic information. Research authoritative practices concerning dataset versions, update cycles, historical geographic records, and temporal validity. Record when information was created, observed, published, updated, or superseded where those dates are available. Include fictional examples involving a facility that closed, a renamed road, a changed administrative boundary, and a newly constructed route. Explain why a source can be geographically accurate but temporally inappropriate for a particular question. Distinguish between “currently verified,” “historically verified,” “unknown date,” and “potentially outdated.” Avoid assuming that the newest source is automatically correct for historical research. Similarly, do not use an old dataset for a current decision without checking whether relevant changes occurred. The tracker should preserve historical evidence instead of replacing it whenever new information appears. This makes the system useful for both current and retrospective geographic analysis and prevents time-dependent errors from being mistaken for simple source disagreement. | ||||
| 100% | 100% | 100% | 100% | |
How to passBuild a discrepancy-classification system for situations where sources disagree. Research authoritative approaches to geographic-data quality and metadata. Classify differences as naming variation, coordinate precision, genuine positional disagreement, outdated information, boundary difference, feature-type mismatch, source methodology difference, or unresolved conflict. Include fictional examples representing each type. Require the reviewer to identify the exact field in conflict rather than recording only “maps disagree.” Compare source authority, recency, geographic scope, methodology, and intended purpose. Avoid resolving conflicts merely by majority vote. Copied map listings should not automatically outweigh one authoritative dataset. At the same time, an old official record may need to be reconsidered when newer authoritative evidence exists. Record the reason for the final resolution and the evidence supporting it. If no reliable resolution exists, mark the issue as unresolved rather than inventing a compromise coordinate or description. This creates an auditable process for geographic discrepancies and prevents false certainty. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a feature-status verification process for places that may change over time. Research authoritative sources relevant to public facilities, transportation infrastructure, environmental features, cultural landmarks, and other geographic entities. Record entity type, operational or physical status where relevant, location evidence, official identifier, source date, and verification confidence. Include fictional cases where a facility is listed in multiple datasets but only one source confirms that it remains active, and another where a landmark's physical location is stable but its official designation has changed. Distinguish between physical existence and operational status. A building can remain physically present even if the organization occupying it has closed. Similarly, a road can exist while access conditions change. The tracker should avoid combining these concepts into one “active/inactive” field. Create separate fields for physical presence, operational status, designation, and evidence where appropriate. This improves accuracy when geographic data is used for planning, research, or operational decisions. | ||||
| 100% | 100% | 100% | 100% | |
How to passCreate a transparent confidence model for each verified geographic claim. Consider source authority, agreement between independent sources, positional accuracy, temporal validity, completeness, methodology, and intended-use requirements. Keep confidence separate from certainty. A claim may be high confidence for general mapping but insufficient for a precise operational decision. Include fictional cases where three sources agree but all derive from the same underlying dataset, and another where two independent authoritative sources disagree slightly. Explain how source independence affects confidence. Use categories such as high, moderate, low, and unresolved with written reasons rather than relying solely on numerical scores. Record what additional evidence would raise confidence. Avoid allowing a numerical score to conceal a serious unresolved contradiction. The objective is to provide decision-ready confidence without pretending that geographic data is perfectly accurate. This confidence system should become one of the most important outputs of the tracker because it tells downstream users not only what the geographic information says, but also how much reliance is justified. | ||||
| 100% | 100% | 100% | 100% | |
How to passDetermine whether a geographic claim is sufficiently reliable for its intended purpose. Research authoritative geographic-data quality and risk concepts. Create use categories such as general research, visualization, planning, public communication, operational decision-making, or high-precision applications where relevant. Define different evidence thresholds for each. Include fictional cases where a coordinate is adequate for a broad map but inadequate for determining a precise property boundary, and another where a minor naming discrepancy has no practical effect. Record consequence if wrong, required precision, confidence, and whether additional verification is necessary. This prevents the system from treating geographic accuracy as an absolute concept. The right question is often whether the data is fit for the decision being made. Include a “decision blocker” field for cases where uncertainty is too consequential to ignore. Also identify cases where further verification would provide little practical benefit. This makes the system more efficient while protecting against high-consequence geographic errors. | ||||
| 100% | 100% | 100% | 0% | |
How to passTest the full system using the difficult fictional cases requested in the prompt: two authoritative sources that disagree, a place with multiple names, an address pointing to the wrong physical location, a facility appearing open in one dataset but closed in another, a changed boundary, and an apparently accurate coordinate associated with the wrong entity. For each case, trace identity verification, coordinate comparison, temporal analysis, source evaluation, discrepancy classification, confidence scoring, and intended-use assessment. Include at least one case where the correct outcome is “unresolved” rather than choosing a winner. Add another where a lower-authority source provides useful corroborating evidence without becoming the primary authority. Review whether the tracker distinguishes source disagreement from genuine geographic disagreement. Use the results to refine thresholds, formulas, categories, and instructions. The goal is to demonstrate that the system works under messy real-world conditions rather than only when every map source agrees. | ||||
| 100% | 100% | 100% | 100% | |
How to passReview the entire tracker as if you were deliberately trying to create a false sense of geographic certainty. Check for biases toward newer sources, commercial map platforms, government datasets, majority voting, excessive coordinate precision, or visually convincing maps. Verify that copied information is not being counted as independent evidence. Check whether outdated data can accidentally override current evidence and whether current data can incorrectly replace historical evidence. Examine whether confidence scores hide unresolved contradictions. Remove fields that encourage unsupported assumptions. Then convert the refined tracker into a reusable workflow: define the claim, identify the entity, collect authoritative sources, verify coordinates and addresses, inspect boundaries, check temporal validity, classify discrepancies, assess confidence, determine fitness for intended use, and document the final decision. Add update triggers for source revisions, boundary changes, facility-status changes, or newly discovered authoritative evidence. The workflow should be usable for future geographic datasets without requiring the creator to redesign the methodology. | ||||
| 100% | 100% | 100% | 0% | |
How to passAssemble the final tracker covering claim intake, source records, entity matching, coordinate and address checks, boundary and temporal analysis, feature status, discrepancies, confidence, intended-use risk, scenario testing, and a decision summary. Ensure every major geographic conclusion has traceable evidence. Preserve original source information and distinguish verified facts from interpretation. Add controlled fields for confidence and verification status where useful, but ensure important decision blockers cannot be hidden by numerical scoring. Include fictional test cases and their final resolutions as examples. Add a final quality checklist confirming that source independence, temporal validity, positional accuracy, entity identity, and intended-use requirements were considered. Finally, perform the stranger test: someone who has never seen the project should be able to enter a geographic claim, find appropriate authoritative evidence, compare conflicting map records, determine whether the information is sufficiently reliable for its intended use, document uncertainty, and produce a defensible verification decision without asking the task creator for additional instructions. | ||||
Failure modes on this task
4 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | — | — | — | 2 |
| Reaching the page1 mode | — | — | — | 1 |
| No signal in the judge’s words1 mode · the grader | — | — | — | 1 |