I’m interested in learning how someone can researchReference Materials
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
88%
Claude Opus 5.5
88%
Claude Opus 5
63%
GPT-6 sol
63%
GPT-5.6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
19m
Claude Opus 5
9m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
83
Claude Opus 5
69
Claude Opus 5.5
39
GPT-5.6 sol
36
GPT-6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (8) | ||||
| 100% | 0% | 100% | 100% | |
How to passDefine the criteria that will be used consistently when comparing Portuguese family-history resources: resource name, institution or provider, record type, historical period covered where stated, geographic coverage, searchable versus browse-only access, availability of original digital images, index availability, language, account requirement, payment requirement, online versus in-person access, request or reproduction options where stated, source type, reliability considerations, known limitations, and official URL. Distinguish an original archival record from an index, catalogue entry, finding aid, transcription, family-tree database, or secondary historical source. Do not treat an index or user-submitted family tree as equivalent to an original record. | ||||
| 100% | 100% | 100% | 100% | |
How to passFind at least five publicly accessible authoritative sources explaining how Portuguese genealogical records are organised and accessed, including national and district archives, civil-registration authorities, and other relevant archival, library, or religious repositories where documented. Record what the sources explicitly state about the location and access of baptism, marriage, death, civil-registration, census, notarial, immigration, military, or other relevant records where addressed. Explain historical transitions such as parish-to-civil registration only when supported by the sources. Clearly identify regional, institutional, or chronological exceptions rather than implying that one archival structure applies uniformly across Portugal. Use "not stated" for information not explicitly provided. | ||||
| 100% | 100% | 0% | 100% | |
How to passFind and verify at least ten resources that can genuinely help with Portuguese ancestry research and that have publicly accessible descriptive or search pages. Include a mixture of official archive catalogues, digitised parish or civil-record collections, library resources, historical newspapers, maps or gazetteers, and recognised genealogical research tools where appropriate. For each resource, verify and record what is available while logged out, what requires a free account, what requires payment or other restricted access, and whether original digital images are actually viewable. Record the institution, resource type, geographic scope, historical coverage, search or browse functionality, image availability, access requirements, and exact URL. Do not count a resource as openly accessible merely because its collection-description page is public, and do not rely on search-result snippets. | ||||
| 0% | 0% | 0% | 0% | |
How to passUse at least five fictional Portuguese genealogy scenarios to test the strongest resources, such as locating a nineteenth-century baptism, finding a twentieth-century civil record, identifying the historical parish belonging to a place, researching an ancestor’s occupation, locating a historical newspaper reference, or investigating a family that moved between Portuguese districts. Apply the resources without using real personal information. Record which resource provides the strongest documented pathway for each scenario and identify where the public resources stop and an archive request or in-person research may be required. Do not invent a successful record match when the resource does not provide one. | ||||
| 100% | 100% | 100% | 100% | |
How to passFor the strongest resources, cross-check claims that materially affect how a researcher would use them, especially historical coverage, geographic coverage, digitisation status, search functionality, access restrictions, and availability of original images. Use a second authoritative or established source where one meaningfully helps verify those claims. If only the holding institution documents a particular collection or access condition, record that limitation rather than requiring artificial corroboration. Preserve discrepancies between sources and identify which source is more specific, authoritative, or current without silently combining different collections. | ||||
| 100% | 100% | 100% | 100% | |
How to passUse authoritative historical or archival sources to investigate practical problems that can affect Portuguese genealogy research, including spelling and naming variation, parish-boundary changes, municipality or district changes, handwritten records, historical Portuguese terminology, incomplete or unavailable records, damaged records, duplicate names, and differences between religious and civil records where addressed. Explain how the researched resources help users deal with these problems, if they do. Do not invent translations, historical administrative boundaries, or search strategies that are unsupported by the sources. | ||||
| 100% | 100% | 100% | 100% | |
How to passSearch publicly readable genealogy, Portuguese-history, family-history, or relevant community discussions where multiple distinct researchers describe researching Portuguese ancestry. Look for recurring experiences involving difficult-to-read records, finding the correct parish, navigating archives, searching indexes, dealing with spelling differences, requesting documents, or discovering records that were not digitised. Open the actual discussions and record their exact URLs. Summarise practical observations and recurring mistakes as "community-reported observations rather than authoritative archival guidance." Do not use community anecdotes to establish record availability or historical facts when official sources provide different information. | ||||
| 100% | 0% | 0% | 100% | |
How to passCreate a guide to at least ten researched Portuguese genealogy resources, grouping them by function or source type—such as original-record collections, archive catalogues and finding aids, indexes or genealogy databases, newspapers, maps/gazetteers, and contextual research resources. Within each category, identify the strongest resources using transparent, source-supported criteria such as primary-source value, documented coverage, public accessibility, searchability, digital-image availability, geographic usefulness, and limitations. If an overall beginner-priority order is provided, explain that it reflects likely usefulness in a typical research workflow rather than claiming that fundamentally different resource types are directly equivalent. Include the best-use scenario and major limitation for every resource, a reusable beginner workflow from known ancestor/location toward stronger documentary evidence, exact source URLs, and a "Researched On" field noting that interfaces, digitisation coverage, and access conditions can change. | ||||
Failure modes on this task
8 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | — | 1 | 2 | — |
| Reaching the page1 mode | — | — | 1 | — |
| No signal in the judge’s words1 mode · the grader | 1 | 2 | — | 1 |