OdysseysMega
Switch benchOdysseysMega
Request dataset samplesby and Superintelligent Agents

Build a reusable long-term video game ownershipGames

15 rubrics · 15 LLM-graded

TaxonomyGamesVideo Games Consoles and Accessories

How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.

Core metrics

Check Pass %

Mean share of the task's checks passing, averaged over runs. N = 1 run per model.

  1. 100%

    Claude Opus 5.5

  2. 100%

    Claude Opus 5

  3. 27%

    GPT-5.6 sol

  4. 7%

    GPT-6 sol

Extended metrics

Median Time

Wall-clock minutes, median across runs. N = 1 run per model.

  1. 62m

    Claude Opus 5

  2. 19m

    GPT-5.6 sol

Tool Calls

Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.

  1. 114

    Claude Opus 5

  2. 25

    Claude Opus 5.5

  3. 21

    GPT-6 sol

  4. 9

    GPT-5.6 sol

Verifier × model pass rate

How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.

VerifierClaude Opus 5.5n=1GPT-6 soln=1GPT-5.6 soln=1Claude Opus 5n=1
checks (15)
100%0%100%100%

How to pass

Start by defining what “long-term usable” means for a video game. Research official platform documentation and recognized digital-preservation guidance to identify relevant dimensions such as access, compatibility, ownership, functionality, dependency, and preservation. Distinguish between owning a physical object, owning a digital license, having continued access to a game, and being able to use all originally advertised functionality. Include fictional examples where a physical disc remains in someone's possession but requires online activation, and another where a digital game can be downloaded but depends entirely on an account. Explain why physical possession does not automatically equal permanent access. Define the intended evaluation horizon without relying on a specific future date. Create a standard profile covering platform, edition, distribution method, account requirements, online dependencies, DLC, save-data behavior, compatibility, accessibility, and hardware dependencies. Establish which questions can be answered from current documentation and which inherently involve future uncertainty. The framework should evaluate resilience rather than make unrealistic promises about permanent access.

100%0%0%100%

How to pass

Build an evidence hierarchy for long-term ownership claims. Prioritize official console documentation, publisher or developer support pages, platform policies, product manuals, compatibility documentation, preservation organizations, and recognized technical standards. Distinguish documentation describing current functionality from statements about future support. Create an evidence register containing claim, source, publication or update information where available, platform, evidence type, confidence, and limitations. Include a fictional case where a retailer describes a game as “complete” while the publisher's documentation reveals additional downloadable content. Add another where a community-maintained compatibility list contains useful information that is not documented by the manufacturer. Explain how community evidence can be useful without automatically becoming authoritative. Also establish how to treat contradictory official sources, such as an older support article versus a newer platform document. The purpose is to ensure that every important conclusion in the ranking can be traced to appropriate evidence. Avoid treating marketing language such as “definitive,” “complete,” or “ultimate” as technical proof of long-term ownership resilience.

100%0%0%100%

How to pass

Design a standardized intake profile for each game or edition being evaluated. Record title, developer or publisher, platform, edition, release format, physical/digital status, required accounts, online requirements, downloadable content, updates, save-data model, compatibility, accessibility features, and relevant hardware dependencies. Include a field for documentation completeness and another for unresolved questions. Create fictional examples involving a standard edition, complete edition, remaster, physical release, and digital-only release. Preserve the exact edition name because similarly titled releases can have materially different contents. Record whether DLC is included, separately licensed, or unavailable through the base product. Do not assume that a physical release contains every required component. Also identify whether functionality may depend on an external platform service. The intake profile should make it immediately clear what is actually being evaluated. A stranger should be able to enter a new game and understand which information must be collected before any ranking is attempted.

100%0%100%100%

How to pass

Compare physical and digital distribution from a long-term resilience perspective. Research official platform and publisher documentation regarding installation, authentication, licensing, updates, account requirements, and offline functionality. Identify the practical differences between possessing physical media and retaining a digital license. Include fictional examples where a physical copy provides strong offline functionality, another requires an online check, and a digital edition remains downloadable only through a platform account. Avoid making blanket claims that physical games are always superior for preservation or that digital games are always less durable. Instead, identify the dependencies that determine the outcome. Record media condition, installation requirements, authentication, patch requirements, account dependency, and whether the game can operate without network access where documented. Also distinguish between the ability to launch a game and the ability to access every feature. The final comparison should explain why ownership resilience depends on the entire access chain rather than simply whether the product was purchased on a disc or digitally.

100%0%0%100%

How to pass

Create a framework for identifying external services that may be required to access or use the game. Research official documentation concerning account linking, authentication, online services, platform accounts, publisher accounts, cloud services, and required network connections. Record each dependency, what functionality it controls, whether it is required for installation, launch, multiplayer, DLC, updates, or save synchronization, and what happens if the service becomes unavailable according to documented information. Include fictional cases where the base game works offline but DLC authentication does not, and another where a publisher account is required even for a primarily single-player experience. Do not speculate that a service will definitely shut down. Instead, identify the dependency and classify the resulting future uncertainty. Create risk categories such as low dependency, moderate dependency, substantial dependency, and critical dependency based on documented functionality. This section should help distinguish games that merely benefit from online services from games whose basic usability depends on them. It should also make future changes easier to reassess.

100%0%0%100%

How to pass

Develop a method for determining what content is actually included with each edition. Research official publisher and platform documentation. Record base game content, expansions, DLC, patches, bonus content, online-only content, and edition-specific differences. Include fictional cases where a “complete” edition contains most DLC but not a later expansion, and another where a physical edition requires a substantial download before becoming fully playable. Distinguish between content physically present on media, content licensed through an account, and content delivered through updates. Avoid assuming that a product label establishes technical completeness. Create a content matrix showing each major component and its access dependency. Record whether the content is essential, optional, cosmetic, or functionality-changing. The goal is to prevent misleading comparisons where one edition appears cheaper or more complete but depends heavily on future downloads or services. This information should feed directly into the long-term resilience ranking and remain useful when comparing future editions of the same game.

100%0%0%100%

How to pass

Evaluate how likely each version is to remain usable when hardware generations change. Research official backwards-compatibility lists, platform support documentation, emulation or compatibility information from recognized sources, and publisher statements where available. Record native compatibility, official compatibility programs, required upgrades, feature limitations, controller requirements, and save-data migration options. Include fictional examples where a game runs on newer hardware but loses certain features, another requires purchasing a separate remaster, and another supports direct backwards compatibility with save transfer. Do not assume that compatibility means identical functionality. Distinguish between launchability, feature parity, save compatibility, DLC compatibility, and performance. Create a migration-risk assessment identifying which parts of the ownership experience could break during a hardware transition. The objective is to evaluate the whole transition path rather than simply asking whether a game appears on a compatibility list. This should become an important ranking factor for people who want to keep using their library across future hardware generations.

100%0%100%100%

How to pass

Create a framework for assessing whether player progress can survive changes in hardware, editions, or platforms. Research official documentation concerning local saves, cloud saves, account synchronization, transfer utilities, cross-generation saves, cross-platform progression, and backup mechanisms. Record whether save data is portable, account-bound, platform-bound, edition-dependent, or otherwise restricted. Include fictional cases where a game is backwards compatible but saves cannot transfer, and another where saves synchronize through an online service but become inaccessible without the relevant account. Avoid assuming that cloud synchronization automatically means permanent preservation. Identify what happens when services, accounts, or platforms change based on documented evidence. Create a continuity score based on actual dependencies rather than convenience alone. Also distinguish between preserving the game and preserving the player's personal progress. This distinction is important because a game may remain playable while years of progress become difficult or impossible to transfer. The final ranking should therefore consider save continuity as a separate resilience dimension.

100%0%0%100%

How to pass

Create an accessibility-focused evaluation that considers whether the game remains practically usable for players with different needs. Research official platform accessibility documentation, developer accessibility information, and recognized accessibility guidance. Record features such as subtitle controls, text size, color options, remapping, audio controls, visual alternatives, difficulty adjustments, input flexibility, and other documented features relevant to the game. Do not assume that the presence of an accessibility menu means every feature is accessible. Include fictional examples where an older game has fewer modern accessibility options but works reliably offline, while a newer version provides extensive accessibility but introduces significant online dependencies. Keep accessibility as a distinct criterion rather than allowing it to disappear inside a general “quality” score. Identify which features are platform-level and which are game-specific. Record evidence and limitations. The goal is to evaluate long-term usability for the intended player, not simply technical preservation. This makes the ranking more meaningful than a generic compatibility assessment.

100%0%0%100%

How to pass

Evaluate the hardware chain required to keep a game usable. Research official hardware specifications, controller requirements, storage requirements, accessory dependencies, and manufacturer support information. Where relevant, research recognized repairability or preservation guidance without making unsupported predictions about future hardware availability. Include fictional examples where a game requires an unusual peripheral, another depends on a specific controller feature, and another works across multiple standard input methods. Separate essential hardware from optional accessories. Record whether a hardware dependency can be substituted, whether documented compatibility exists, and whether losing the component would materially affect functionality. Avoid current retail availability as a core factor because it changes over time. Instead, evaluate structural dependency: how many specialized components are necessary and how replaceable they are according to documented information. This section should identify preservation risks that are not visible when evaluating software alone. A game can be digitally preserved yet become practically inaccessible if essential hardware cannot be replaced or substituted.

100%0%0%100%

How to pass

Create a structured assessment of what remains usable without external network services. Research official documentation regarding offline modes, required downloads, authentication, updates, multiplayer dependencies, and installation. Include fictional examples where a game is primarily single-player but requires periodic authentication, another works fully offline after installation, and another loses major content when servers are unavailable. Distinguish between offline launch, offline gameplay, offline DLC access, offline save access, and offline installation. Avoid assuming that “single-player” automatically means “offline.” Record documented requirements separately for each functionality. Create a preservation-resilience classification based on the number and importance of external dependencies. Where documentation is unclear, mark the claim as uncertain rather than inferring behavior. The objective is to identify which parts of a game experience are structurally dependent on external systems. This should become one of the most important criteria for long-term evaluation because service dependency can materially change the practical value of an otherwise well-preserved game.

100%0%0%100%

How to pass

Create the core ranking framework using factors such as platform resilience, ownership model, service dependency, DLC completeness, compatibility, save continuity, offline functionality, accessibility, hardware dependency, documentation quality, and uncertainty. Keep personal preference separate from technical resilience. Create distinct decision profiles, such as maximum preservation resilience, best balance of usability and resilience, and best accessibility-preservation balance. Test how rankings change when priorities change. Add decision blockers for severe dependencies that should not be hidden by an otherwise high score. Include a written rationale for every ranking position. Avoid allowing the number of features to automatically determine the winner. A game with fewer features may be more resilient if its essential functionality has fewer external dependencies. Conversely, a highly accessible version may rank higher for a specific user even if it has greater service dependence. The ranking should therefore explain trade-offs rather than claiming one universally superior option. Include a “what could change this ranking?” field so the system can be updated when official compatibility or service information changes.

100%0%0%100%

How to pass

Apply the complete framework to the difficult fictional cases requested in the prompt: a physical game requiring online activation, a complete edition with missing DLC, a backwards-compatible game with inaccessible save data, and a lower-feature version with substantially lower long-term dependency. For each case, evaluate ownership model, account dependency, DLC, updates, compatibility, saves, accessibility, hardware, offline functionality, evidence confidence, and preservation risk. Include at least one case where the correct result is “insufficient evidence” rather than forcing a ranking. Add another where a lower-feature edition ranks higher because it is structurally more resilient. Examine whether the framework rewards genuine long-term resilience rather than novelty or feature count. Use the scenarios to refine the weighting model and decision blockers. The final ranking should demonstrate that materially different ownership profiles can produce different conclusions.

100%100%100%100%

How to pass

Review the system as if you were deliberately trying to produce a misleading “best ownership option.” Check whether physical media receives an unjustified advantage, whether digital editions are automatically penalized, whether feature-rich editions dominate because they have more content, or whether current availability is being confused with long-term resilience. Verify that community reports are not being counted as independent evidence when they originate from the same underlying documentation. Check whether uncertainty is visible and whether future service changes can materially alter the ranking. Examine whether accessibility is being treated as a genuine requirement rather than a cosmetic feature. Remove criteria that cannot be consistently evaluated. Then create a reusable workflow: identify the edition, document dependencies, verify official evidence, map content, evaluate compatibility, assess saves, inspect accessibility, test offline resilience, evaluate hardware dependencies, rank options, document uncertainty, and establish update triggers. The workflow should be reusable for future games without rebuilding the methodology from scratch.

100%0%0%100%

How to pass

Create the final ranked list with reasoning, showing how the fictional game editions or ownership options perform under each decision profile. For every ranking, explain the strongest evidence, important dependencies, preservation advantages, major weaknesses, uncertainty, and what future change could alter the result. Include the evidence hierarchy, intake template, ownership-model comparison, DLC matrix, compatibility framework, save-continuity assessment, accessibility checklist, offline-resilience framework, hardware-dependency analysis, ranking methodology, fictional stress-test results, and update workflow. Clearly distinguish documented facts from community evidence, interpretation, and future uncertainty. Do not use current prices, discounts, popularity, or review scores as core ranking factors. Add a final quality-control checklist confirming that edition differences, account requirements, DLC, saves, compatibility, accessibility, offline functionality, and hardware dependencies were separately examined. Finally, perform the stranger test: someone unfamiliar with the project should be able to evaluate a new game edition, trace its evidence, identify long-term dependencies, compare alternatives, and produce a defensible ranking without asking the task creator for additional instructions.

Failure modes on this task

25 tagged verifier failures, by the stage they were assigned to. Higher is worse.

Failure stageClaude Opus 5.5GPT-6 solGPT-5.6 solClaude Opus 5
Delivering the result1 mode—114—
Standing up the evidence1 mode—14—
Reaching the page1 mode—2——
No signal in the judge’s words1 mode · the grader——3—