I'm a beginner chess player and wantGames
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
88%
Claude Opus 5.5
63%
GPT-6 sol
50%
GPT-5.6 sol
13%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
28m
Claude Opus 5
19m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
115
Claude Opus 5
34
GPT-5.6 sol
29
Claude Opus 5.5
17
GPT-6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (8) | ||||
| 100% | 100% | 0% | 0% | |
How to passNavigate to https://www.wikipedia.org/ | ||||
| 100% | 100% | 100% | 0% | |
How to passSearch and open the Wikipedia articles for "Italian Game", "Ruy López", and "Queen's Gambit". For each, record a one-line summary of the core strategic idea and any note on whether the opening is described as suited to or recommended for beginners. If a field isn't stated, record "not shown". | ||||
| 100% | 100% | 100% | 100% | |
How to passNavigate to https://lichess.org/ | ||||
| 0% | 0% | 0% | 0% | |
How to passOpen the lichess opening explorer pages at https://lichess.org/opening/Italian_Game, https://lichess.org/opening/Ruy_Lopez, and https://lichess.org/opening/Queens_Gambit. For each, record the mainline first moves in algebraic notation and any popularity or win-rate stat the page shows. If a stat isn't displayed, record "not shown". | ||||
| 100% | 100% | 100% | 0% | |
How to passNavigate to https://www.chess.com/openings/Italian-Game (the chess.com opening explorer for the Italian Game). | ||||
| 100% | 100% | 100% | 0% | |
How to passOpen the three chess.com opening-explorer pages — https://www.chess.com/openings/Italian-Game, https://www.chess.com/openings/Ruy-Lopez, and https://www.chess.com/openings/Queens-Gambit. For each, record the mainline first moves in algebraic notation shown on the page and the win/draw percentages chess.com reports for White, Black, and draws (or whichever of those the page displays). If a stat isn't displayed, record "not shown". | ||||
| 100% | 0% | 0% | 0% | |
How to passWeigh the three openings against each other on three criteria: how easy the core ideas are for a beginner to grasp, how tactical versus positional the resulting middlegame is, and how popular the opening is at lower levels. Pick the single opening to learn first and a single runner-up. | ||||
| 100% | 0% | 0% | 0% | |
How to passWrite one concise recommendation naming the chosen opening with a clear reason that references the strategic idea, the mainline moves, how tactical versus positional it is, and the popularity/win-rate you recorded, plus the single runner-up and why it came second. Keep it to a few sentences a busy beginner could act on. | ||||
Failure modes on this task
15 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | — | 2 | — | 3 |
| Reaching the page1 mode | 1 | — | 1 | — |
| Standing up the evidence1 mode | — | — | — | 1 |
| No signal in the judge’s words1 mode · the grader | — | 1 | 3 | 3 |