Develop a 6-month language learning planScience and Education
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5
80%
Claude Opus 5.5
20%
GPT-5.6 sol
0%
GPT-6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
74m
Claude Opus 5
24m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
110
Claude Opus 5
24
GPT-6 sol
14
GPT-5.6 sol
11
Claude Opus 5.5
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (5) | ||||
| 100% | 0% | 0% | 100% | |
How to passDefine the learner as an English speaker aiming for practical conversational Japanese over six months. Establish the starting level, available weekly study time, preferred learning format, motivation, and target situations for conversation. Define measurable outcomes for listening, speaking, reading, writing, vocabulary, grammar, kana, and pronunciation. Set a realistic six-month proficiency target rather than assuming full fluency; for example, distinguish basic survival conversation from broader conversational independence. Use an established proficiency framework where appropriate, such as the Japan Foundation's JF Standard. Make clear which outcomes are targets for the end of six months and which would require longer study. | ||||
| 100% | 0% | 0% | 100% | |
How to passCreate a structured six-month progression covering kana, pronunciation, core grammar, vocabulary, listening, speaking, reading, and writing. Organize the plan by month and then by weekly study pattern, showing what should be learned, practised, reviewed, and assessed at each stage. Make speaking and listening central to the conversational goal rather than treating them as activities added after grammar and vocabulary. Include spaced review and cumulative practice so earlier material continues to be used. Where a resource has a stated proficiency level, use that information when sequencing it; for example, Japan Foundation resources explicitly map materials such as Irodori and Marugoto to JF/CEFR-style levels. | ||||
| 0% | 0% | 100% | 100% | |
How to passResearch at least three textbook options and at least three online-course options suitable for an English-speaking beginner or early-stage learner. Use official publisher/provider pages and reputable educational sources. For every option, record provider/publisher, level, curriculum structure, grammar coverage, vocabulary, kanji/kana coverage, listening and speaking practice, exercise format, estimated study time, delivery format, cost where publicly stated, prerequisites, assessment method, strengths, limitations, and progression pathway. Do not treat a resource as equivalent to another merely because both describe themselves as “beginner Japanese.” Include free and paid options where relevant without purchasing or enrolling. For example, the Japan Foundation's Marugoto platform offers self-study and tutor-supported courses, while Irodori is designed around practical everyday communication and provides downloadable materials and audio. | ||||
| 100% | 0% | 0% | 100% | |
How to passResearch at least three options for each of the following resource categories: vocabulary tools, grammar resources, and pronunciation/listening practice. For each option, record the learning method, Japanese level, content covered, review mechanism, audio quality or speech model where documented, opportunities for active production, feedback or correction, platform availability, cost, limitations, and suitability for the six-month plan. Distinguish passive exposure from active recall and speaking practice. Do not assume that a vocabulary app teaches grammar or that a grammar reference provides sufficient conversational practice. Select resources that complement rather than unnecessarily duplicate the core course. | ||||
| 100% | 0% | 0% | 100% | |
How to passResearch at least three language-exchange, tutoring, conversation-practice, or structured speaking options. Compare how learners actually obtain speaking time, whether interaction is peer-to-peer or teacher-led, correction/feedback mechanisms, scheduling, level matching, safety or moderation information where publicly documented, free versus paid access, and whether the service requires an account. Include the practical limitations of relying on language exchange alone, such as inconsistent partners or uneven correction. Identify which option is best suited to regular weekly speaking practice and which could serve as a backup. Do not enroll, contact other users, or create an account. | ||||
Failure modes on this task
10 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | — | 3 | 3 | — |
| Standing up the evidence1 mode | 1 | — | 1 | — |
| No signal in the judge’s words1 mode · the grader | — | 2 | — | — |