I am an international student researching Master's programsScience and Education
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
82%
Claude Opus 5
64%
Claude Opus 5.5
64%
GPT-6 sol
27%
GPT-5.6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
21m
Claude Opus 5
19m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
85
Claude Opus 5
60
Claude Opus 5.5
53
GPT-6 sol
12
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (11) | ||||
| 0% | 0% | 0% | 0% | |
How to passEstablish a hypothetical applicant profile including undergraduate background, computer science interests, programming experience, preferred specialization, language ability, and career objectives. Identify which characteristics should influence university selection. | ||||
| 100% | 100% | 0% | 100% | |
How to passResearch German universities offering Master's programs in Computer Science or closely related fields. Select programs based on documented program content, specializations, teaching language, degree structure, and admission requirements rather than university reputation or general rankings. Record the specific academic areas each program supports and note when information is not stated. | ||||
| 100% | 100% | 100% | 100% | |
How to passUse official university sources to compare undergraduate prerequisites, required credits, language requirements, aptitude tests where applicable, application documents, and other eligibility requirements. Highlight requirements that could disqualify an applicant before they spend time preparing an application. | ||||
| 100% | 100% | 0% | 100% | |
How to passExamine module catalogs, specialization options, electives, thesis opportunities, practical projects, and research components. Compare what students can actually study rather than relying on program descriptions. | ||||
| 100% | 100% | 0% | 100% | |
How to passResearch departments, research groups, laboratories, professors, and publicly documented projects related to the student's interests. Determine whether the program provides meaningful opportunities to pursue those areas. | ||||
| 0% | 0% | 0% | 0% | |
How to passCompare cities based on transportation, housing environment, student infrastructure, technology ecosystem, international student support, and proximity to relevant industries. Avoid relying solely on generic “best city” rankings. | ||||
| 100% | 100% | 100% | 100% | |
How to passResearch official information about tuition, semester contributions, administrative fees, student services, and other recurring study-related expenses. Distinguish university charges from general living costs. | ||||
| 100% | 100% | 100% | 100% | |
How to passInvestigate whether applications are submitted directly to the university or through another platform, whether documents require specific formats or verification, and whether additional steps are required for international applicants. | ||||
| 100% | 100% | 0% | 100% | |
How to passInvestigate university career services, industry collaborations, research partnerships, internships, and publicly available graduate or employment information. Avoid assuming that the presence of a famous company automatically means strong employment outcomes. | ||||
| 0% | 0% | 0% | 100% | |
How to passCreate a transparent scoring framework based on academic fit, curriculum, admission feasibility, research opportunities, location, cost, student support, and career relevance. Explain the reasoning behind the weighting rather than producing arbitrary scores. | ||||
| 0% | 0% | 0% | 100% | |
How to passRecommend a primary option, strong alternatives, and a safer admission option. For every recommendation, explain the academic fit, major advantages, weaknesses, admission risks, and information that should be reconfirmed on the official university website before applying. | ||||
Failure modes on this task
18 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | 1 | 3 | 1 | — |
| Standing up the evidence1 mode | 1 | — | 1 | 1 |
| No signal in the judge’s words1 mode · the grader | 2 | 1 | 6 | 1 |