Chemistry
Frozen-Core Spin-Component-Scaled RMP2 Analytic Nuclear Gradient
# Frozen-Core Spin-Component-Scaled RMP2 Analytic Nuclear Gradient
STEM Scientific Coding
6,781 scientific coding tasks across physics, materials science, chemistry, mathematics and biology — 3,451 of them scored by both frontier models below. Each is a main problem decomposed into 2–4 subproblems with structured reasoning and executable test cases, calibrated so frontier models land across pass bands rather than clustering at ceiling or floor.
STEM Scientific Coding
6,781 scientific coding tasks across physics, materials science, chemistry, mathematics and biology — 3,451 of them scored by both frontier models below. Each is a main problem decomposed into 2–4 subproblems with structured reasoning and executable test cases, calibrated so frontier models land across pass bands rather than clustering at ceiling or floor.
STEM Scientific Coding
6,781 scientific coding tasks across physics, materials science, chemistry, mathematics and biology — 3,451 of them scored by both frontier models below. Each is a main problem decomposed into 2–4 subproblems with structured reasoning and executable test cases, calibrated so frontier models land across pass bands rather than clustering at ceiling or floor.
- 6,781
- Tasks in SciCode corpus
- 3,451
- Tasks measured on both models
- 2
- Models
- 8
- Trials per task
The Evidence
Explore all tasksFive of the 3,451 measured problems, published in full — the brief a model is handed, every subproblem it must implement, and the verifiers it was graded on. The rest of the corpus is indexed with its scores.
Where GPT-5.6 (sol) breaks
Explore all tasksFive of the 3,451 measured problems, published in full — the brief a model is handed, every subproblem it must implement, and the verifiers it was graded on. The rest of the corpus is indexed with its scores.
Where Gemini 3.1 Pro Preview breaks
Explore all tasksFive of the 3,451 measured problems, published in full — the brief a model is handed, every subproblem it must implement, and the verifiers it was graded on. The rest of the corpus is indexed with its scores.
Physics
FKCH History-State Preparation with Adversary-Decoupling and Trotterized Locality
Calculate the final logical-state fidelity and certify CPTP/normalization invariants for a coupled universal-quantum-computing model that combines (i) a sequential adversary interaction (QTM/MPS-style) encoding stage with explicit decoupling, and (ii) a geometrically local Hamiltonian (QCA-style) logical-gate stage implemented by first-order Lie–Trotter product evolution, and (iii) a Feynman–Kitaev clock-history Hamiltonian (FKCH) constraint enforced self-consistently.
Mathematics
Sketched Block Krylov QR for a Stiff Chebyshev Collocation System
Calculate the sketched block-GMRES approximation of a stiff Chebyshev-Lobatto collocation linear system while certifying the coupled QR, Krylov, and residual invariants of the randomized block basis.
Physics
FKCH History-State Preparation with Adversary-Decoupling and Trotterized Locality
Calculate the final logical-state fidelity and certify CPTP/normalization invariants for a coupled universal-quantum-computing model that combines (i) a sequential adversary interaction (QTM/MPS-style) encoding stage with explicit decoupling, and (ii) a geometrically local Hamiltonian (QCA-style) logical-gate stage implemented by first-order Lie–Trotter product evolution, and (iii) a Feynman–Kitaev clock-history Hamiltonian (FKCH) constraint enforced self-consistently.
Mathematics
Sketched Block Krylov QR for a Stiff Chebyshev Collocation System
Calculate the sketched block-GMRES approximation of a stiff Chebyshev-Lobatto collocation linear system while certifying the coupled QR, Krylov, and residual invariants of the randomized block basis.
Biology
Bifurcation and Full-Spectrum Stability Analysis of the Mini-Ultimatum Game with Information
Analyze the mini-ultimatum game with information under replicator dynamics. The four strategies are indexed
Chemistry
Frozen-Core Spin-Component-Scaled RMP2 Analytic Nuclear Gradient
# Frozen-Core Spin-Component-Scaled RMP2 Analytic Nuclear Gradient
Biology
Bifurcation and Full-Spectrum Stability Analysis of the Mini-Ultimatum Game with Information
Analyze the mini-ultimatum game with information under replicator dynamics. The four strategies are indexed
Material Science
Scaled tight binding model for a two dimensional electron gas at the (001) LaAlO₃/SrTiO₃ interface
Compute the band structure of the (001) LaAlO_3/SrTiO_3 two-dimensional electron gas along a specified k-path, then build the corresponding scaled real-space tight-binding Hamiltonian on a finite periodic lattice and verify that both representations yield identical low-energy spectra at scaling factor s=1.
Evaluated against
- Gemini 3.1 Pro PreviewGoogle · high reasoning · scicode-pass8-eval
- GPT-5.6 (sol)OpenAI · high reasoning · scicode-pass8-eval
How frontier models actually score
- 2 models
- Main problem pass@8
Reference scores use pass@8 — eight independent trials per task — with Gemini 3.1 Pro Preview and GPT-5.6 (sol), both at reasoning effort high. Tasks are calibrated so scores spread across pass bands rather than clustering at ceiling or floor.
Aggregate main-problem pass@8
| Statistic | Value |
|---|---|
| Mean | 24% |
| Median | 0% |
n = 3,451 measured tasks
Difficulty spread (main problem)
Tasks are calibrated so frontier models land across pass bands — not clustered at ceiling or floor.
| Main problem pass@8 | Share of tasks |
|---|---|
| 0% | 67.3% |
| Above 0%, below 40% | 7.1% |
| 40%–69% | 5.2% |
| 70%–99% | 5.8% |
| 100% | 14.7% |
n = 3,451 measured tasks
Mean main pass@8 by domain
| domain | Main problem pass@8 |
|---|---|
| Physics | 23.8% |
| Material Science | 28.6% |
| Chemistry | 19.4% |
| Mathematics | 22.4% |
| Biology | 25.9% |
n = 3,447 tasks · 5 domains
Task pass rate by model
Full leaderboardThe share of each model's trials that passed, graded by execution verdict.
Measured on 3,451 tasks · 55,216 trials · 2 models
Grounded in five STEM domainsGrounded in five STEM domainsGrounded in five STEM domains
Full taxonomyPhysics leads at 47.0% of the delivery — if you're evaluating physics-heavy reasoning, this is the best-covered domain to pull samples from.
Physics leads at 47.0% of the delivery — if you're evaluating physics-heavy reasoning, this is the best-covered domain to pull samples from.
Physics leads at 47.0% of the delivery — if you're evaluating physics-heavy reasoning, this is the best-covered domain to pull samples from.
5STEM domains
- Physics
- 47%
- Material Science
- 19.1%
- Chemistry
- 14.4%
- Mathematics
- 12.4%
- Biology
- 7.2%
5 STEM domains · 114 subdomains
Subproblem distribution
| subproblems per task | Tasks | Share |
|---|---|---|
| 0 | 4 | 0.1% |
| 2 | 2,415 | 70% |
| 3 | 859 | 24.9% |
| 4 | 173 | 5% |
n = 3,451 tasks · 8,099 subproblems in total
What the numbers say
The strictest reading on this page: a trial counts only where every core verifier passed, so one missed check costs the whole trial. The rates above ask a gentler question — what share of a task’s verifiers each model cleared — which is why they are higher.
What the numbers say
The strictest reading on this page: a trial counts only where every core verifier passed, so one missed check costs the whole trial. The rates above ask a gentler question — what share of a task’s verifiers each model cleared — which is why they are higher.
What the numbers say
The strictest reading on this page: a trial counts only where every core verifier passed, so one missed check costs the whole trial. The rates above ask a gentler question — what share of a task’s verifiers each model cleared — which is why they are higher.
GPT-5.6 (sol)
17.7%
best model's pass rate
Across 3451 tasks and 2 frontier models, GPT-5.6 (sol) clears 17.7% of trials outright — there is no partial credit here, so a task is solved or it is not.
GPT-5.6 (sol)
17.7%
pass rate
Across 3451 tasks and 2 frontier models, GPT-5.6 (sol) clears 17.7% of trials outright — there is no partial credit here, so a task is solved or it is not.
Gemini 3.1 Pro Preview
14.6%
pass rate
Across 3451 tasks and 2 frontier models, Gemini 3.1 Pro Preview clears 14.6% of trials outright — there is no partial credit here, so a task is solved or it is not.
Failure modes
Failure modes
Failure modes
Every failed verifier was reviewed and filed under exactly one failure mode, at the earliest point in the pipeline that broke. What follows is the reading those labels support — the field’s while nothing is pinned, one model’s against the rest once you pick one.
Every failed verifier was reviewed and filed under exactly one failure mode, at the earliest point in the pipeline that broke. What follows is the reading those labels support — the field’s while nothing is pinned, one model’s against the rest once you pick one.
Every failed verifier was reviewed and filed under exactly one failure mode, at the earliest point in the pipeline that broke. What follows is the reading those labels support — the field’s while nothing is pinned, one model’s against the rest once you pick one.
Most failures land at Verification
The candidate ran to completion and the task's own tests rejected it. It accounts for 40,417 of the drop's 46,292 verifier failures — more than any other stage, and every model on the board is in that number.
GPT-5.6 (sol) carries the smallest share of this drop's failures
22,696 verifier failures out of the 46,242 charged to any model, over 27608 trials — 49% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Its characteristic failure is No candidate produced
Tagged at Solver in 1% of its 27608 trials, against 1% for the median of the other 1 models — the widest gap it opens anywhere in the taxonomy.
It handles Assertion failed better than the field
Tagged at Verification in 72% of its 27608 trials, against 74% for the median of the other 1 models. The same measurement as above, read the other way — a failure reading that only ever accuses is not a reading.
Gemini 3.1 Pro Preview carries the largest share of this drop's failures
23,546 verifier failures out of the 46,242 charged to any model, over 27608 trials — 51% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Its characteristic failure is Assertion failed
Tagged at Verification in 74% of its 27608 trials, against 72% for the median of the other 1 models — the widest gap it opens anywhere in the taxonomy.
It handles No candidate produced better than the field
Tagged at Solver in 1% of its 27608 trials, against 1% for the median of the other 1 models. The same measurement as above, read the other way — a failure reading that only ever accuses is not a reading.
Top subdomains
Full taxonomyHow the pack is built
Sophisticated task design
Calibrated to frontier-model pass bands; 2–4 subproblems per main problem, with structured reasoning steps and executable test cases.
Elite scientific talent
Created by top-tier practicing PhDs and professors who can code, spanning condensed matter physics to computational chemistry.
Multi-stage quality control
Two-level review by domain experts, robust agentic review, and comprehensive test validation.
Production-ready delivery
6,781 SciCode STEM tasks available, 3,451 of them scored by both models in this release, through a productionized pipeline built for scale and quality; OTS bundles available in 24–48 hours.