What the results say
Every verifier failure is counted under exactly one mode — 46,292 of them across this drop.
- Failure stage
- 4 of them — WHERE in the work a trial broke, at the earliest point that broke.
- Failure mode
- 8 of them — WHAT went wrong, named, inside one stage. Every mode belongs to exactly one stage.
Most failures land at Verification
The candidate ran to completion and the task's own tests rejected it. It accounts for 40,417 of the drop's 46,292 verifier failures — more than any other stage, and every model on the board is in that number.
GPT-5.6 (sol) carries the smallest share of this drop's failures
22,696 verifier failures out of the 46,242 charged to any model, over 27608 trials — 49% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Its characteristic failure is No candidate produced
Tagged at Solver in 1% of its 27608 trials, against 1% for the median of the other 1 models — the widest gap it opens anywhere in the taxonomy.
It handles Assertion failed better than the field
Tagged at Verification in 72% of its 27608 trials, against 74% for the median of the other 1 models. The same measurement as above, read the other way — a failure reading that only ever accuses is not a reading.
Gemini 3.1 Pro Preview carries the largest share of this drop's failures
23,546 verifier failures out of the 46,242 charged to any model, over 27608 trials — 51% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Its characteristic failure is Assertion failed
Tagged at Verification in 74% of its 27608 trials, against 72% for the median of the other 1 models — the widest gap it opens anywhere in the taxonomy.
It handles No candidate produced better than the field
Tagged at Solver in 1% of its 27608 trials, against 1% for the median of the other 1 models. The same measurement as above, read the other way — a failure reading that only ever accuses is not a reading.
Failure modes
Every run in this drop carries a one-line sandbox verdict, and every failed run carries a reason. These modes are read off that line by prefix: a timeout, a solver that produced no candidate, a dependency install that failed, and the Python exceptions raised before the tests could run. Everything else is the tests rejecting the candidate, which is what the benchmark is for. One label per failed run, so every count here counts runs. On 1,491 runs the reason arrives as a fragment of the assertion report rather than the whole of it; those are counted as verification failures with their evidence missing, not as a separate mechanism. The environment stage is excluded from model-ranking charts: a sandbox that could not be built says nothing about the model inside it. The bar under each stage is its share of every classified failure — how much of the drop it accounts for. The dots are how often each model’s own trials were tagged with it, on one scale across every card. Two different denominators, so a large stage is not the same claim as a model that hits one often.
- Gemini 3.1 Pro Previewhigh
- GPT-5.6 (sol)high
Verification87.3%
The candidate ran to completion and the task's own tests rejected it.
1 mode- Assertion failed3009 tasks · 40417 failures · 87.3%FAILED <test>: <message>, or the tail of an assertion report
Most affected tasks · failures2des_linear_response_validity 16304605_fixed 16304619_fixed 16354004 16354050 16354386-scicode_math_ddr_elasticity_contact_locking_free_001 16
Trials tagged, per model2 pts spread
0%25%50%75%
Gemini 3.1 Pro Previewhigh20467 / 27608 trialsGPT-5.6 (sol)high19950 / 27608 trialsExecution11.6%
The candidate never reached a verdict on its own merits.
5 modes- Syntax error420 tasks · 4768 failures · 10.3%SyntaxError
- Timeout88 tasks · 459 failures · 1.0%Sandbox timeout after 1800s.
- Runtime exception15 tasks · 90 failures · 0.2%<Name>Error, <Name>Exception
- Indentation error3 tasks · 28 failures · 0.1%IndentationError, TabError
- Native crash1 task · 4 failures · <0.1%free(), malloc, Segmentation fault
Most affected tasks · failures354030 16adaptive_edft_hubbard_scaling 16anderson_lattice_topological_phase 16axion_interferometer_g_precision 16bcs_rydberg_ramsey_spectrum 16bgk_mottsmith_shock 16
Trials tagged, per model2 pts spread
0%25%50%75%
Gemini 3.1 Pro Previewhigh2914 / 27608 trialsGPT-5.6 (sol)high2435 / 27608 trialsSolver1.0%
No candidate solution was produced to run.
1 mode- No candidate produced227 tasks · 476 failures · 1.0%Solver failed.
Most affected tasks · failuresdirac_soliton_propagation_bwa 8dispersive_feedforward_cat_state_fidelity 8dm_admixed_ns_tov 8double_layer_fqhe_tunneling_moments 8emri_dark_matter_orbital_evolution 8emri_dehnen_orbital_frequencies 8
Trials tagged, per model1 pts spread
0%25%50%75%
Gemini 3.1 Pro Previewhigh165 / 27608 trialsGPT-5.6 (sol)high311 / 27608 trialsEnvironmentthe grader0.1%
The sandbox could not be built for this run.
1 mode- Dependency install failed15 tasks · 50 failures · 0.1%Dependency installation failed
Most affected tasks · failuresscicode_colab_1479 16atomic_physics_6d2e94 8physics_063 8chemistry_319 5bio_pde_cpm_motility_001 2chemistry_399 2
Trials tagged, per model0 pts spread
0%25%50%75%
Gemini 3.1 Pro Previewhigh23 / 27608 trialsGPT-5.6 (sol)high27 / 27608 trials
Failure profile
How often each model broke down at each stage of the work, as a share of its trials — a different measurement from the counts below, and the one that compares models. The chart opens on the stages the field does worst at and on the models furthest apart there; pin a model in the picker and it re-picks to that model’s own worst. Every stage and every model stays one chip away.
Point at a shape — or a model above — to read its values.
Incidence — model × failure stageTrials tagged with each plotted stage, as a share of the trials that model made. The figure beside a stage is its spread — the gap between the model that hits it most and the one that hits it least.
1Verification2 pts
2Execution2 pts
3Solver1 pts