Both models ran eight trials on every task they were both given, so these rates are measured on the same 3,451 problems rather than on whatever subset each was handed. Grading is all-or-nothing: a trial passes every test its task defines, or it fails. 2 models.
Last updated 2026-09-09
Total Tasks
3451
each attempted 8× by 2 models
Best Reward
17.7%
GPT-5.6 (sol)high
Model card · evaluated Sep 2026
Reward
17.7%
Tokens
12k
Every verifier failure was filed under one failure mode by a stated rule, not by a reviewer, grouped into the stages above. The shares split this model's own 22696 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake.
Model card · evaluated Sep 2026
Reward
14.6%
Tokens
8k
Every verifier failure was filed under one failure mode by a stated rule, not by a reviewer, grouped into the stages above. The shares split this model's own 23546 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake.
Tokens per trial, averaged over every run the drop recorded them on. Read the two bars against the board above: they are the only other dimension this delivery measures, and the models are not close on them. There is no clock and no price in the drop, so cost per task is not a number this bench can publish.
Input tokens consumed, averaged across trials and tasks — cumulative across turns, not peak context
2k
Gemini 3.1 Pro Previewhigh
2k
GPT-5.6 (sol)high
Output tokens generated, averaged across trials and tasks
10k
GPT-5.6 (sol)high
6k
Gemini 3.1 Pro Previewhigh