Weighted Verifier Score
Every verifier carries its own weight, so a reward is the share of a task a trial cleared — and a failed verifier costs its own weight, nothing more.
- 1351verifiers
- 69scored tasks
What counts as a pass
A trial's score is the share of the task's verifier weight it earned. There is no threshold to clear and no penalty for missing one verifier: every verifier carries a published weight, a trial collects the weight of the ones it satisfied, and the reward is that fraction. Missing a core verifier does not scale that fraction down: a trial that produced the workbook, got most of it right and omitted one required audit row keeps everything it earned. Whether a trial cleared every core verifier without exception is a separate question, published separately as the all-core rate, because it answers a different one: not how much of the work landed, but whether the deliverable could be handed on.
The verifiers
2 kinds of check, and what each is for. Every verifier on a task is exactly one of them.
core932 verifiers
What the deliverable is not usable without. The producing team set these per task, not as a house ratio: a required figure that another figure is computed from carries more than one that stands alone, and a task whose entire answer is one reconciliation puts nearly all its weight in this tier. They are what a reviewer would send the work back for.
W_core × (core passed, by weight)
secondary419 verifiers
The judgment around the answer — the alternative that was considered and rejected, the rule that governed the call, the summary a reader sees before the rows. Weighted well below core because a memo missing its rationale is still a memo, but weighted at all because on this benchmark the argument is itself the professional work, and a scheme that ignored it would score a spreadsheet and call it a deliverable.
W_secondary × (secondary passed, by weight)
Reward shape · formula
- W_core
- 90.2% of a task on average · 0.835–0.980 across tasks
- W_secondary
- 9.8% of a task on average · 0.020–0.165 across tasks
The W’s are the task’s own verifier weights summed by tier, not platform constants — they are set per task and sum to 1.00 on every one, so a reward is always a share of a whole task.
No gate. Reward equals the weight passed on every one of the 72 trials this drop publishes check by check, including the 65 that missed a core verifier. Missing one costs that verifier's weight and the trial's place in the all-core pass rate, which is published beside the reward rather than multiplied into it.
Weights
Shares of one whole task, averaged across the 3 tasks that publish their verifier weights — not summed, since each task's weights already sum to 1.00. The range beside a share is its spread across those tasks: these are set per task, not as a house ratio.
The weighting is meant to make a reward readable as how much of this task the trial actually did, rather than as a count of verifiers it happened to satisfy.
By grading roleWhether a verifier reads the artifact or the way it was reached
- outcome80.5%37
- process11.5%8
- mixed8.0%6
The trailing figure is the number of verifiers in that slice.
Score anatomy · same task, two trials
Both trials below are Gemini 3.1 Pro on task_53. The gap between them is variance in one model on one task, not a difference between models.
trial_3missed a core verifier
- 5 / 13core
- 3 / 4secondary
- All-core pass rate sparse
- 0%8 core verifiers failed → 0
- Verifiers passed dense
- 47.1%8 / 17 verifiers, unweighted
- Reward weighted
- 47%
0.835 · 0.413 + 0.165 · 0.758
trial_1missed a core verifier
- 1 / 13core
- 3 / 4secondary
- All-core pass rate sparse
- 0%12 core verifiers failed → 0
- Verifiers passed dense
- 23.5%4 / 17 verifiers, unweighted
- Reward weighted
- 17.5%
0.835 · 0.060 + 0.165 · 0.758
Each term above is the tier's share of the task times the share of that tier the trial cleared, so the two multiply out to the reward beside them.
Both trials below are Claude Opus 5 on task_111. The gap between them is variance in one model on one task, not a difference between models.
trial_1missed a core verifier
- 14 / 15core
- 1 / 1secondary
- All-core pass rate sparse
- 0%1 core verifier failed → 0
- Verifiers passed dense
- 93.8%15 / 16 verifiers, unweighted
- Reward weighted
- 93%
0.980 · 0.929 + 0.020 · 1.000
trial_3missed a core verifier
- 12 / 15core
- 1 / 1secondary
- All-core pass rate sparse
- 0%3 core verifiers failed → 0
- Verifiers passed dense
- 81.3%13 / 16 verifiers, unweighted
- Reward weighted
- 82%
0.980 · 0.816 + 0.020 · 1.000
Each term above is the tier's share of the task times the share of that tier the trial cleared, so the two multiply out to the reward beside them.
Both trials below are Claude Sonnet 5 on task_111. The gap between them is variance in one model on one task, not a difference between models.
trial_1missed a core verifier
- 11 / 15core
- 1 / 1secondary
- All-core pass rate sparse
- 0%4 core verifiers failed → 0
- Verifiers passed dense
- 75%12 / 16 verifiers, unweighted
- Reward weighted
- 75%
0.980 · 0.745 + 0.020 · 1.000
trial_4missed a core verifier
- 8 / 15core
- 1 / 1secondary
- All-core pass rate sparse
- 0%7 core verifiers failed → 0
- Verifiers passed dense
- 56.3%9 / 16 verifiers, unweighted
- Reward weighted
- 57%
0.980 · 0.561 + 0.020 · 1.000
Each term above is the tier's share of the task times the share of that tier the trial cleared, so the two multiply out to the reward beside them.
Both trials below are Gemini 3.1 Pro on task_53. The gap between them is variance in one model on one task, not a difference between models.
trial_3missed a core verifier
- 5 / 13core
- 3 / 4secondary
- All-core pass rate sparse
- 0%8 core verifiers failed → 0
- Verifiers passed dense
- 47.1%8 / 17 verifiers, unweighted
- Reward weighted
- 47%
0.835 · 0.413 + 0.165 · 0.758
trial_1missed a core verifier
- 1 / 13core
- 3 / 4secondary
- All-core pass rate sparse
- 0%12 core verifiers failed → 0
- Verifiers passed dense
- 23.5%4 / 17 verifiers, unweighted
- Reward weighted
- 17.5%
0.835 · 0.060 + 0.165 · 0.758
Each term above is the tier's share of the task times the share of that tier the trial cleared, so the two multiply out to the reward beside them.
Both trials below are Gemini 3.6 Flash on task_111. The gap between them is variance in one model on one task, not a difference between models.
trial_3missed a core verifier
- 8 / 15core
- 1 / 1secondary
- All-core pass rate sparse
- 0%7 core verifiers failed → 0
- Verifiers passed dense
- 56.3%9 / 16 verifiers, unweighted
- Reward weighted
- 54%
0.980 · 0.531 + 0.020 · 1.000
trial_1missed a core verifier
- 4 / 15core
- 1 / 1secondary
- All-core pass rate sparse
- 0%11 core verifiers failed → 0
- Verifiers passed dense
- 31.3%5 / 16 verifiers, unweighted
- Reward weighted
- 29%
0.980 · 0.276 + 0.020 · 1.000
Each term above is the tier's share of the task times the share of that tier the trial cleared, so the two multiply out to the reward beside them.
Both trials below are GPT-5.6 Sol on task_111. The gap between them is variance in one model on one task, not a difference between models.
trial_1missed a core verifier
- 14 / 15core
- 1 / 1secondary
- All-core pass rate sparse
- 0%1 core verifier failed → 0
- Verifiers passed dense
- 93.8%15 / 16 verifiers, unweighted
- Reward weighted
- 93%
0.980 · 0.929 + 0.020 · 1.000
trial_3missed a core verifier
- 9 / 15core
- 1 / 1secondary
- All-core pass rate sparse
- 0%6 core verifiers failed → 0
- Verifiers passed dense
- 62.5%10 / 16 verifiers, unweighted
- Reward weighted
- 64%
0.980 · 0.633 + 0.020 · 1.000
Each term above is the tier's share of the task times the share of that tier the trial cleared, so the two multiply out to the reward beside them.
Both trials below are GPT-5.6 Terra on task_53. The gap between them is variance in one model on one task, not a difference between models.
trial_2all core passed
- 13 / 13core
- 4 / 4secondary
- All-core pass rate sparse
- 100%every core verifier passed
- Verifiers passed dense
- 100%17 / 17 verifiers, unweighted
- Reward weighted
- 100%
0.835 · 1.000 + 0.165 · 1.000
trial_3missed a core verifier
- 11 / 13core
- 3 / 4secondary
- All-core pass rate sparse
- 0%2 core verifiers failed → 0
- Verifiers passed dense
- 82.4%14 / 17 verifiers, unweighted
- Reward weighted
- 81%
0.835 · 0.820 + 0.165 · 0.758
Each term above is the tier's share of the task times the share of that tier the trial cleared, so the two multiply out to the reward beside them.