Terminal-Bench 3.0 Workflow & Quality — Turing Benches

§2QC gates

A task advances only when the criterion is met. A failure sends it back — nothing ships on reviewer intuition alone.

Stage	Gate	Who runs it	Pass criterion	On failure
Oracle run	The reference solution scores full reward	Producer	Every task ships a reference solution, and it is run through the exact verifier the models face, in the same isolated container. A task ships only if its own solution passes.	A task whose reference solution does not pass is not a hard task, it is a broken one, and it does not enter the suite.
Empty-solution check	An empty solution fails	Producer	The verifier is run against an empty `solve.sh`. It must fail — a check that passes on nothing is measuring nothing.	A verifier that accepts an empty solution is rewritten before the task is scored.
Adversarial review	The verifier resists a plausible wrong answer	Reviewer	A plausible solution that does not actually work must fail, and the verifier must grade outcomes through execution rather than by reading the source.	A verifier that can be satisfied without doing the work is treated as a reward-hacking route and re-cut.
Determinism	The verifier is re-runnable and stable	Producer	The same submission scores the same way on a re-run, so a rank movement between drops is a change in the models rather than in the grader.	A verifier that disagrees with itself is fixed before its task counts.
Both directions	No false passes, and no false failures	Reviewer	The rubric checks the verifier in both directions, so a correct solution is not rejected and an incorrect one is not accepted.	Either direction failing sends the task back to its author.
Attribution	A failure is the agent's, not the harness's	Producer	A trial that could not finish for an infrastructure reason is recorded separately and excluded from every share, rather than counted as a score of zero.	A harness fault read as a model failure would make the board a measure of the runner.

§3QC results

Outcomes of the process above, as of 2026-08-13.

100
tasks in the published suite, each with a reference solution
Tasks
11
model + scaffold configs, every one run over the whole suite
Leaderboard
6400
scored attempts behind every figure on this site
Leaderboard
8
attempts at each task, per config — reliability is measurable, not inferred