Workflow & Quality
How this bench is built, checked, and solved. Process is documented per bench — every figure below is dated and links to its evidence.
§1Task lifecycle
Every task travels the same pipeline. The gates are where it can be sent back.
- Authored by hand
- Quality gate
- Automated
Session 1 · Creation
Authored by handDomain expert
Choose a real workflow
A domain expert picks a job they actually do, in one of the nine sectors in scope, and works from the operational reality of it rather than from a description of it.
Authored by handDomain expert
Write the prompt
The brief names each deliverable at an exact path and states what the work has to get right, in the words the job itself would use.
Authored by handAI assistant
Draft the source documents
An assistant drafts the working files the task hands over — bills of material, price masters, certification snapshots, policy excerpts.
Authored by handDomain expert
Refine and validate the sources
The expert takes the drafts back and makes them true to the job, including the disagreements between files that the real work has to resolve.
Session 2 · Run and evaluation
AutomatedModel under test
Run the model evaluation
The task is put to the model under test in a fresh session, with none of the authoring conversation behind it.
Quality gateDomain expert
Check for headroom
The expert reads the output against the standard their own work would be held to. A task is only worth grading if there is room between the two.
No headroom — the model already matches expert quality, and the task is dropped rather than shipped as one every model passes.
Authored by handDomain expert
Build the golden solution
The expert produces the deliverables themselves. Three reference outputs per task ship for grading and are held back from everything the agent can read.
Authored by handDomain expert
Finalise the solution
The reference outputs are settled to the value, so a verifier can be written against a number rather than against an impression.
Authored by handAI assistant
Generate the verifiers
An assistant drafts one check per thing the deliverable has to get right, against the finished solution.
Authored by handDomain expert
Improve the verifiers
The expert rewrites them into the rubrics that ship: each names the values it expects and the reason the check exists.
Quality gateQC
Packet and verifier QC
A quality check over the packet, the solution and the verifier set, raising flags rather than deciding the task.
Authored by handDomain expert
Resolve the flags
Every flag goes back to the creator, and the task does not move until each is answered.
Peer review · two gates
Quality gatePeer reviewer
Gate 1 · Blind review
A second domain expert, who did not write the task, attempts it blind and judges whether it is the work it claims to be.
Reject — the task leaves the pipeline and is not used.
Quality gatePeer reviewer
Gate 2 · Verifier review
The same reviewer reads the verifier set against the solution: whether each check grades the thing it says it grades.
Updates required — the verifiers go back to the creator to revise, and return to this gate rather than skipping it.
Quality gatePeer reviewer
Approved
A task ships only when both gates accept. One acceptance is not a pass.
Evaluation
Automated
Packet freeze
Each task is checksummed before any model sees it, so a packet that changes becomes a different task rather than the same one re-run.
Automated
Agent runs
Every model runs the same containerised task under the same declared limits — a four-hour agent budget, a fixed build and verifier timeout, the same memory and storage ceiling.
Automated
Grading
One judge model grades every verifier on every trial at a fixed temperature and reasoning effort, against the rubric rather than against its own opinion.
Quality gate
Failure labelling
Every verifier failure is reviewed and assigned to exactly one mode at the earliest point in the pipeline that broke, including the modes that fault the grader.
§2QC gates
A task advances only when the criterion is met. A failure sends it back — nothing ships on reviewer intuition alone.
| Stage | Gate | Who runs it | Pass criterion | On failure |
|---|---|---|---|---|
| Gate 1 · Blind review | A second expert attempts the task blind | Peer reviewer | A domain expert who did not write the task works it through and judges it to be the job it claims to be. | Reject. The task leaves the pipeline — it is not revised and re-submitted at this gate. |
| Gate 2 · Verifier review | Each check grades what it says it grades | Peer reviewer | The reviewer reads every verifier against the reference solution and accepts the set. A task ships only when both gates accept. | Updates required. The verifiers go back to the creator and return to this gate rather than skipping it. |
| Packet freeze | Task digest recorded | Producer | Every task ships with a SHA-256 over its packet, published in the drop beside the task it belongs to. | A re-cut packet is a new digest and therefore a new task; results are never carried across one. |
| Reference outputs | Answers held out of the workspace | Producer | Three reference deliverables per task, shipped for grading and absent from everything the agent can read. | A reference file reachable from the workspace invalidates the task, not the trial — the task is re-cut and re-frozen. |
| Agent runs | Every trial terminates inside its limits | Harness | The trial ends and its outputs are collected, whether or not it produced the deliverables. | A trial that could not finish is recorded as errored rather than as a zero, so a harness fault is never read as a model failure. |
| Failure labelling | One mode per verifier failure | Labelling pass | Every verifier failure carries exactly one mode, assigned at the earliest stage that broke, with the method published beside the labels. | A failure whose cause cannot be established from the trajectory and the artifact is inspected directly rather than left to the nearest cluster. |
§3QC results
Outcomes of the process above, as of 2026-08-24.
69 / 69
tasks shipped with a checksummed packet and held-out answers
Tasks1631 / 1631
trials that ran to completion — none errored before grading
Leaderboard6
models run end to end under identical limits
Leaderboard4
independent attempts per model per task
This page has two sources and they are not equally checkable. The Evaluation chapter of §1, and §2's freeze, held-out-answer, run and labelling gates, are what the delivered drop evidences — the artifacts that arrived, their checksums, the limits every trial declared, the judge configuration recorded against every verifier, and the labelling pass whose method the drop publishes in full. Each of those is inspectable in the task files. The authoring and peer-review chapters of §1, and §2's two review gates, are EKWbench's published account of how it builds a task; this delivery neither confirms nor contradicts them, since it carries no reviewer, no review date and no authoring record. Read that half as a described process rather than an audited one, and the attestation line below as the measure of who currently stands behind either half. Two further sections are written and deliberately not published: the reviewer and expert roster, and the expert path through one of these tasks. Neither can be checked against this delivery — no human attempted these tasks, so there is no human baseline and no timing behind the path, and the roster is part delivery record and part description — and an unverifiable claim on the page about whether this bench can be trusted is worth less than its absence. They are one line away in the config for whoever can stand behind them.