OdysseysMega Workflow & Quality

§1Task lifecycle

Every task travels the same pipeline, left to right. The gates are where it can be sent back, and the dashed loops show where to.

Authored by hand
Quality gate
Automated
Sent back

Authoring
Task review
Evaluation

1Authored by handWrite the request and its rubricsDomain expert
An expert writes a request a person would really make of the web — find, compare, build something usable — and breaks what a complete answer has to contain into rubric steps.
2AutomatedAutomated feasibility check
Before anyone reviews it, an automated pass checks that the task is coherent and that every rubric matches the request and can be met on the public web. It advises; it never edits the task.
3Quality gateHuman task reviewPeer reviewer
A different person reads the request, every rubric and the automated notes, edits where needed, and approves or rejects. The original and the reviewed version are both kept.
Back to step 1 · Rejected — the author sees the reason and may revise and appeal once, to a different reviewer.
4Authored by handAuthor sign-offDomain expert
The author sees the reviewed task beside their own and accepts it or amends it into the final version.
5AutomatedAgent runModel under test
The agent works the final task in a fresh Chrome session inside a virtual machine, through the mouse and keyboard, and every step is recorded with a screenshot.
6AutomatedRubric judging
A judge model reads the recorded actions and screenshots and decides, rubric by rubric, whether each was satisfied — and writes down which steps show it.
7Quality gateHuman run gradeDomain expert
The task's author grades every rubric and the run as a whole, without seeing the judge's verdict.
Back to step 3 · Edit needed or needs rerun — the task goes back to review, or waits for a new run.

Point at a step — or tap it — to read what happens there.

1Write the request and its rubricsAuthored by hand · Domain expert

An expert writes a request a person would really make of the web — find, compare, build something usable — and breaks what a complete answer has to contain into rubric steps.

2Automated feasibility checkAutomated

Before anyone reviews it, an automated pass checks that the task is coherent and that every rubric matches the request and can be met on the public web. It advises; it never edits the task.

3Human task reviewQuality gate · Peer reviewer

A different person reads the request, every rubric and the automated notes, edits where needed, and approves or rejects. The original and the reviewed version are both kept.

Rejected — the author sees the reason and may revise and appeal once, to a different reviewer. Back to step 1, “Write the request and its rubrics”.

4Author sign-offAuthored by hand · Domain expert

The author sees the reviewed task beside their own and accepts it or amends it into the final version.

5Agent runAutomated · Model under test

The agent works the final task in a fresh Chrome session inside a virtual machine, through the mouse and keyboard, and every step is recorded with a screenshot.

6Rubric judgingAutomated

A judge model reads the recorded actions and screenshots and decides, rubric by rubric, whether each was satisfied — and writes down which steps show it.

7Human run gradeQuality gate · Domain expert

The task's author grades every rubric and the run as a whole, without seeing the judge's verdict.

Edit needed or needs rerun — the task goes back to review, or waits for a new run. Back to step 3, “Human task review”.

§2QC gates

A task advances only when the criterion is met. A failure sends it back — nothing ships on reviewer intuition alone.

Stage	Gate	Who runs it	Pass criterion	On failure
Feasibility check	Every rubric reachable on the public web	Automated check	The task is coherent, and each rubric is compatible with the request and can be met from the live web.	The task is not offered for review until a complete check exists for its current text.
Task review	A second person reviews request and rubrics	Task reviewer	Approved, with the reviewer's edits kept beside the author's original.	Rejected with a reason; one appeal, heard by a different reviewer.
Rubric judging	Every rubric gets a verdict and its reasoning	Judge model	Satisfied or not, with the judge's account of which recorded steps show it.	A rubric the judge could not grade is recorded as a judge error, left out of the score, and never counted as the agent's failure.
Human run grade	The author grades the run blind to the judge	Task author	Every rubric and the run as a whole graded independently of the model's verdict.	Edit needed or needs rerun — the task returns to review or waits for a new run.

§3QC results

Outcomes of the process above, as of 2026-09-28.

100
tasks published, every rubric judged on every run
Tasks
4
models compared on the same hundred tasks
Leaderboard
399
recorded runs, each playable frame by frame
Tasks