Soft-Gated Reward
- 496verifiers
- 64scored tasks
What counts as a pass
A trial passes a task only when every check on it passes. That is the rate the leaderboard ranks on, and it is the only rate this bench publishes. The harness also emits a reward for each trial — the share of the task's checks it cleared, so a trial that reaches the right number by the wrong route still earns something — but partial credit is not a pass, and nothing on the board is ranked by it.
Score anatomy · same task, two trials
Both trials below are Gemini 3.5 Flash on Goal Automation Throughput Analysis. The gap between them is variance in one model on one task, not a difference between models.
trial_8all checks passed
- 19 / 19checks
- All-core pass rate sparse
- 100%every check passed
- Reward weighted
- 100%
trial_9missed a check
- 1 / 19checks
- All-core pass rate sparse
- 0%18 checks failed → 0
- Reward weighted
- 5.3%
Both trials below are Claude Opus 4.8 on Audit manual-review check deposit release latency & stranded funds. The gap between them is variance in one model on one task, not a difference between models.
trial_1missed a check
- 12 / 13checks
- All-core pass rate sparse
- 0%1 check failed → 0
- Reward weighted
- 92.3%
trial_10missed a check
- 3 / 13checks
- All-core pass rate sparse
- 0%9 checks failed → 0
- Reward weighted
- 23.1%
Both trials below are Gemini 3.1 Pro on Duplicate card auth holds never released. The gap between them is variance in one model on one task, not a difference between models.
trial_2all checks passed
- 14 / 14checks
- All-core pass rate sparse
- 100%every check passed
- Reward weighted
- 100%
trial_7missed a check
- 1 / 14checks
- All-core pass rate sparse
- 0%7 checks failed → 0
- Reward weighted
- 7.1%
Both trials below are Gemini 3.5 Flash on Goal Automation Throughput Analysis. The gap between them is variance in one model on one task, not a difference between models.
trial_8all checks passed
- 19 / 19checks
- All-core pass rate sparse
- 100%every check passed
- Reward weighted
- 100%
trial_9missed a check
- 1 / 19checks
- All-core pass rate sparse
- 0%18 checks failed → 0
- Reward weighted
- 5.3%
Both trials below are GPT-5.5 on Additional-owner card activation funnel. The gap between them is variance in one model on one task, not a difference between models.
trial_3all checks passed
- 13 / 13checks
- All-core pass rate sparse
- 100%every check passed
- Reward weighted
- 100%
trial_2missed a check
- 1 / 13checks
- All-core pass rate sparse
- 0%8 checks failed → 0
- Reward weighted
- 7.7%