EKWbench
Switch benchEKWbench
Request dataset samplesby

Global Rankings

Every model attempts every task 10 times independently, under its own published harness, and every deliverable is graded verifier by verifier against a written rubric. The headline number is the share of each task's verifier weight a trial earned, averaged over its trials — not a pass rate: a trial that misses a core verifier still scores the weight it did earn. The all-core and verifier figures under each bar are the two pass rates, published beside the reward rather than folded into it. Ranked across 6 model/harness combinations.

Last updated 2026-08-21

Model Scorecard · open a row for its model cardWeighted Verifier Scoreall-coreAvg CostOut TokensTool Calls
  1. 1Claude Opus 5 · claude-codemax74.4%266 trials20.7%$12.28151k78

    Model card · evaluated Aug 2026

    claude-code 2.1.237 · reasoning: max · 266 trials

    • Reward

      74.4%±2.7 pts

    • All-core pass

      20.7%

    • Verifier pass

      76.4%

    • $ / task

      $12.28

    • Tokens

      13.8M

    Failure Modes

    1. Reasoning and Computation51%
    2. Presentation and Communication23%
    3. Deliverable Completion and Structure22%
    4. Input and Instruction Comprehension4%

    Every verifier failure was reviewed and filed under one failure mode, grouped into the stages above. The shares split this model's own 1182 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake. Counted over the 3 of 69 tasks whose trials carry labels.

    Verifier pass rate by taxonomy subdomain

    1. Government93.1%
    2. Health care and social assistance86.1%
    3. Real estate and rental and leasing83.9%
    4. Professional, scientific, and technical services82%
    5. Information78.4%
    6. Finance and insurance73%
    7. Manufacturing72.1%
    8. Retail trade71.9%

    The share of each subdomain’s verifiers this model’s trials cleared, its tasks’ trials added together.

    Reward is gated on 79.3% of trials by a failed core verifier.

  2. 2GPT-5.6 Sol · codexmax72.5%276 trials19.6%$4.7562k47

    Model card · evaluated Aug 2026

    codex 0.148.0 · reasoning: max · 276 trials

    • Reward

      72.5%±2.7 pts

    • All-core pass

      19.6%

    • Verifier pass

      74.8%

    • $ / task

      $4.75

    • Tokens

      4.6M

    Failure Modes

    1. Reasoning and Computation50%
    2. Presentation and Communication24%
    3. Deliverable Completion and Structure22%
    4. Input and Instruction Comprehension3%

    Every verifier failure was reviewed and filed under one failure mode, grouped into the stages above. The shares split this model's own 1275 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake. Counted over the 3 of 69 tasks whose trials carry labels.

    Verifier pass rate by taxonomy subdomain

    1. Professional, scientific, and technical services87.3%
    2. Health care and social assistance79.8%
    3. Real estate and rental and leasing78.6%
    4. Information78.3%
    5. Government77.8%
    6. Retail trade72.9%
    7. Finance and insurance71.6%
    8. Manufacturing71.1%

    The share of each subdomain’s verifiers this model’s trials cleared, its tasks’ trials added together.

    Reward is gated on 80.4% of trials by a failed core verifier.

  3. 3GPT-5.6 Terra · codexmax68.3%276 trials8.3%$1.8763k44

    Model card · evaluated Aug 2026

    codex 0.148.0 · reasoning: max · 276 trials

    • Reward

      68.3%±2.7 pts

    • All-core pass

      8.3%

    • Verifier pass

      70.6%

    • $ / task

      $1.87

    • Tokens

      4.4M

    Failure Modes

    1. Reasoning and Computation46%
    2. Presentation and Communication28%
    3. Deliverable Completion and Structure22%
    4. Input and Instruction Comprehension3%

    Every verifier failure was reviewed and filed under one failure mode, grouped into the stages above. The shares split this model's own 1495 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake. Counted over the 3 of 69 tasks whose trials carry labels.

    Verifier pass rate by taxonomy subdomain

    1. Professional, scientific, and technical services86.1%
    2. Health care and social assistance76.8%
    3. Real estate and rental and leasing76.6%
    4. Information71.4%
    5. Government70.8%
    6. Manufacturing69.6%
    7. Finance and insurance66.6%
    8. Retail trade66.5%

    The share of each subdomain’s verifiers this model’s trials cleared, its tasks’ trials added together.

    Reward is gated on 91.7% of trials by a failed core verifier.

  4. 4Claude Sonnet 5 · claude-codemax66.9%262 trials8.8%$12.49189k126

    Model card · evaluated Aug 2026

    claude-code 2.1.237 · reasoning: max · 262 trials

    • Reward

      66.9%±3 pts

    • All-core pass

      8.8%

    • Verifier pass

      68.9%

    • $ / task

      $12.49

    • Tokens

      28.6M

    Failure Modes

    1. Reasoning and Computation45%
    2. Deliverable Completion and Structure26%
    3. Presentation and Communication26%
    4. Input and Instruction Comprehension4%

    Every verifier failure was reviewed and filed under one failure mode, grouped into the stages above. The shares split this model's own 1515 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake. Counted over the 3 of 69 tasks whose trials carry labels.

    Verifier pass rate by taxonomy subdomain

    1. Professional, scientific, and technical services84.3%
    2. Real estate and rental and leasing83.9%
    3. Health care and social assistance79.2%
    4. Government76.4%
    5. Information68.1%
    6. Retail trade64.6%
    7. Finance and insurance63.2%
    8. Manufacturing63.1%

    The share of each subdomain’s verifiers this model’s trials cleared, its tasks’ trials added together.

    Reward is gated on 91.2% of trials by a failed core verifier.

  5. 5Gemini 3.6 Flash · gemini-clihigh53.7%275 trials4.4%$1.31115k54

    Model card · evaluated Aug 2026

    gemini-cli 0.56.0 · reasoning: high · 275 trials

    • Reward

      53.7%±3.1 pts

    • All-core pass

      4.4%

    • Verifier pass

      54.6%

    • $ / task

      $1.31

    • Tokens

      5.6M

    Failure Modes

    1. Reasoning and Computation37%
    2. Deliverable Completion and Structure32%
    3. Presentation and Communication26%
    4. Input and Instruction Comprehension5%

    Every verifier failure was reviewed and filed under one failure mode, grouped into the stages above. The shares split this model's own 2353 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake. Counted over the 3 of 69 tasks whose trials carry labels.

    Verifier pass rate by taxonomy subdomain

    1. Professional, scientific, and technical services81.9%
    2. Real estate and rental and leasing62.4%
    3. Health care and social assistance60.7%
    4. Government59.7%
    5. Retail trade52.9%
    6. Information52.7%
    7. Finance and insurance49.5%
    8. Manufacturing47.9%

    The share of each subdomain’s verifiers this model’s trials cleared, its tasks’ trials added together.

    Reward is gated on 95.6% of trials by a failed core verifier.

  6. 6Gemini 3.1 Pro · gemini-clihigh37.4%276 trials0.7%$0.5317k22

    Model card · evaluated Aug 2026

    gemini-cli 0.56.0 · reasoning: high · 276 trials

    • Reward

      37.4%±2.8 pts

    • All-core pass

      0.7%

    • Verifier pass

      39%

    • $ / task

      $0.53

    • Tokens

      689k

    Failure Modes

    1. Deliverable Completion and Structure39%
    2. Reasoning and Computation30%
    3. Presentation and Communication28%
    4. Input and Instruction Comprehension3%

    Every verifier failure was reviewed and filed under one failure mode, grouped into the stages above. The shares split this model's own 3192 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake. Counted over the 3 of 69 tasks whose trials carry labels.

    Verifier pass rate by taxonomy subdomain

    1. Professional, scientific, and technical services57.9%
    2. Real estate and rental and leasing50.1%
    3. Information43%
    4. Health care and social assistance42.4%
    5. Government41.7%
    6. Manufacturing36.6%
    7. Finance and insurance35.9%
    8. Retail trade31.3%

    The share of each subdomain’s verifiers this model’s trials cleared, its tasks’ trials added together.

    Reward is gated on 99.3% of trials by a failed core verifier.

Benchmark Scope

What this sample covers, and what it does not. Each task is filed under the occupations it draws on, so a task spanning two shows up under both — the bars count filings, not tasks. Each domain is headed by its total with its sub-domains under it. This is a sample rather than a full sweep of the taxonomy, and the Taxonomy tab lists every domain including the ones carrying no task yet.

Task Domain & Sub-domain Coverage

How the task set distributes across industry domains, each domain headed by its total with the specific processes under it

Finance and insurance
23 of 69 tasks
Financial & investment analysis
8 of 23 tasks
Insurance Underwriting
2 of 23 tasks
Investment banking & capital markets
2 of 23 tasks
Commercial Lending & Credit
1 of 23 tasks
Corporate Finance & Treasury
1 of 23 tasks
Credit - IG
1 of 23 tasks
Credit - IG Bonds
1 of 23 tasks
Credit - Junk Bonds
1 of 23 tasks
Equity Capital Markets
1 of 23 tasks
Equity research valuation (DCF)
1 of 23 tasks
Hedge Funds
1 of 23 tasks
Insurance
1 of 23 tasks
Investment Banking - M&A
1 of 23 tasks
Reinsurance
1 of 23 tasks
Retail trade
13 of 69 tasks
Allocation Planning / Size-Pack Allocation
1 of 13 tasks
Buying / open-to-buy management
1 of 13 tasks
Category Management
1 of 13 tasks
Category Management / Assortment Planning
1 of 13 tasks
Consumables/CPG
1 of 13 tasks
Grocery / Food Retail
1 of 13 tasks
Grocery Category Management
1 of 13 tasks
Loss Prevention and Asset Protection
1 of 13 tasks
Prepared Foods Assortment Planning
1 of 13 tasks
Retail category management
1 of 13 tasks
Retail Merchandising / Category Management
1 of 13 tasks
Retail Merchandising Finance / Trade Promotion Funding
1 of 13 tasks
Supply Chain and Inventory
1 of 13 tasks
Manufacturing
9 of 69 tasks
Production & operations management
2 of 9 tasks
Apparel Manufacturing / Production Planning
1 of 9 tasks
Apparel Manufacturing / Production Recovery
1 of 9 tasks
Cost Engineering / Product Costing & Yield
1 of 9 tasks
Management Accounting / Manufacturing Cost Control
1 of 9 tasks
Plant Controllership / Cost & Variance Analysis
1 of 9 tasks
Production Planning / Capacity Allocation
1 of 9 tasks
Supply chain & procurement
1 of 9 tasks
Health care and social assistance
8 of 69 tasks
Nursing and Advanced Practice
2 of 8 tasks
Behavioral and Mental Health
1 of 8 tasks
Clinical Documentation / Care Pathway Analysis
1 of 8 tasks
Critical Care Medicine
1 of 8 tasks
Health Administration and Management
1 of 8 tasks
Interventional Pulmonology cross-disciplinary with Cardiology
1 of 8 tasks
Outcome Reporting
1 of 8 tasks
Real estate and rental and leasing
6 of 69 tasks
CRE Debt & Loan Asset Management
1 of 6 tasks
CRE Underwriting
1 of 6 tasks
Equipment Leasing
1 of 6 tasks
Property & Lease
1 of 6 tasks
Real estate investment
1 of 6 tasks
Special Situations / Distressed Real Estate Credit
1 of 6 tasks
Information
5 of 69 tasks
Producers & Directors
5 of 5 tasks
Professional, scientific, and technical services
4 of 69 tasks
Advisory / Tax
1 of 4 tasks
Assurance - Publc Accounting
1 of 4 tasks
Consulting
1 of 4 tasks
Transaction Services / Financial Due Diligence
1 of 4 tasks
Government
1 of 69 tasks
Public administration and operations
1 of 1 task

Aggregate Metrics

Reward

Mean of the reward emitted by the verifier across all trials and tasks

  1. 74.4%

    Claude Opus 5max

  2. 72.5%

    GPT-5.6 Solmax

  3. 68.3%

    GPT-5.6 Terramax

  4. 66.9%

    Claude Sonnet 5max

  5. 53.7%

    Gemini 3.6 Flashhigh

  6. 37.4%

    Gemini 3.1 Prohigh

All-Core Pass Rate

Percentage of trials where every core verifier passed, averaged across tasks

  1. 20.7%

    Claude Opus 5max

  2. 19.6%

    GPT-5.6 Solmax

  3. 8.8%

    Claude Sonnet 5max

  4. 8.3%

    GPT-5.6 Terramax

  5. 4.4%

    Gemini 3.6 Flashhigh

  6. 0.7%

    Gemini 3.1 Prohigh

Verifier Pass %

Mean passing fraction across all verifiers (core + secondary), across trials and tasks

  1. 76.4%

    Claude Opus 5max

  2. 74.8%

    GPT-5.6 Solmax

  3. 70.6%

    GPT-5.6 Terramax

  4. 68.9%

    Claude Sonnet 5max

  5. 54.6%

    Gemini 3.6 Flashhigh

  6. 39%

    Gemini 3.1 Prohigh

Core Pass %

Mean pass rate across core verifiers, averaged across trials and tasks

  1. 76.8%

    Claude Opus 5max

  2. 74.4%

    GPT-5.6 Solmax

  3. 69.9%

    GPT-5.6 Terramax

  4. 68.6%

    Claude Sonnet 5max

  5. 56.5%

    Gemini 3.6 Flashhigh

  6. 37.8%

    Gemini 3.1 Prohigh

Secondary Pass %

Mean pass rate across secondary verifiers, averaged across trials and tasks

  1. 76.2%

    GPT-5.6 Solmax

  2. 75.7%

    Claude Opus 5max

  3. 72.7%

    GPT-5.6 Terramax

  4. 70.3%

    Claude Sonnet 5max

  5. 50.0%

    Gemini 3.6 Flashhigh

  6. 41.5%

    Gemini 3.1 Prohigh

Domain Difficulty

The board's order is not the order in every domain. Read a column and you find models that trade places — one leads the field in Manufacturing and sits mid-table in Health care, where the model it beat is first by a distance. A row that is uniformly dark is a domain nobody has cracked; on this five-task sample a domain can be a single task deep, so read the Benchmark Profile above before treating a row as a verdict.

Mean Reward — Taxonomy × Model

Mean reward per top-level taxonomy domain — a domain is its tasks’ trials re-added, weighted as the evaluation ran them. Colour bands the value; every cell also prints it.

Claude Opus 5max
GPT-5.6 Solmax
GPT-5.6 Terramax
Claude Sonnet 5max
Gemini 3.6 Flashhigh
Gemini 3.1 Prohigh
Finance and insurance
72.9%
70.7%
65.5%
61.9%
49.2%
34.5%
Retail trade
70.8%
72.7%
66%
64%
53.3%
31%
Manufacturing
73.2%
71.6%
69.2%
63%
48.5%
36.1%
Health care and social assistance
73%
68.1%
65.7%
67.7%
53.2%
35.1%
Information
76.8%
75%
68.3%
67.4%
52.1%
40.4%
Real estate and rental and leasing
83.8%
75.7%
74.8%
83.6%
62%
49.3%
Government
95.3%
80.3%
76.5%
76.5%
62.7%
41.8%
Professional, scientific, and technical services
78%
83.3%
82.5%
82.1%
81.2%
59.3%
Domains at or above the mean
8/8
8/8
8/8
8/8
1/8
0/8

0%95.3%

Economics

Avg Cost

Total API cost, averaged across trials and all tasks (label shows min–max trial)

  1. $12.49$1.14–$36.98

    Claude Sonnet 5max

  2. $12.28$2.25–$33.92

    Claude Opus 5max

  3. $4.75$0.94–$11.18

    GPT-5.6 Solmax

  4. $1.87$0.40–$4.29

    GPT-5.6 Terramax

  5. $1.31$0.01–$6.51

    Gemini 3.6 Flashhigh

  6. $0.53$0.02–$1.73

    Gemini 3.1 Prohigh

Avg Time

Wall-clock duration, averaged across trials and all tasks

  1. 41.1 min

    Claude Sonnet 5max

  2. 33.7 min

    Claude Opus 5max

  3. 14.5 min

    GPT-5.6 Solmax

  4. 11.7 min

    Gemini 3.6 Flashhigh

  5. 11.3 min

    GPT-5.6 Terramax

  6. 3.2 min

    Gemini 3.1 Prohigh

Avg Trajectory Steps

Agent-loop steps — every event, not only tool calls — averaged across trials and all tasks, counted from the full trajectories before truncation

  1. 191

    Claude Sonnet 5max

  2. 119

    Claude Opus 5max

  3. 97

    Gemini 3.6 Flashhigh

  4. 51

    GPT-5.6 Solmax

  5. 42

    GPT-5.6 Terramax

  6. 27

    Gemini 3.1 Prohigh

Avg Tool Calls

Tool invocations, averaged across trials and all tasks

  1. 126

    Claude Sonnet 5max

  2. 78

    Claude Opus 5max

  3. 54

    Gemini 3.6 Flashhigh

  4. 47

    GPT-5.6 Solmax

  5. 44

    GPT-5.6 Terramax

  6. 22

    Gemini 3.1 Prohigh

Avg Output Tokens

Output tokens generated, averaged across trials and tasks

  1. 189k

    Claude Sonnet 5max

  2. 151k

    Claude Opus 5max

  3. 115k

    Gemini 3.6 Flashhigh

  4. 63k

    GPT-5.6 Terramax

  5. 62k

    GPT-5.6 Solmax

  6. 17k

    Gemini 3.1 Prohigh

Avg Input Tokens

Input tokens consumed, averaged across trials and tasks — cumulative across turns, not peak context

  1. 28.4M

    Claude Sonnet 5max

  2. 13.6M

    Claude Opus 5max

  3. 5.5M

    Gemini 3.6 Flashhigh

  4. 4.5M

    GPT-5.6 Solmax

  5. 4.3M

    GPT-5.6 Terramax

  6. 672k

    Gemini 3.1 Prohigh

Frontier

Spend and score are barely related. Across this board the cost of a trial varies by more than an order of magnitude and the input it reads by more again, while the reward those trials earn spans little more than a factor of two. The cheapest model on the page is the best value per dollar and the worst on the board; the two dearest are separated from the leader by less than the gap between their bills. Both charts plot the same six models — up is better, left is cheaper, and the interesting models are the ones nearest the top left.

Reward vs Cost

Soft-gated reward against API cost across trials — top-left is more reward for less money

  • Claude Opus 5
  • GPT-5.6 Sol
  • GPT-5.6 Terra
  • Claude Sonnet 5
  • Gemini 3.6 Flash
  • Gemini 3.1 Pro

$0.53Cost$12.49

Reward vs Input Tokens

Soft-gated reward against input tokens consumed, summed across the trial rather than peak context

  • Claude Opus 5
  • GPT-5.6 Sol
  • GPT-5.6 Terra
  • Claude Sonnet 5
  • Gemini 3.6 Flash
  • Gemini 3.1 Pro

672kInput tokens28.4M