Terminal-Bench 3.0
Switch benchTerminal-Bench 3.0
Request dataset samplesby

Global Rankings

Every config ran the whole suite, eight attempts per task. The board is the producer's own fold: their raw trial output was not part of this delivery, so no figure here is recomputed from a finer grain, and the page says so. Model and scaffold are confounded — three scaffolds appear across the configs, so every difference between two rows is a joint model-or-scaffold effect. 8 model/harness combinations.

Last updated 2026-08-13

Model Scorecard · open a row for its model cardReward rateAvg CostOut Tokens
  1. 1GPT-5.6 Sol · codexmax56.6%800 trials$0.9918k

    Model card · evaluated Aug 2026

    reasoning: max · 800 trials

    • Reward

      56.6%

    • $ / task

      $0.99

    • Tokens

      577k

    Failure Modes

    1. Misread the problem39%
    2. Under-modeled a stated requirement30%
    3. Missed the numeric bar23%
    4. Broke under re-entry or concurrency5%
    5. Resolved a spec conflict unilaterally2%
    6. Failed at delivery, not reasoning0%

    Every verifier failure was reviewed and filed under one failure mode, grouped into the stages above. The shares split this model's own 338 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake.

  2. 2Fable-5 · claude-codehigh47.3%800 trials$1.5919k

    Model card · evaluated Aug 2026

    reasoning: high · 800 trials

    • Reward

      47.3%

    • $ / task

      $1.59

    • Tokens

      455k

    Failure Modes

    1. Misread the problem35%
    2. Under-modeled a stated requirement31%
    3. Missed the numeric bar21%
    4. Broke under re-entry or concurrency11%
    5. Resolved a spec conflict unilaterally2%
    6. Failed at delivery, not reasoning0%

    Every verifier failure was reviewed and filed under one failure mode, grouped into the stages above. The shares split this model's own 406 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake.

  3. 3Grok-4.5 · terminus-2high44%800 trials$0.4824k

    Model card · evaluated Aug 2026

    reasoning: high · 800 trials

    • Reward

      44%

    • $ / task

      $0.48

    • Tokens

      555k

    Failure Modes

    1. Misread the problem43%
    2. Under-modeled a stated requirement28%
    3. Missed the numeric bar18%
    4. Broke under re-entry or concurrency9%
    5. Failed at delivery, not reasoning2%
    6. Resolved a spec conflict unilaterally1%

    Every verifier failure was reviewed and filed under one failure mode, grouped into the stages above. The shares split this model's own 435 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake.

  4. 4GLM-5.2 · terminus-2high41.5%800 trials$0.2146k

    Model card · evaluated Aug 2026

    reasoning: high · 800 trials

    • Reward

      41.5%

    • $ / task

      $0.21

    • Tokens

      251k

    Failure Modes

    1. Misread the problem44%
    2. Under-modeled a stated requirement25%
    3. Missed the numeric bar23%
    4. Broke under re-entry or concurrency7%
    5. Resolved a spec conflict unilaterally1%
    6. Failed at delivery, not reasoning0%

    Every verifier failure was reviewed and filed under one failure mode, grouped into the stages above. The shares split this model's own 445 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake.

  5. 5KIMI K3 · terminus-2thinking39.8%800 trials$0.3515k

    Model card · evaluated Aug 2026

    reasoning: thinking:enabled · 800 trials

    • Reward

      39.8%

    • $ / task

      $0.35

    • Tokens

      126k

    Failure Modes

    1. Misread the problem41%
    2. Under-modeled a stated requirement30%
    3. Missed the numeric bar20%
    4. Broke under re-entry or concurrency9%
    5. Resolved a spec conflict unilaterally0%
    6. Failed at delivery, not reasoning0%

    Every verifier failure was reviewed and filed under one failure mode, grouped into the stages above. The shares split this model's own 462 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake.

  6. 6Opus-4.8 · terminus-2xhigh36.6%800 trials$5.5452k

    Model card · evaluated Aug 2026

    reasoning: xhigh · 800 trials

    • Reward

      36.6%

    • $ / task

      $5.54

    • Tokens

      870k

    Failure Modes

    1. Misread the problem38%
    2. Under-modeled a stated requirement28%
    3. Missed the numeric bar24%
    4. Broke under re-entry or concurrency9%
    5. Resolved a spec conflict unilaterally1%
    6. Failed at delivery, not reasoning0%

    Every verifier failure was reviewed and filed under one failure mode, grouped into the stages above. The shares split this model's own 473 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake.

  7. 7KIMI K2.7 · terminus-2max27.6%800 trials$0.2328k

    Model card · evaluated Aug 2026

    reasoning: max · 800 trials

    • Reward

      27.6%

    • $ / task

      $0.23

    • Tokens

      575k

    Failure Modes

    1. Misread the problem39%
    2. Under-modeled a stated requirement26%
    3. Missed the numeric bar22%
    4. Broke under re-entry or concurrency9%
    5. Failed at delivery, not reasoning3%
    6. Resolved a spec conflict unilaterally0%

    Every verifier failure was reviewed and filed under one failure mode, grouped into the stages above. The shares split this model's own 528 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake.

  8. 8Inkling · terminus-2xhigh21.9%800 trials$0.3030k

    Model card · evaluated Aug 2026

    reasoning: xhigh · 800 trials

    • Reward

      21.9%

    • $ / task

      $0.30

    • Tokens

      514k

    Failure Modes

    1. Misread the problem40%
    2. Under-modeled a stated requirement27%
    3. Missed the numeric bar21%
    4. Broke under re-entry or concurrency7%
    5. Failed at delivery, not reasoning5%
    6. Resolved a spec conflict unilaterally0%

    Every verifier failure was reviewed and filed under one failure mode, grouped into the stages above. The shares split this model's own 602 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake.

Benchmark Scope

What the suite covers, before any config is ranked on it.

Task Domain & Sub-domain Coverage

How the task set distributes across industry domains, each domain headed by its total with the specific processes under it

Business & Domain Workflows
15 of 100 tasks
Distributed Systems & Databases
15 of 100 tasks
Developer Toolchain & Build
12 of 100 tasks
Media & Graphics
12 of 100 tasks
Scientific Computing & Simulation
12 of 100 tasks
ML & Forecasting
11 of 100 tasks
Data Engineering & Pipelines
10 of 100 tasks
Systems & SRE
7 of 100 tasks
Security & Forensics
6 of 100 tasks

Aggregate Metrics

One bar per config, over the whole suite. Model and scaffold are confounded — read a gap as a joint effect, not as a model result.

Reward

Mean of the reward emitted by the verifier across all trials and tasks

  1. 56.6%

    GPT-5.6 Solmax

  2. 47.3%

    Fable-5high

  3. 44%

    Grok-4.5high

  4. 41.5%

    GLM-5.2high

  5. 39.8%

    KIMI K3thinking

  6. 36.6%

    Opus-4.8xhigh

  7. 27.6%

    KIMI K2.7max

  8. 21.9%

    Inklingxhigh

Domain Difficulty

The same reward, cut by domain. A domain is a fixed set of tasks, so read this as a map of where difficulty sits rather than as evidence that a domain causes it.

Mean Reward — Taxonomy × Model

Mean reward per top-level taxonomy domain — a domain is its tasks’ trials re-added, weighted as the evaluation ran them. Colour bands the value; every cell also prints it.

GPT-5.6 Solmax
Fable-5high
Grok-4.5high
GLM-5.2high
KIMI K3thinking:enabled
Opus-4.8xhigh
KIMI K2.7max
Inklingxhigh
Systems & SRE
35.7%
32.1%
16.1%
32.1%
25%
27.3%
14.3%
17.9%
Developer Toolchain & Build
38.5%
38.5%
34.4%
21.9%
17.7%
28.1%
20.8%
11.5%
Data Engineering & Pipelines
61.3%
35%
41.3%
46.3%
23.8%
17.5%
30%
13.8%
Distributed Systems & Databases
60.8%
32.5%
33.3%
51.7%
36.4%
36.7%
22.5%
27.5%
Security & Forensics
62.5%
56.3%
41.7%
22.9%
60.4%
41.7%
10.4%
20.8%
Scientific Computing & Simulation
58.3%
46.9%
45.8%
36.5%
41.7%
40.6%
33.3%
27.1%
ML & Forecasting
61.4%
69.3%
60.7%
40.2%
47.7%
40.9%
27.3%
23.9%
Business & Domain Workflows
62.5%
55%
44.2%
45.8%
59.2%
36.2%
39.2%
32.5%
Media & Graphics
61.5%
59.4%
70.8%
61.5%
47.5%
57%
38.5%
13.5%
Domains at or above the mean
9/9
8/9
6/9
4/9
4/9
4/9
0/9
0/9

0%70.8%

Economics

What a config costs to run the suite once. A handful of runaway trials carry most of the bill, so the means here are dragged by them and the median in the board table is the steadier figure.

Avg Cost

Total API cost, averaged across trials and all tasks (label shows min–max trial)

  1. $5.54

    Opus-4.8xhigh

  2. $1.59

    Fable-5high

  3. $0.99

    GPT-5.6 Solmax

  4. $0.48

    Grok-4.5high

  5. $0.35

    KIMI K3thinking

  6. $0.30

    Inklingxhigh

  7. $0.23

    KIMI K2.7max

  8. $0.21

    GLM-5.2high

Avg Time

Wall-clock duration, averaged across trials and all tasks

  1. 32.5 min

    Inklingxhigh

  2. 21.1 min

    KIMI K2.7max

  3. 16.9 min

    GLM-5.2high

  4. 14.2 min

    Opus-4.8xhigh

  5. 9.5 min

    KIMI K3thinking

  6. 8.4 min

    Grok-4.5high

  7. 5.5 min

    GPT-5.6 Solmax

  8. 4.9 min

    Fable-5high

Avg Trajectory Steps

Agent-loop steps — every event, not only tool calls — averaged across trials and all tasks, counted from the full trajectories before truncation

  1. 30

    Inklingxhigh

  2. 20

    GPT-5.6 Solmax

  3. 19

    Opus-4.8xhigh

  4. 19

    KIMI K2.7max

  5. 16

    Grok-4.5high

  6. 12

    GLM-5.2high

  7. 11

    Fable-5high

  8. 9

    KIMI K3thinking

Avg Output Tokens

Output tokens generated, averaged across trials and tasks

  1. 52k

    Opus-4.8xhigh

  2. 46k

    GLM-5.2high

  3. 30k

    Inklingxhigh

  4. 28k

    KIMI K2.7max

  5. 24k

    Grok-4.5high

  6. 19k

    Fable-5high

  7. 18k

    GPT-5.6 Solmax

  8. 15k

    KIMI K3thinking

Avg Input Tokens

Input tokens consumed, averaged across trials and tasks — cumulative across turns, not peak context

  1. 818k

    Opus-4.8xhigh

  2. 560k

    GPT-5.6 Solmax

  3. 547k

    KIMI K2.7max

  4. 532k

    Grok-4.5high

  5. 484k

    Inklingxhigh

  6. 436k

    Fable-5high

  7. 205k

    GLM-5.2high

  8. 112k

    KIMI K3thinking

Frontier

Reward against what it took to get there. Up and to the left is better.

Reward vs Cost

Soft-gated reward against API cost across trials — top-left is more reward for less money

  • GPT-5.6 Sol
  • Fable-5
  • Grok-4.5
  • GLM-5.2
  • KIMI K3
  • Opus-4.8
  • KIMI K2.7
  • Inkling

$0.21Cost$5.54

Reward vs Input Tokens

Soft-gated reward against input tokens consumed, summed across the trial rather than peak context

  • GPT-5.6 Sol
  • Fable-5
  • Grok-4.5
  • GLM-5.2
  • KIMI K3
  • Opus-4.8
  • KIMI K2.7
  • Inkling

112kInput tokens818k