Task pass rate by model
Full leaderboardThe share of each model's trials that passed, graded by execution verdict.
Measured on 100 tasks · 6,400 trials · 8 models
Hard for the right reasons.
Terminal-Bench 3.0 pairs the economic realism of expert work with the rigor of terminal-native evaluation, spanning well beyond typical software engineering — distributed systems, GPU performance, security forensics, geospatial and clinical-data workflows. Every task stalls a competent expert not because it is long, but because it demands a non-obvious leap that cannot be retrieved. Each is adversarially reviewed and graded only by execution, so a passing score is capability rather than pattern-matching.
Hard for the right reasons.
Terminal-Bench 3.0 pairs the economic realism of expert work with the rigor of terminal-native evaluation, spanning well beyond typical software engineering — distributed systems, GPU performance, security forensics, geospatial and clinical-data workflows. Every task stalls a competent expert not because it is long, but because it demands a non-obvious leap that cannot be retrieved. Each is adversarially reviewed and graded only by execution, so a passing score is capability rather than pattern-matching.
Hard for the right reasons.
Terminal-Bench 3.0 pairs the economic realism of expert work with the rigor of terminal-native evaluation, spanning well beyond typical software engineering — distributed systems, GPU performance, security forensics, geospatial and clinical-data workflows. Every task stalls a competent expert not because it is long, but because it demands a non-obvious leap that cannot be retrieved. Each is adversarially reviewed and graded only by execution, so a passing score is capability rather than pattern-matching.
Hard for the right reasons.
Terminal-Bench 3.0 pairs the economic realism of expert work with the rigor of terminal-native evaluation, spanning well beyond typical software engineering — distributed systems, GPU performance, security forensics, geospatial and clinical-data workflows. Every task stalls a competent expert not because it is long, but because it demands a non-obvious leap that cannot be retrieved. Each is adversarially reviewed and graded only by execution, so a passing score is capability rather than pattern-matching.
Hard for the right reasons.
Terminal-Bench 3.0 pairs the economic realism of expert work with the rigor of terminal-native evaluation, spanning well beyond typical software engineering — distributed systems, GPU performance, security forensics, geospatial and clinical-data workflows. Every task stalls a competent expert not because it is long, but because it demands a non-obvious leap that cannot be retrieved. Each is adversarially reviewed and graded only by execution, so a passing score is capability rather than pattern-matching.
Hard for the right reasons.
Terminal-Bench 3.0 pairs the economic realism of expert work with the rigor of terminal-native evaluation, spanning well beyond typical software engineering — distributed systems, GPU performance, security forensics, geospatial and clinical-data workflows. Every task stalls a competent expert not because it is long, but because it demands a non-obvious leap that cannot be retrieved. Each is adversarially reviewed and graded only by execution, so a passing score is capability rather than pattern-matching.
Hard for the right reasons.
Terminal-Bench 3.0 pairs the economic realism of expert work with the rigor of terminal-native evaluation, spanning well beyond typical software engineering — distributed systems, GPU performance, security forensics, geospatial and clinical-data workflows. Every task stalls a competent expert not because it is long, but because it demands a non-obvious leap that cannot be retrieved. Each is adversarially reviewed and graded only by execution, so a passing score is capability rather than pattern-matching.
Hard for the right reasons.
Terminal-Bench 3.0 pairs the economic realism of expert work with the rigor of terminal-native evaluation, spanning well beyond typical software engineering — distributed systems, GPU performance, security forensics, geospatial and clinical-data workflows. Every task stalls a competent expert not because it is long, but because it demands a non-obvious leap that cannot be retrieved. Each is adversarially reviewed and graded only by execution, so a passing score is capability rather than pattern-matching.
Hard for the right reasons.
Terminal-Bench 3.0 pairs the economic realism of expert work with the rigor of terminal-native evaluation, spanning well beyond typical software engineering — distributed systems, GPU performance, security forensics, geospatial and clinical-data workflows. Every task stalls a competent expert not because it is long, but because it demands a non-obvious leap that cannot be retrieved. Each is adversarially reviewed and graded only by execution, so a passing score is capability rather than pattern-matching.
- 100
- Tasks
- 8
- Models
- 8
- Trials per task
The Verdict
Within the competitive field the configs reach almost the same set of tasks. What they do not share is the ability to hit the same task twice.
The Verdict
Within the competitive field the configs reach almost the same set of tasks. What they do not share is the ability to hit the same task twice.
The Verdict
Within the competitive field the configs reach almost the same set of tasks. What they do not share is the ability to hit the same task twice.
The Verdict
Within the competitive field the configs reach almost the same set of tasks. What they do not share is the ability to hit the same task twice.
The Verdict
Within the competitive field the configs reach almost the same set of tasks. What they do not share is the ability to hit the same task twice.
The Verdict
Within the competitive field the configs reach almost the same set of tasks. What they do not share is the ability to hit the same task twice.
The Verdict
Within the competitive field the configs reach almost the same set of tasks. What they do not share is the ability to hit the same task twice.
The Verdict
Within the competitive field the configs reach almost the same set of tasks. What they do not share is the ability to hit the same task twice.
The Verdict
Within the competitive field the configs reach almost the same set of tasks. What they do not share is the ability to hit the same task twice.
GPT-5.6 Sol
56.6%
best model's pass rate
GPT-5.6 Sol leads at 56.6%, and the spread in what a config can solve at least once is far narrower than the spread in what it solves every time.
GPT-5.6 Sol
56.6%
pass rate
GPT-5.6 Sol leads at 56.6%, and the spread in what a config can solve at least once is far narrower than the spread in what it solves every time.
Fable-5
47.3%
pass rate
Fable-5 leads at 47.3%, and the spread in what a config can solve at least once is far narrower than the spread in what it solves every time.
Grok-4.5
44%
pass rate
Grok-4.5 leads at 44%, and the spread in what a config can solve at least once is far narrower than the spread in what it solves every time.
GLM-5.2
41.5%
pass rate
GLM-5.2 leads at 41.5%, and the spread in what a config can solve at least once is far narrower than the spread in what it solves every time.
KIMI K3
39.8%
pass rate
KIMI K3 leads at 39.8%, and the spread in what a config can solve at least once is far narrower than the spread in what it solves every time.
Opus-4.8
36.6%
pass rate
Opus-4.8 leads at 36.6%, and the spread in what a config can solve at least once is far narrower than the spread in what it solves every time.
KIMI K2.7
27.6%
pass rate
KIMI K2.7 leads at 27.6%, and the spread in what a config can solve at least once is far narrower than the spread in what it solves every time.
Inkling
21.9%
pass rate
Inkling leads at 21.9%, and the spread in what a config can solve at least once is far narrower than the spread in what it solves every time.
Failue Modes
Failue Modes
Failue Modes
Failue Modes
Failue Modes
Failue Modes
Failue Modes
Failue Modes
Failue Modes
Every failing trial is labelled — nothing is sampled. The mechanism a failure is filed under says what its task traps, not what that trajectory did; the producer is explicit about that, and it is why the mechanism mix barely moves between configs.
Every failing trial is labelled — nothing is sampled. The mechanism a failure is filed under says what its task traps, not what that trajectory did; the producer is explicit about that, and it is why the mechanism mix barely moves between configs.
Every failing trial is labelled — nothing is sampled. The mechanism a failure is filed under says what its task traps, not what that trajectory did; the producer is explicit about that, and it is why the mechanism mix barely moves between configs.
Every failing trial is labelled — nothing is sampled. The mechanism a failure is filed under says what its task traps, not what that trajectory did; the producer is explicit about that, and it is why the mechanism mix barely moves between configs.
Every failing trial is labelled — nothing is sampled. The mechanism a failure is filed under says what its task traps, not what that trajectory did; the producer is explicit about that, and it is why the mechanism mix barely moves between configs.
Every failing trial is labelled — nothing is sampled. The mechanism a failure is filed under says what its task traps, not what that trajectory did; the producer is explicit about that, and it is why the mechanism mix barely moves between configs.
Every failing trial is labelled — nothing is sampled. The mechanism a failure is filed under says what its task traps, not what that trajectory did; the producer is explicit about that, and it is why the mechanism mix barely moves between configs.
Every failing trial is labelled — nothing is sampled. The mechanism a failure is filed under says what its task traps, not what that trajectory did; the producer is explicit about that, and it is why the mechanism mix barely moves between configs.
Every failing trial is labelled — nothing is sampled. The mechanism a failure is filed under says what its task traps, not what that trajectory did; the producer is explicit about that, and it is why the mechanism mix barely moves between configs.
Most failures land at Misread the problem
Built a coherent, internally consistent solution to a different reading of the spec. The work is complete and confidently delivered; the interpretation is wrong. It accounts for 1,468 of the drop's 3,833 verifier failures — more than any other stage, and every model on the board is in that number.
GPT-5.6 Sol carries the smallest share of this drop's failures
338 verifier failures out of the 3,689 charged to any model, over 800 trials — 9% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Fable-5 carries the 7th largest share of this drop's failures
406 verifier failures out of the 3,689 charged to any model, over 800 trials — 11% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Grok-4.5 carries the 6th largest share of this drop's failures
435 verifier failures out of the 3,689 charged to any model, over 800 trials — 12% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
GLM-5.2 carries the 5th largest share of this drop's failures
445 verifier failures out of the 3,689 charged to any model, over 800 trials — 12% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
KIMI K3 carries the 4th largest share of this drop's failures
462 verifier failures out of the 3,689 charged to any model, over 800 trials — 13% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Opus-4.8 carries the 3rd largest share of this drop's failures
473 verifier failures out of the 3,689 charged to any model, over 800 trials — 13% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
KIMI K2.7 carries the 2nd largest share of this drop's failures
528 verifier failures out of the 3,689 charged to any model, over 800 trials — 14% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Inkling carries the largest share of this drop's failures
602 verifier failures out of the 3,689 charged to any model, over 800 trials — 16% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
The Evidence
Browse every taskThe lowest-scoring tasks in the suite, and what each one asks for.
The Evidence
Browse every taskThe lowest-scoring tasks in the suite, and what each one asks for.
The Evidence
Browse every taskThe lowest-scoring tasks in the suite, and what each one asks for.
The Evidence
Browse every taskThe lowest-scoring tasks in the suite, and what each one asks for.
The Evidence
Browse every taskThe lowest-scoring tasks in the suite, and what each one asks for.
The Evidence
Browse every taskThe lowest-scoring tasks in the suite, and what each one asks for.
The Evidence
Browse every taskThe lowest-scoring tasks in the suite, and what each one asks for.
The Evidence
Browse every taskThe lowest-scoring tasks in the suite, and what each one asks for.
The Evidence
Browse every taskThe lowest-scoring tasks in the suite, and what each one asks for.
Media & Graphics
Edl Drop Frame
You are given a 29.97 fps drop-frame master video (`/app/master.mov`) and an edit decision list (`/app/edit.edl`). The EDL lists source in/out points using non-drop-frame timecode (counting frames as if the video were exactly 30 fps).
Business & Domain Workflows
Consolidation Upstream Nci
You are given a consolidation accounting engine in `/app/consolidation/engine.py`. The engine combines a parent and subsidiary and attempts to eliminate intercompany inventory sales and unrealized profit. The engine passes the example scenarios in the…
Security & Forensics
Firmware Patch Backdoor
Two signed RISC-V firmware images for a small device are in `/app`: - `firmware_v1.bin` — the release from day N. - `firmware_v2.bin` — the release from day N+1, signed and claiming only bugfixes. - `bootloader.elf` — the day N+1 bootloader as an ELF with…
Media & Graphics
Edl Drop Frame
You are given a 29.97 fps drop-frame master video (`/app/master.mov`) and an edit decision list (`/app/edit.edl`). The EDL lists source in/out points using non-drop-frame timecode (counting frames as if the video were exactly 30 fps).
Distributed Systems & Databases
Kafka Transactional Id
A Kafka-based payment processing pipeline, deployed as a replicated Kubernetes workload, has started producing duplicate records in the downstream accounting database after a Kubernetes migration. - Duplication rate: ~0.02% (about 200 duplicates per million…
Security & Forensics
Firmware Patch Backdoor
Two signed RISC-V firmware images for a small device are in `/app`: - `firmware_v1.bin` — the release from day N. - `firmware_v2.bin` — the release from day N+1, signed and claiming only bugfixes. - `bootloader.elf` — the day N+1 bootloader as an ELF with…
Business & Domain Workflows
Consolidation Upstream Nci
You are given a consolidation accounting engine in `/app/consolidation/engine.py`. The engine combines a parent and subsidiary and attempts to eliminate intercompany inventory sales and unrealized profit. The engine passes the example scenarios in the…
Distributed Systems & Databases
Kafka Transactional Id
A Kafka-based payment processing pipeline, deployed as a replicated Kubernetes workload, has started producing duplicate records in the downstream accounting database after a Kubernetes migration. - Duplication rate: ~0.02% (about 200 duplicates per million…
Media & Graphics
Edl Drop Frame
You are given a 29.97 fps drop-frame master video (`/app/master.mov`) and an edit decision list (`/app/edit.edl`). The EDL lists source in/out points using non-drop-frame timecode (counting frames as if the video were exactly 30 fps).
Business & Domain Workflows
Consolidation Upstream Nci
You are given a consolidation accounting engine in `/app/consolidation/engine.py`. The engine combines a parent and subsidiary and attempts to eliminate intercompany inventory sales and unrealized profit. The engine passes the example scenarios in the…
Media & Graphics
Edl Drop Frame
You are given a 29.97 fps drop-frame master video (`/app/master.mov`) and an edit decision list (`/app/edit.edl`). The EDL lists source in/out points using non-drop-frame timecode (counting frames as if the video were exactly 30 fps).
ML & Forecasting
Label Shift Monitor
The drift-monitoring pipeline in `/app` compares a reference CSV to a current production CSV and emits a retraining trigger when it believes feature distributions have changed. The on-call team has noticed that during fraud-rate surges the monitor fires…
Security & Forensics
Firmware Patch Backdoor
Two signed RISC-V firmware images for a small device are in `/app`: - `firmware_v1.bin` — the release from day N. - `firmware_v2.bin` — the release from day N+1, signed and claiming only bugfixes. - `bootloader.elf` — the day N+1 bootloader as an ELF with…
Business & Domain Workflows
Consolidation Upstream Nci
You are given a consolidation accounting engine in `/app/consolidation/engine.py`. The engine combines a parent and subsidiary and attempts to eliminate intercompany inventory sales and unrealized profit. The engine passes the example scenarios in the…
ML & Forecasting
Label Shift Monitor
The drift-monitoring pipeline in `/app` compares a reference CSV to a current production CSV and emits a retraining trigger when it believes feature distributions have changed. The on-call team has noticed that during fraud-rate surges the monitor fires…
Media & Graphics
Edl Drop Frame
You are given a 29.97 fps drop-frame master video (`/app/master.mov`) and an edit decision list (`/app/edit.edl`). The EDL lists source in/out points using non-drop-frame timecode (counting frames as if the video were exactly 30 fps).
ML & Forecasting
Label Shift Monitor
The drift-monitoring pipeline in `/app` compares a reference CSV to a current production CSV and emits a retraining trigger when it believes feature distributions have changed. The on-call team has noticed that during fraud-rate surges the monitor fires…
Distributed Systems & Databases
Scheduler Offer Reservation
## Goal The containerized two-level resource scheduler in `/app/scheduler.py` has a flawed scheduling contract. A resource offer is currently treated as a non-binding proposal: once an offer is sent, the same resources may be offered to another framework, and…
Business & Domain Workflows
Consolidation Upstream Nci
You are given a consolidation accounting engine in `/app/consolidation/engine.py`. The engine combines a parent and subsidiary and attempts to eliminate intercompany inventory sales and unrealized profit. The engine passes the example scenarios in the…
Media & Graphics
Edl Drop Frame
You are given a 29.97 fps drop-frame master video (`/app/master.mov`) and an edit decision list (`/app/edit.edl`). The EDL lists source in/out points using non-drop-frame timecode (counting frames as if the video were exactly 30 fps).
ML & Forecasting
Label Shift Monitor
The drift-monitoring pipeline in `/app` compares a reference CSV to a current production CSV and emits a retraining trigger when it believes feature distributions have changed. The on-call team has noticed that during fraud-rate surges the monitor fires…
Security & Forensics
Firmware Patch Backdoor
Two signed RISC-V firmware images for a small device are in `/app`: - `firmware_v1.bin` — the release from day N. - `firmware_v2.bin` — the release from day N+1, signed and claiming only bugfixes. - `bootloader.elf` — the day N+1 bootloader as an ELF with…
Distributed Systems & Databases
Scheduler Offer Reservation
## Goal The containerized two-level resource scheduler in `/app/scheduler.py` has a flawed scheduling contract. A resource offer is currently treated as a non-binding proposal: once an offer is sent, the same resources may be offered to another framework, and…
ML & Forecasting
Label Shift Monitor
The drift-monitoring pipeline in `/app` compares a reference CSV to a current production CSV and emits a retraining trigger when it believes feature distributions have changed. The on-call team has noticed that during fraud-rate surges the monitor fires…
Security & Forensics
Firmware Patch Backdoor
Two signed RISC-V firmware images for a small device are in `/app`: - `firmware_v1.bin` — the release from day N. - `firmware_v2.bin` — the release from day N+1, signed and claiming only bugfixes. - `bootloader.elf` — the day N+1 bootloader as an ELF with…
Media & Graphics
Edl Drop Frame
You are given a 29.97 fps drop-frame master video (`/app/master.mov`) and an edit decision list (`/app/edit.edl`). The EDL lists source in/out points using non-drop-frame timecode (counting frames as if the video were exactly 30 fps).
Distributed Systems & Databases
Scheduler Offer Reservation
## Goal The containerized two-level resource scheduler in `/app/scheduler.py` has a flawed scheduling contract. A resource offer is currently treated as a non-binding proposal: once an offer is sent, the same resources may be offered to another framework, and…
Our tasks in the published suite
Three of the tasks in the published Terminal-Bench 3.0 set are ours. Each opens on the Harbor hub — browse it, or run it against your own agent.