Task pass rate by model
Full leaderboardThe share of each model's trials in which every core verifier passed.
Measured on 69 tasks · 1,631 trials · 6 models
EKWbench evaluates agents on multi-hour operational knowledge work — the sourcing reviews, cost roll-ups, assortment resets and referral triage that keep an enterprise running between its systems of record. Every task hands over the working files the job actually runs on — bills of material, price masters, certification snapshots, signed handoff notes, policy excerpts — and asks for the two artifacts the role really produces: an auditable workbook and a decision memo, at exact paths, inside a four-hour budget. Nothing is graded on style; every verifier is named and graded against an expert value. The full corpus carries 191 expert-built tasks across nine enterprise domains, 103 of them finance and accounting and 187 built around Excel workbooks. The board below measures 69 of them, and published results cover that measured subset rather than the full corpus.
EKWbench evaluates agents on multi-hour operational knowledge work — the sourcing reviews, cost roll-ups, assortment resets and referral triage that keep an enterprise running between its systems of record. Every task hands over the working files the job actually runs on — bills of material, price masters, certification snapshots, signed handoff notes, policy excerpts — and asks for the two artifacts the role really produces: an auditable workbook and a decision memo, at exact paths, inside a four-hour budget. Nothing is graded on style; every verifier is named and graded against an expert value. The full corpus carries 191 expert-built tasks across nine enterprise domains, 103 of them finance and accounting and 187 built around Excel workbooks. The board below measures 69 of them, and published results cover that measured subset rather than the full corpus.
EKWbench evaluates agents on multi-hour operational knowledge work — the sourcing reviews, cost roll-ups, assortment resets and referral triage that keep an enterprise running between its systems of record. Every task hands over the working files the job actually runs on — bills of material, price masters, certification snapshots, signed handoff notes, policy excerpts — and asks for the two artifacts the role really produces: an auditable workbook and a decision memo, at exact paths, inside a four-hour budget. Nothing is graded on style; every verifier is named and graded against an expert value. The full corpus carries 191 expert-built tasks across nine enterprise domains, 103 of them finance and accounting and 187 built around Excel workbooks. The board below measures 69 of them, and published results cover that measured subset rather than the full corpus.
EKWbench evaluates agents on multi-hour operational knowledge work — the sourcing reviews, cost roll-ups, assortment resets and referral triage that keep an enterprise running between its systems of record. Every task hands over the working files the job actually runs on — bills of material, price masters, certification snapshots, signed handoff notes, policy excerpts — and asks for the two artifacts the role really produces: an auditable workbook and a decision memo, at exact paths, inside a four-hour budget. Nothing is graded on style; every verifier is named and graded against an expert value. The full corpus carries 191 expert-built tasks across nine enterprise domains, 103 of them finance and accounting and 187 built around Excel workbooks. The board below measures 69 of them, and published results cover that measured subset rather than the full corpus.
EKWbench evaluates agents on multi-hour operational knowledge work — the sourcing reviews, cost roll-ups, assortment resets and referral triage that keep an enterprise running between its systems of record. Every task hands over the working files the job actually runs on — bills of material, price masters, certification snapshots, signed handoff notes, policy excerpts — and asks for the two artifacts the role really produces: an auditable workbook and a decision memo, at exact paths, inside a four-hour budget. Nothing is graded on style; every verifier is named and graded against an expert value. The full corpus carries 191 expert-built tasks across nine enterprise domains, 103 of them finance and accounting and 187 built around Excel workbooks. The board below measures 69 of them, and published results cover that measured subset rather than the full corpus.
EKWbench evaluates agents on multi-hour operational knowledge work — the sourcing reviews, cost roll-ups, assortment resets and referral triage that keep an enterprise running between its systems of record. Every task hands over the working files the job actually runs on — bills of material, price masters, certification snapshots, signed handoff notes, policy excerpts — and asks for the two artifacts the role really produces: an auditable workbook and a decision memo, at exact paths, inside a four-hour budget. Nothing is graded on style; every verifier is named and graded against an expert value. The full corpus carries 191 expert-built tasks across nine enterprise domains, 103 of them finance and accounting and 187 built around Excel workbooks. The board below measures 69 of them, and published results cover that measured subset rather than the full corpus.
EKWbench evaluates agents on multi-hour operational knowledge work — the sourcing reviews, cost roll-ups, assortment resets and referral triage that keep an enterprise running between its systems of record. Every task hands over the working files the job actually runs on — bills of material, price masters, certification snapshots, signed handoff notes, policy excerpts — and asks for the two artifacts the role really produces: an auditable workbook and a decision memo, at exact paths, inside a four-hour budget. Nothing is graded on style; every verifier is named and graded against an expert value. The full corpus carries 191 expert-built tasks across nine enterprise domains, 103 of them finance and accounting and 187 built around Excel workbooks. The board below measures 69 of them, and published results cover that measured subset rather than the full corpus.
The share of each model's trials in which every core verifier passed.
Measured on 69 tasks · 1,631 trials · 6 models
Each task represents a specific occupation from the domain-and-subdomain reference and uses the original files associated with that job. The chart groups tasks by domain. Domains that do not yet have any tasks are left out.
Each cell shows a model's performance across all trials for tasks in that domain. The rate is calculated from those individual trials, rather than by averaging the pass rate of each task.
Each task represents a specific occupation from the domain-and-subdomain reference and uses the original files associated with that job. The chart groups tasks by domain, excluding domains that do not yet have any tasks.
Claude Opus 5 is highlighted below so you can compare its results with the other models on the same tasks. Each cell shows performance across all trials for tasks in that domain. The rate is calculated from those individual trials, rather than by averaging each task's pass rate.
Each task represents a specific occupation from the domain-and-subdomain reference and uses the original files associated with that job. The chart groups tasks by domain, excluding domains that do not yet have any tasks.
GPT-5.6 Sol is highlighted below so you can compare its results with the other models on the same tasks. Each cell shows performance across all trials for tasks in that domain. The rate is calculated from those individual trials, rather than by averaging each task's pass rate.
Each task represents a specific occupation from the domain-and-subdomain reference and uses the original files associated with that job. The chart groups tasks by domain, excluding domains that do not yet have any tasks.
Claude Sonnet 5 is highlighted below so you can compare its results with the other models on the same tasks. Each cell shows performance across all trials for tasks in that domain. The rate is calculated from those individual trials, rather than by averaging each task's pass rate.
Each task represents a specific occupation from the domain-and-subdomain reference and uses the original files associated with that job. The chart groups tasks by domain, excluding domains that do not yet have any tasks.
GPT-5.6 Terra is highlighted below so you can compare its results with the other models on the same tasks. Each cell shows performance across all trials for tasks in that domain. The rate is calculated from those individual trials, rather than by averaging each task's pass rate.
Each task represents a specific occupation from the domain-and-subdomain reference and uses the original files associated with that job. The chart groups tasks by domain, excluding domains that do not yet have any tasks.
Gemini 3.6 Flash is highlighted below so you can compare its results with the other models on the same tasks. Each cell shows performance across all trials for tasks in that domain. The rate is calculated from those individual trials, rather than by averaging each task's pass rate.
Each task represents a specific occupation from the domain-and-subdomain reference and uses the original files associated with that job. The chart groups tasks by domain, excluding domains that do not yet have any tasks.
Gemini 3.1 Pro is highlighted below so you can compare its results with the other models on the same tasks. Each cell shows performance across all trials for tasks in that domain. The rate is calculated from those individual trials, rather than by averaging each task's pass rate.
Covers 8 domains · 57 subdomains
Measured on 69 tasks · 1,631 trials · 6 models
The board ranks models by reward rather than full task passes. Each task has multiple checks, with more important checks carrying more weight. Models receive partial credit for every check they pass, even when they don’t pass the whole task. This means a model can rank highly while fully passing only a small number of tasks.
Claude Opus 5leads the board
Claude Opus 5leads the board
GPT-5.6 Solon the board
GPT-5.6 Terraon the board
Claude Sonnet 5on the board
Gemini 3.6 Flashon the board
Gemini 3.1 Proon the board
Measured on 69 tasks · 1,631 trials · 6 models
How often a trial clears every core verifier — the strictest reading amongst all. A trial counts here only where every core verifier passed.
How often a Claude Opus 5 trial clears every core verifier — the strictest reading amongst all. A trial counts here only where every core verifier passed.
How often a GPT-5.6 Sol trial clears every core verifier — the strictest reading amongst all. A trial counts here only where every core verifier passed.
How often a Claude Sonnet 5 trial clears every core verifier — the strictest reading amongst all. A trial counts here only where every core verifier passed.
How often a GPT-5.6 Terra trial clears every core verifier — the strictest reading amongst all. A trial counts here only where every core verifier passed.
How often a Gemini 3.6 Flash trial clears every core verifier — the strictest reading amongst all. A trial counts here only where every core verifier passed.
How often a Gemini 3.1 Pro trial clears every core verifier — the strictest reading amongst all. A trial counts here only where every core verifier passed.
Claude Opus 5
20.7%
best model's pass rate
Across 69 tasks attempted 4× each, Claude Opus 5 cleared every core verifier on 20.7% of trials. Individual verifiers mostly pass; it is holding all of them at once — through a four-hour job, across a workbook and the memo that has to justify it — that these tasks make hard.
Claude Opus 5
20.7%
pass rate
Across 69 tasks attempted 4× each, Claude Opus 5 cleared every core verifier on 20.7% of trials. Individual verifiers mostly pass; it is holding all of them at once — through a four-hour job, across a workbook and the memo that has to justify it — that these tasks make hard.
GPT-5.6 Sol
19.6%
pass rate
Across 69 tasks attempted 4× each, GPT-5.6 Sol cleared every core verifier on 19.6% of trials. Individual verifiers mostly pass; it is holding all of them at once — through a four-hour job, across a workbook and the memo that has to justify it — that these tasks make hard.
Claude Sonnet 5
8.8%
pass rate
Across 69 tasks attempted 4× each, Claude Sonnet 5 cleared every core verifier on 8.8% of trials. Individual verifiers mostly pass; it is holding all of them at once — through a four-hour job, across a workbook and the memo that has to justify it — that these tasks make hard.
GPT-5.6 Terra
8.3%
pass rate
Across 69 tasks attempted 4× each, GPT-5.6 Terra cleared every core verifier on 8.3% of trials. Individual verifiers mostly pass; it is holding all of them at once — through a four-hour job, across a workbook and the memo that has to justify it — that these tasks make hard.
Gemini 3.6 Flash
4.4%
pass rate
Across 69 tasks attempted 4× each, Gemini 3.6 Flash cleared every core verifier on 4.4% of trials. Individual verifiers mostly pass; it is holding all of them at once — through a four-hour job, across a workbook and the memo that has to justify it — that these tasks make hard.
Gemini 3.1 Pro
0.7%
pass rate
Across 69 tasks attempted 4× each, Gemini 3.1 Pro cleared every core verifier on 0.7% of trials — not one attempt, on any task. This is a different failure from the rest of the board, where a trial is usually lost to a single missed check; the modes below say which one it is.
Every failed verifier in this drop was reviewed and filed under exactly one failure mode, at the earliest point in the pipeline that broke. What follows is the reading those labels support.
Every failed verifier Claude Opus 5 produced in this drop was reviewed and filed under exactly one failure mode, at the earliest point in the pipeline that broke. What follows is where it breaks, read against the rest of the field.
Every failed verifier GPT-5.6 Sol produced in this drop was reviewed and filed under exactly one failure mode, at the earliest point in the pipeline that broke. What follows is where it breaks, read against the rest of the field.
Every failed verifier Claude Sonnet 5 produced in this drop was reviewed and filed under exactly one failure mode, at the earliest point in the pipeline that broke. What follows is where it breaks, read against the rest of the field.
Every failed verifier GPT-5.6 Terra produced in this drop was reviewed and filed under exactly one failure mode, at the earliest point in the pipeline that broke. What follows is where it breaks, read against the rest of the field.
Every failed verifier Gemini 3.6 Flash produced in this drop was reviewed and filed under exactly one failure mode, at the earliest point in the pipeline that broke. What follows is where it breaks, read against the rest of the field.
Every failed verifier Gemini 3.1 Pro produced in this drop was reviewed and filed under exactly one failure mode, at the earliest point in the pipeline that broke. What follows is where it breaks, read against the rest of the field.
Claude Opus 5 leads on individual verifiers with 76% of them passed, and clears every core verifier on 21% of its trials. The board's headline figure and its pass rate are answering two different questions, and the second is the one a buyer is asking.
The inputs are available, but extraction, rule application, selection, calculation, or reconciliation produces the wrong substantive result. It accounts for 4,429 of the drop's 11,391 verifier failures — more than any other stage, and every model on the board is in that number.
It accounts for 2,398 of the drop's 11,391 verifier failures, and shows up on 3 of the 3 tasks this drop publishes — so it is a property of the work rather than of one task. From the runs: A reservation table contained client-slot pairs but omitted start time and reserved-minutes fields required to verify the mappings.
GPT-5.6 Sol and Gemini 3.6 Flash make almost the same number of tool calls — 47 and 54, at $4.75 and $1.31 — and score 72.5% against 53.7%, 18.8 pts apart. What separates them is what they do with what they read, not how much of it they touch.
1,182 verifier failures out of the 11,012 charged to any model, over 266 trials — 11% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Tagged at Presentation and Communication in 2% of its 266 trials, against 1% for the median of the other 5 models — the widest gap it opens anywhere in the taxonomy. From the trials: A reconciliation memo added diagnostic monitoring and new case-management instructions beyond the requested routing report.
Tagged at Deliverable Completion and Structure in 0% of its 266 trials, against 3% for the median of the other 5 models. The same measurement as above, read the other way — a failure reading that only ever accuses is not a reading.
1,275 verifier failures out of the 11,012 charged to any model, over 276 trials — 12% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Tagged at Presentation and Communication in 1% of its 276 trials, against 1% for the median of the other 5 models — the widest gap it opens anywhere in the taxonomy. From the trials: A reconciliation memo added diagnostic monitoring and new case-management instructions beyond the requested routing report.
Tagged at Deliverable Completion and Structure in 1% of its 276 trials, against 3% for the median of the other 5 models. The same measurement as above, read the other way — a failure reading that only ever accuses is not a reading.
1,495 verifier failures out of the 11,012 charged to any model, over 276 trials — 14% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Tagged at Presentation and Communication in 1% of its 276 trials, against 0% for the median of the other 5 models — the widest gap it opens anywhere in the taxonomy. From the trials: A memo described four clusters as over capacity but omitted each proposal-specific used-versus-capacity comparison.
Tagged at Deliverable Completion and Structure in 0% of its 276 trials, against 3% for the median of the other 5 models. The same measurement as above, read the other way — a failure reading that only ever accuses is not a reading.
1,515 verifier failures out of the 11,012 charged to any model, over 262 trials — 14% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Tagged at Deliverable Completion and Structure in 3% of its 262 trials, against 1% for the median of the other 5 models — the widest gap it opens anywhere in the taxonomy. From the trials: A reservation table contained client-slot pairs but omitted start time and reserved-minutes fields required to verify the mappings.
Tagged at Presentation and Communication in 1% of its 262 trials, against 1% for the median of the other 5 models. The same measurement as above, read the other way — a failure reading that only ever accuses is not a reading.
2,353 verifier failures out of the 11,012 charged to any model, over 275 trials — 21% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Tagged at Reasoning and Computation in 4% of its 275 trials, against 1% for the median of the other 5 models — the widest gap it opens anywhere in the taxonomy. From the trials: China-melt inventory was allocated to a requirement whose governing origin rule excluded it.
Tagged at Presentation and Communication in 0% of its 275 trials, against 0% for the median of the other 5 models. The same measurement as above, read the other way — a failure reading that only ever accuses is not a reading.
3,192 verifier failures out of the 11,012 charged to any model, over 276 trials — 29% of them. Verifier failures labelled grader error are left out of that denominator, here as everywhere else on the site. What follows is what it does differently from the rest of the board, not its biggest numbers, which mostly repeat the field's.
Tagged at Deliverable Completion and Structure in 4% of its 276 trials, against 1% for the median of the other 5 models — the widest gap it opens anywhere in the taxonomy. From the trials: A reservation table contained client-slot pairs but omitted start time and reserved-minutes fields required to verify the mappings.
Tagged at Deliverable Completion and Structure in 0% of its 276 trials, against 1% for the median of the other 5 models. The same measurement as above, read the other way — a failure reading that only ever accuses is not a reading.
Each card is one task the whole field attempted, headed by the id the packet carries and opened by the task's own brief. Under it is every model, ranked by the share of that task's verifiers its ten trials cleared — the number that separates this field, since almost no trial clears all of them. Open a card for the full brief, the packet the agent had to read, and how it was graded.
The three tasks Claude Opus 5 does worst on, worst first, out of the 69 the whole field attempted. Each card ranks every model by the share of that task's verifiers its 4 trials cleared, so the anchored number is read against the others rather than on its own. Open a card for the full brief, the packet the agent had to read, and how it was graded.
The three tasks GPT-5.6 Sol does worst on, worst first, out of the 69 the whole field attempted. Each card ranks every model by the share of that task's verifiers its 4 trials cleared, so the anchored number is read against the others rather than on its own. Open a card for the full brief, the packet the agent had to read, and how it was graded.
The three tasks Claude Sonnet 5 does worst on, worst first, out of the 69 the whole field attempted. Each card ranks every model by the share of that task's verifiers its 4 trials cleared, so the anchored number is read against the others rather than on its own. Open a card for the full brief, the packet the agent had to read, and how it was graded.
The three tasks GPT-5.6 Terra does worst on, worst first, out of the 69 the whole field attempted. Each card ranks every model by the share of that task's verifiers its 4 trials cleared, so the anchored number is read against the others rather than on its own. Open a card for the full brief, the packet the agent had to read, and how it was graded.
The three tasks Gemini 3.6 Flash does worst on, worst first, out of the 69 the whole field attempted. Each card ranks every model by the share of that task's verifiers its 4 trials cleared, so the anchored number is read against the others rather than on its own. Open a card for the full brief, the packet the agent had to read, and how it was graded.
The three tasks Gemini 3.1 Pro does worst on, worst first, out of the 69 the whole field attempted. Each card ranks every model by the share of that task's verifiers its 4 trials cleared, so the anchored number is read against the others rather than on its own. Open a card for the full brief, the packet the agent had to read, and how it was graded.
Retail trade
Assortment Reset Decision Analysis
You are a category planning analyst at Blue Mesa Market, an 84-store convenience/grocery retailer. The prepared-foods merchant has asked you to review a July reset proposal for the refrigerated grab-and-go case before it is released to Space Planning and…
Manufacturing
Material sourcing / material allocation / sourcing risk review
You are supporting a material sourcing review for an aerospace components manufacturer that builds certified door and hinge mechanism assemblies. Upcoming production lots require aluminum and steel raw materials, including plate, bar, extrusion, and…
Health care and social assistance
Behavioral Health Access Triage and Referral Reconciliation
You are helping the urgent-access lead at Meadowbrook Behavioral Health Access Services finish the Friday cutover queue before the early-afternoon scheduling huddle. The EHR migration left a mixed behavioral-health referral batch in the access desk…
Retail trade
Assortment Reset Decision Analysis
You are a category planning analyst at Blue Mesa Market, an 84-store convenience/grocery retailer. The prepared-foods merchant has asked you to review a July reset proposal for the refrigerated grab-and-go case before it is released to Space Planning and…
Manufacturing
Material sourcing / material allocation / sourcing risk review
You are supporting a material sourcing review for an aerospace components manufacturer that builds certified door and hinge mechanism assemblies. Upcoming production lots require aluminum and steel raw materials, including plate, bar, extrusion, and…
Health care and social assistance
Behavioral Health Access Triage and Referral Reconciliation
You are helping the urgent-access lead at Meadowbrook Behavioral Health Access Services finish the Friday cutover queue before the early-afternoon scheduling huddle. The EHR migration left a mixed behavioral-health referral batch in the access desk…
Retail trade
Assortment Reset Decision Analysis
You are a category planning analyst at Blue Mesa Market, an 84-store convenience/grocery retailer. The prepared-foods merchant has asked you to review a July reset proposal for the refrigerated grab-and-go case before it is released to Space Planning and…
Manufacturing
Material sourcing / material allocation / sourcing risk review
You are supporting a material sourcing review for an aerospace components manufacturer that builds certified door and hinge mechanism assemblies. Upcoming production lots require aluminum and steel raw materials, including plate, bar, extrusion, and…
Health care and social assistance
Behavioral Health Access Triage and Referral Reconciliation
You are helping the urgent-access lead at Meadowbrook Behavioral Health Access Services finish the Friday cutover queue before the early-afternoon scheduling huddle. The EHR migration left a mixed behavioral-health referral batch in the access desk…
Retail trade
Assortment Reset Decision Analysis
You are a category planning analyst at Blue Mesa Market, an 84-store convenience/grocery retailer. The prepared-foods merchant has asked you to review a July reset proposal for the refrigerated grab-and-go case before it is released to Space Planning and…
Manufacturing
Material sourcing / material allocation / sourcing risk review
You are supporting a material sourcing review for an aerospace components manufacturer that builds certified door and hinge mechanism assemblies. Upcoming production lots require aluminum and steel raw materials, including plate, bar, extrusion, and…
Health care and social assistance
Behavioral Health Access Triage and Referral Reconciliation
You are helping the urgent-access lead at Meadowbrook Behavioral Health Access Services finish the Friday cutover queue before the early-afternoon scheduling huddle. The EHR migration left a mixed behavioral-health referral batch in the access desk…
Retail trade
Assortment Reset Decision Analysis
You are a category planning analyst at Blue Mesa Market, an 84-store convenience/grocery retailer. The prepared-foods merchant has asked you to review a July reset proposal for the refrigerated grab-and-go case before it is released to Space Planning and…
Manufacturing
Material sourcing / material allocation / sourcing risk review
You are supporting a material sourcing review for an aerospace components manufacturer that builds certified door and hinge mechanism assemblies. Upcoming production lots require aluminum and steel raw materials, including plate, bar, extrusion, and…
Health care and social assistance
Behavioral Health Access Triage and Referral Reconciliation
You are helping the urgent-access lead at Meadowbrook Behavioral Health Access Services finish the Friday cutover queue before the early-afternoon scheduling huddle. The EHR migration left a mixed behavioral-health referral batch in the access desk…
Manufacturing
Material sourcing / material allocation / sourcing risk review
You are supporting a material sourcing review for an aerospace components manufacturer that builds certified door and hinge mechanism assemblies. Upcoming production lots require aluminum and steel raw materials, including plate, bar, extrusion, and…
Retail trade
Assortment Reset Decision Analysis
You are a category planning analyst at Blue Mesa Market, an 84-store convenience/grocery retailer. The prepared-foods merchant has asked you to review a July reset proposal for the refrigerated grab-and-go case before it is released to Space Planning and…
Health care and social assistance
Behavioral Health Access Triage and Referral Reconciliation
You are helping the urgent-access lead at Meadowbrook Behavioral Health Access Services finish the Friday cutover queue before the early-afternoon scheduling huddle. The EHR migration left a mixed behavioral-health referral batch in the access desk…
Retail trade
Assortment Reset Decision Analysis
You are a category planning analyst at Blue Mesa Market, an 84-store convenience/grocery retailer. The prepared-foods merchant has asked you to review a July reset proposal for the refrigerated grab-and-go case before it is released to Space Planning and…
Manufacturing
Material sourcing / material allocation / sourcing risk review
You are supporting a material sourcing review for an aerospace components manufacturer that builds certified door and hinge mechanism assemblies. Upcoming production lots require aluminum and steel raw materials, including plate, bar, extrusion, and…
Health care and social assistance
Behavioral Health Access Triage and Referral Reconciliation
You are helping the urgent-access lead at Meadowbrook Behavioral Health Access Services finish the Friday cutover queue before the early-afternoon scheduling huddle. The EHR migration left a mixed behavioral-health referral batch in the access desk…