Every model attempts every task 10 times independently under identical conditions, and every trial is graded check by check — exact values against expert-computed golds, database state for write-actions, tool-trace checks for process. The board ranks on full task passes: a trial counts only when it cleared every check on the task, and a model's figure is the share of its trials that did. Nothing here is partial credit — a trial that is right about most of a fee-revenue reconstruction and wrong about the total counts exactly as a trial that got none of it. Ranked across4 models.
Last updated 2026-06-15
Total Tasks
64
each attempted 10× by 4 models
Best Pass Rate
44.4%
GPT-5.5
Model Scorecard · open a row for its model cardall-core pass rateTool Calls
1GPT-5.544.4%640 trials63
Model card · evaluated Jun 2026
640 trials
All-core pass
44.4%
Failure Modes
Answer97%Answer346 of 356 verifier failures
Execution3%Execution10 of 356 verifier failures
Every verifier failure was reviewed and filed under one failure mode, grouped into the stages above. The shares split this model's own 356 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake.
Pass rate by taxonomy subdomain
Payments & Transfers56%Payments & Transfers56% of 100 trials passed
Deposits & Funding55%Deposits & Funding55% of 20 trials passed
Risk & Fraud50.8%Risk & Fraud50.8% of 120 trials passed
Subscriptions & Billing47.5%Subscriptions & Billing47.5% of 40 trials passed
Compliance & Regulatory44.3%Compliance & Regulatory44.3% of 70 trials passed
Cards & Interchange43.3%Cards & Interchange43.3% of 120 trials passed
Revenue & Fees33.3%Revenue & Fees33.3% of 60 trials passed
Reconciliation & Settlement23.3%Reconciliation & Settlement23.3% of 60 trials passed
The share of each subdomain’s verifiers this model’s trials cleared, its tasks’ trials added together.
2Claude Opus 4.835.9%640 trials31
Model card · evaluated Jun 2026
640 trials
All-core pass
35.9%
Failure Modes
Answer100%Answer410 of 410 verifier failures
Execution0%Execution0 of 410 verifier failures
Every verifier failure was reviewed and filed under one failure mode, grouped into the stages above. The shares split this model's own 410 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake.
Pass rate by taxonomy subdomain
Cards & Interchange52.5%Cards & Interchange52.5% of 120 trials passed
Payments & Transfers43%Payments & Transfers43% of 100 trials passed
Risk & Fraud35.8%Risk & Fraud35.8% of 120 trials passed
Reconciliation & Settlement30%Reconciliation & Settlement30% of 60 trials passed
Subscriptions & Billing27.5%Subscriptions & Billing27.5% of 40 trials passed
Compliance & Regulatory25.7%Compliance & Regulatory25.7% of 70 trials passed
Revenue & Fees25%Revenue & Fees25% of 60 trials passed
Deposits & Funding10%Deposits & Funding10% of 20 trials passed
The share of each subdomain’s verifiers this model’s trials cleared, its tasks’ trials added together.
3Gemini 3.1 Pro18.3%640 trials46
Model card · evaluated Jun 2026
640 trials
All-core pass
18.3%
Failure Modes
Answer96%Answer503 of 523 verifier failures
Execution4%Execution20 of 523 verifier failures
Every verifier failure was reviewed and filed under one failure mode, grouped into the stages above. The shares split this model's own 523 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake.
Pass rate by taxonomy subdomain
Subscriptions & Billing42.5%Subscriptions & Billing42.5% of 40 trials passed
Compliance & Regulatory24.3%Compliance & Regulatory24.3% of 70 trials passed
Cards & Interchange24.2%Cards & Interchange24.2% of 120 trials passed
Payments & Transfers22%Payments & Transfers22% of 100 trials passed
Revenue & Fees18.3%Revenue & Fees18.3% of 60 trials passed
Risk & Fraud14.2%Risk & Fraud14.2% of 120 trials passed
Reconciliation & Settlement5%Reconciliation & Settlement5% of 60 trials passed
Deposits & Funding0%Deposits & Funding0% of 20 trials passed
The share of each subdomain’s verifiers this model’s trials cleared, its tasks’ trials added together.
4Gemini 3.5 Flash7.7%640 trials54
Model card · evaluated Jun 2026
640 trials
All-core pass
7.7%
Failure Modes
Answer91%Answer540 of 591 verifier failures
Execution9%Execution51 of 591 verifier failures
Every verifier failure was reviewed and filed under one failure mode, grouped into the stages above. The shares split this model's own 591 verifier failures, worst first — grader error is excluded, so a model is never charged with its grader's mistake.
Pass rate by taxonomy subdomain
Subscriptions & Billing25%Subscriptions & Billing25% of 40 trials passed
Risk & Fraud22.5%Risk & Fraud22.5% of 120 trials passed
Revenue & Fees5%Revenue & Fees5% of 60 trials passed
Reconciliation & Settlement3.3%Reconciliation & Settlement3.3% of 60 trials passed
Compliance & Regulatory2.9%Compliance & Regulatory2.9% of 70 trials passed
Cards & Interchange2.5%Cards & Interchange2.5% of 120 trials passed
Payments & Transfers2%Payments & Transfers2% of 100 trials passed
Deposits & Funding0%Deposits & Funding0% of 20 trials passed
The share of each subdomain’s verifiers this model’s trials cleared, its tasks’ trials added together.
Benchmark Scope
What this set covers. Each task is filed under exactly one of the bench's nine operational categories, so the bars count tasks. Payments and risk dominate the corpus; the Taxonomy tab lists every category with its tasks.
Task Domain & Sub-domain Coverage
How the task set distributes across industry domains, each domain headed by its total with the specific processes under it
Cards & Interchange
12 of 64 tasks
Risk & Fraud
12 of 64 tasks
Payments & Transfers
10 of 64 tasks
Compliance & Regulatory
7 of 64 tasks
Reconciliation & Settlement
6 of 64 tasks
Revenue & Fees
6 of 64 tasks
Customer & Account Analytics
5 of 64 tasks
Subscriptions & Billing
4 of 64 tasks
Deposits & Funding
2 of 64 tasks
Aggregate Metrics
All-Core Pass Rate
Percentage of trials that cleared every check on the task, averaged across tasks
100%0%
44.4%
GPT-5.5
35.9%
Claude Opus 4.8
18.3%
Gemini 3.1 Pro
7.7%
Gemini 3.5 Flash
Category Difficulty
The board's order is not the order in every category. Read a column and you find models trading places between reconciliation work and cross-system coordination; a row that is uniformly dark is a category nobody has cracked. Each cell is the share of a model's trials on that category's tasks that cleared every check — the same reading as the board above, one category at a time.
All-Core Pass Rate — Taxonomy × Model
Share of trials that cleared every check on the task, per top-level taxonomy domain — a domain is its tasks' trials re-added, weighted as the evaluation ran them. Colour bands the value; every cell also prints it.
GPT-5.5
Claude Opus 4.8
Gemini 3.1 Pro
Gemini 3.5 Flash
Reconciliation & Settlement
23.3%
30%
5%
3.3%
Deposits & Funding
55%
10%
0%
0%
Customer & Account Analytics
40%
34%
2%
0%
Revenue & Fees
33.3%
25%
18.3%
5%
Compliance & Regulatory
44.3%
25.7%
24.3%
2.9%
Cards & Interchange
43.3%
52.5%
24.2%
2.5%
Payments & Transfers
56%
43%
22%
2%
Risk & Fraud
50.8%
35.8%
14.2%
22.5%
Subscriptions & Billing
47.5%
27.5%
42.5%
25%
Domains at or above the mean
9/9
7/9
2/9
0/9
0%56%
Economics
What the trials spent, in the units this drop actually measures: tool calls against the connected systems, and the length of the published agent traces. Cost, token and time charts light up the day the producer ships the trial-level metrics CSV they hold upstream.
Avg Tool Calls
Tool invocations, averaged across trials and all tasks
630
63
GPT-5.5
54
Gemini 3.5 Flash
46
Gemini 3.1 Pro
31
Claude Opus 4.8
Avg Trajectory Steps
Agent-loop steps — every event, not only tool calls — averaged across trials and all tasks, counted from the full trajectories before truncation