Task pass rate by model
Full leaderboardThe share of each model's trials in which every check passed.
Measured on 64 tasks · 2,560 trials · 4 models
Enterprise Financial Analysis
CompanyBench evaluates agents on the investigative analysis work of a fintech operations team — reconciliations, conformance audits, fraud triage, fee-revenue closeouts — run against a simulated consumer-fintech neobank. Every task starts from a terse business question and no schema hints: the agent must discover the relevant warehouse tables, find the policy or definition that scopes the number in Confluence, Jira or Slack, avoid the seeded near-miss readings, and reply with exact values, or complete write-actions in the connected systems. Grading is per verifier: exact-value checks against expert-computed golds, database-state checks on what the agent actually persisted, and tool-trace checks on how it worked.
Enterprise Financial Analysis
CompanyBench evaluates agents on the investigative analysis work of a fintech operations team — reconciliations, conformance audits, fraud triage, fee-revenue closeouts — run against a simulated consumer-fintech neobank. Every task starts from a terse business question and no schema hints: the agent must discover the relevant warehouse tables, find the policy or definition that scopes the number in Confluence, Jira or Slack, avoid the seeded near-miss readings, and reply with exact values, or complete write-actions in the connected systems. Grading is per verifier: exact-value checks against expert-computed golds, database-state checks on what the agent actually persisted, and tool-trace checks on how it worked.
Enterprise Financial Analysis
CompanyBench evaluates agents on the investigative analysis work of a fintech operations team — reconciliations, conformance audits, fraud triage, fee-revenue closeouts — run against a simulated consumer-fintech neobank. Every task starts from a terse business question and no schema hints: the agent must discover the relevant warehouse tables, find the policy or definition that scopes the number in Confluence, Jira or Slack, avoid the seeded near-miss readings, and reply with exact values, or complete write-actions in the connected systems. Grading is per verifier: exact-value checks against expert-computed golds, database-state checks on what the agent actually persisted, and tool-trace checks on how it worked.
Enterprise Financial Analysis
CompanyBench evaluates agents on the investigative analysis work of a fintech operations team — reconciliations, conformance audits, fraud triage, fee-revenue closeouts — run against a simulated consumer-fintech neobank. Every task starts from a terse business question and no schema hints: the agent must discover the relevant warehouse tables, find the policy or definition that scopes the number in Confluence, Jira or Slack, avoid the seeded near-miss readings, and reply with exact values, or complete write-actions in the connected systems. Grading is per verifier: exact-value checks against expert-computed golds, database-state checks on what the agent actually persisted, and tool-trace checks on how it worked.
Enterprise Financial Analysis
CompanyBench evaluates agents on the investigative analysis work of a fintech operations team — reconciliations, conformance audits, fraud triage, fee-revenue closeouts — run against a simulated consumer-fintech neobank. Every task starts from a terse business question and no schema hints: the agent must discover the relevant warehouse tables, find the policy or definition that scopes the number in Confluence, Jira or Slack, avoid the seeded near-miss readings, and reply with exact values, or complete write-actions in the connected systems. Grading is per verifier: exact-value checks against expert-computed golds, database-state checks on what the agent actually persisted, and tool-trace checks on how it worked.
- 64
- Tasks
- 4
- Models
- 10
- Trials per task
Grounded in a simulated consumer-fintech neobankGrounded in a simulated consumer-fintech neobank, read against GPT-5.5Grounded in a simulated consumer-fintech neobank, read against Claude Opus 4.8Grounded in a simulated consumer-fintech neobank, read against Gemini 3.1 ProGrounded in a simulated consumer-fintech neobank, read against Gemini 3.5 Flash
Full taxonomyEvery task runs against the same seeded company: a read-only SQL warehouse beside the team's Confluence knowledge base, Jira boards, Slack channels, and — on some tasks — Freshdesk, Gmail and Drive. Definitions, thresholds and business rules live in those systems, not in the prompt, and the warehouse is salted with plausible near-miss readings the policies rule out.
Each cell is the share of a model's trials on that domain's tasks that cleared every check. The rate is calculated from those individual trials, rather than by averaging each task's rate.
Every task runs against the same seeded company: a read-only SQL warehouse beside the team's Confluence knowledge base, Jira boards, Slack channels, and — on some tasks — Freshdesk, Gmail and Drive. Definitions, thresholds and business rules live in those systems, not in the prompt, and the warehouse is salted with plausible near-miss readings the policies rule out.
GPT-5.5 is highlighted below so you can compare its results with the other models on the same tasks. Each cell is the share of a model's trials on that domain's tasks that cleared every check, calculated from those individual trials rather than by averaging each task's rate.
Every task runs against the same seeded company: a read-only SQL warehouse beside the team's Confluence knowledge base, Jira boards, Slack channels, and — on some tasks — Freshdesk, Gmail and Drive. Definitions, thresholds and business rules live in those systems, not in the prompt, and the warehouse is salted with plausible near-miss readings the policies rule out.
Claude Opus 4.8 is highlighted below so you can compare its results with the other models on the same tasks. Each cell is the share of a model's trials on that domain's tasks that cleared every check, calculated from those individual trials rather than by averaging each task's rate.
Every task runs against the same seeded company: a read-only SQL warehouse beside the team's Confluence knowledge base, Jira boards, Slack channels, and — on some tasks — Freshdesk, Gmail and Drive. Definitions, thresholds and business rules live in those systems, not in the prompt, and the warehouse is salted with plausible near-miss readings the policies rule out.
Gemini 3.1 Pro is highlighted below so you can compare its results with the other models on the same tasks. Each cell is the share of a model's trials on that domain's tasks that cleared every check, calculated from those individual trials rather than by averaging each task's rate.
Every task runs against the same seeded company: a read-only SQL warehouse beside the team's Confluence knowledge base, Jira boards, Slack channels, and — on some tasks — Freshdesk, Gmail and Drive. Definitions, thresholds and business rules live in those systems, not in the prompt, and the warehouse is salted with plausible near-miss readings the policies rule out.
Gemini 3.5 Flash is highlighted below so you can compare its results with the other models on the same tasks. Each cell is the share of a model's trials on that domain's tasks that cleared every check, calculated from those individual trials rather than by averaging each task's rate.
GPT-5.5
Claude Opus 4.8
Gemini 3.1 Pro
Gemini 3.5 Flash
Cards & Interchange12 tasks
43.3%
52.5%
24.2%
2.5%
Risk & Fraud12 tasks
50.8%
35.8%
14.2%
22.5%
Payments & Transfers10 tasks
56%
43%
22%
2%
Compliance & Regulatory7 tasks
44.3%
25.7%
24.3%
2.9%
Reconciliation & Settlement6 tasks
23.3%
30%
5%
3.3%
Revenue & Fees6 tasks
33.3%
25%
18.3%
5%
Customer & Account Analytics5 tasks
40%
34%
2%
0%
Subscriptions & Billing4 tasks
47.5%
27.5%
42.5%
25%
Deposits & Funding2 tasks
55%
10%
0%
0%
Covers 9 domains · 9 subdomains
Measured on 64 tasks · 2,560 trials · 4 models
The verdict
One missed check costs the whole trial. These are exact-value tasks against a live warehouse — an analyst who is almost right about a fee-revenue line is wrong — so a trial counts only where every check passed, and the figure below is the share that did.
The verdict
One missed check costs the whole trial. These are exact-value tasks against a live warehouse — an analyst who is almost right about a fee-revenue line is wrong — so a trial counts only where every check passed, and the figure below is the share that did.
The verdict
One missed check costs the whole trial. These are exact-value tasks against a live warehouse — an analyst who is almost right about a fee-revenue line is wrong — so a trial counts only where every check passed, and the figure below is the share that did.
The verdict
One missed check costs the whole trial. These are exact-value tasks against a live warehouse — an analyst who is almost right about a fee-revenue line is wrong — so a trial counts only where every check passed, and the figure below is the share that did.
The verdict
One missed check costs the whole trial. These are exact-value tasks against a live warehouse — an analyst who is almost right about a fee-revenue line is wrong — so a trial counts only where every check passed, and the figure below is the share that did.
GPT-5.5
44.4%
best model's pass rate
Across 64 tasks attempted 10× each, the strongest model — GPT-5.5 — passes only 44.4% of trials.
GPT-5.5
44.4%
pass rate
Across 64 tasks attempted 10× each, GPT-5.5 passes 44.4% of trials.
Claude Opus 4.8
35.9%
pass rate
Across 64 tasks attempted 10× each, Claude Opus 4.8 passes 35.9% of trials.
Gemini 3.1 Pro
18.3%
pass rate
Across 64 tasks attempted 10× each, Gemini 3.1 Pro passes 18.3% of trials.
Gemini 3.5 Flash
7.7%
pass rate
Across 64 tasks attempted 10× each, Gemini 3.5 Flash passes 7.7% of trials.
The Evidence
Explore all tasksEach card is one task the whole field attempted, headed by its title and opened by the task's own brief. Under it is every model, ranked by the share of its ten trials on that task that cleared every check. Open a card for the full brief, the systems the agent worked against, and how it was graded.
Where GPT-5.5 breaks
Explore all tasksThe three tasks GPT-5.5 does worst on, worst first, out of the 64 the whole field attempted. Each card ranks every model by the share of its 10 trials on that task that cleared every check, so the anchored number is read against the others rather than on its own.
Where Claude Opus 4.8 breaks
Explore all tasksThe three tasks Claude Opus 4.8 does worst on, worst first, out of the 64 the whole field attempted. Each card ranks every model by the share of its 10 trials on that task that cleared every check, so the anchored number is read against the others rather than on its own.
Where Gemini 3.1 Pro breaks
Explore all tasksThe three tasks Gemini 3.1 Pro does worst on, worst first, out of the 64 the whole field attempted. Each card ranks every model by the share of its 10 trials on that task that cleared every check, so the anchored number is read against the others rather than on its own.
Where Gemini 3.5 Flash breaks
Explore all tasksThe three tasks Gemini 3.5 Flash does worst on, worst first, out of the 64 the whole field attempted. Each card ranks every model by the share of its 10 trials on that task that cleared every check, so the anchored number is read against the others rather than on its own.
Payments & Transfers
Transfer balance NSF chain
Funding ops is checking first-time account-funding pulls from linked counterparties that hit insufficient counterparty balance during processing and then failed again after self-healing retried them. Count original funding pulls, not retry rows.
Compliance & Regulatory
Reg CC large-deposit exception (aggregate-per-banking-day) hold exposure
I'm putting together a Reg CC large-deposit hold exposure brief for mobile check deposits (MRDC). I only want deposits where the money actually reached the customer. For any banking day where a customer's deposits crossed the $5,525 large-deposit exception…
Risk & Fraud
Q4 Fraud Impact Report - Email & Ticket Investigation
My lead asked for a Q4 fraud impact report and I need to understand the scope of fraud related issues our customers have been experiencing. Rather than looking at specific cases, I need you to help me discover what is out there across our mailbox and support…
Reconciliation & Settlement
Reconstruct cancelled-check return-then-re-debit ledger and net balance impact
CX escalated a customer complaint in #banking: their check was debited, credited back after the processor returned it, and then debited again. Finance needs to confirm that this customer ultimately paid once, and Ops wants the whole-book exposure before any…
Cards & Interchange
Duplicate-pending complaint audit — CX hold-confusion triage
We keep getting the same escalation. CP-123 is the latest: user 22576 says Enterprise hit their Finoco card. But that member also complained three weeks earlier about a *pending* Enterprise transaction, and CX thinks that is the real pattern.
Reconciliation & Settlement
FBO charge-off sweep vs loss reserve
It's the monthly loss-reserve close and Finance cannot sign off — the charge-off sweep doesn't tie to loss_records, and CX says some closing customers briefly saw $0 instead of their true negative balance. Start in Jira with the Engineering ticket on charge…
Compliance & Regulatory
Billing Write-Off Pack + ACATS Transfer Compliance — Jira + Confluence + SQL + Slack
Finance Ops is pulling together a quarter-end write-off + ACATS compliance pack and wants the findings posted where the team actually works. Please trace the engineering Jira that implements the past-due subscription invoice billing trigger and confirm the…
Cards & Interchange
Duplicate-pending complaint audit — CX hold-confusion triage
We keep getting the same escalation. CP-123 is the latest: user 22576 says Enterprise hit their Finoco card. But that member also complained three weeks earlier about a *pending* Enterprise transaction, and CX thinks that is the real pattern.
Reconciliation & Settlement
Flex Reserve funding: reconstruct in-red Overview balances
Flex Reserve is fully rolled out and the PM needs a launch-health audit before Thursday's review. Start from the Flex Reserve page in Confluence for how Overview bill-reserve balance is defined (active Smart + Flex reserves, and when Overview should show in…
Customer & Account Analytics
Additional-owner OwnerSync write-back coverage audit
ENG-4821 follow-up: OwnerSync owner-write coverage. We need to know how completely the AdditionalPersonApplication OwnerSync pipeline actually fired — start on the 'Add Additional Account Owner' Engineering page in Confluence so you're using the same…
Cards & Interchange
ATM decline vs usage-limit reconciliation
A cardholder has complained through CX in #bugs_in_the_mainframe: they keep getting "limit reached" declines at ATMs even though they're well under their limit and have barely withdrawn anything. Find out whether this is a real, widespread problem, what it's…
Reconciliation & Settlement
Bill-pay lifecycle: reconcile paid bills to ledger transactions
Priya (Bills PM) needs the paid-vs-outstanding tie-out before Zetaverse ships. The Bills calendar and history spec in Confluence ('4.3.4 Bill Cal & History') is clear: a paid bill must show the matching settled debit in history — start there for the lifecycle…
Payments & Transfers
ACH-debit holds released before scheduled settlement
Treasury recon flagged ACH pull-in deposits that showed as available before the processor file actually settled. Our Engineering Confluence page 'TP Updated ACH Api Build Plan' spells out the disbursement rule for ACH-debit deposits (money pulled from an…
Payments & Transfers
ACH debit hold-schedule conformance audit
Ops thinks some inbound ACH pulls are freeing up funds to members earlier than our hold rules allow. The hold policy is written up in the Bank KB Confluence page 'ACH Debit (Pull) Processing & Hold Schedule' — read that first. It tells you when the hold clock…
Risk & Fraud
ACH-return fraud containment blast-radius audit
Fraud ops wants a backfill on ACH-return containment. Priya flagged another forbidden-code return where the owner had other Finoco accounts — per the Lock Handling runbook in Confluence (Product space, page 403 / p.629), when someone is the Creator of an ACH…