I’m helping a group of EV drivers decideVehicles
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
90%
Claude Opus 5.5
50%
GPT-6 sol
10%
GPT-5.6 sol
0%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
77m
Claude Opus 5
0m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
113
Claude Opus 5
33
GPT-6 sol
23
Claude Opus 5.5
1
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (10) | ||||
| 100% | 0% | 100% | 0% | |
How to passEstablish the evaluation framework before comparing networks. Define the difference between a charger being installed, listed, operational, available, accessible, compatible, affordable, and practically usable. Establish the dimensions that matter to an EV driver: connector compatibility, charging speed, reliability, payment access, app or account requirements, pricing, idle or parking charges, opening arrangements, accessibility, maintenance, location quality, and practical restrictions. Separate network-wide characteristics from individual-site conditions and define how evidence from each level should be treated. Identify which findings can be verified through official sources and which may require independent or driver evidence. | ||||
| 100% | 100% | 0% | 0% | |
How to passResearch public charging networks operating across a meaningful selection of cities or regions and identify the strongest candidates for deeper investigation. Use official network, government, municipal, or public charging-information sources to establish network coverage, charger types, connector availability, stated charging speeds, operating arrangements, payment methods, app/account requirements, and pricing structures. Do not select candidates simply because they have the largest number of chargers. Filter the landscape according to practical usability criteria from Step 1 and retain enough variation to expose meaningful differences between network models and locations. | ||||
| 100% | 100% | 0% | 0% | |
How to passFor the shortlisted networks, investigate how a driver actually gains access to charging. Verify connector standards, supported vehicle types where stated, charging-speed limitations, payment methods, contactless availability, app or account requirements, subscription requirements, roaming arrangements, parking restrictions, idle fees, accessibility provisions, opening hours or access restrictions, and any conditions that could prevent a driver from using the charger as expected. Prefer official network, municipal, transport-authority, or site-operator sources. Record where information differs between network-level documentation and individual charging-location pages. | ||||
| 0% | 100% | 0% | 0% | |
How to passCompare the pricing structures of the shortlisted networks rather than comparing a single headline charging price. Investigate energy-based pricing, time-based charges, connection fees, subscription requirements, idle fees, parking charges, minimum charges, payment surcharges, and other mandatory or conditional costs where publicly documented. Distinguish standard pricing from temporary promotions. Where the final cost depends on vehicle behavior, parking arrangements, charging duration, or membership status, explain the dependency rather than presenting a misleading single figure. Use current official pricing information and record when a price is location-specific or otherwise uncertain. | ||||
| 100% | 100% | 0% | 0% | |
How to passFor representative charging locations from the shortlisted networks, compare official availability and operational information with credible driver experiences. Search public driver reviews, EV communities, automotive publications, and other accessible sources for reports of broken chargers, unavailable connectors, blocked bays, payment failures, inaccurate status information, unexpectedly low charging speeds, access problems, or maintenance issues. Distinguish isolated reports from recurring patterns and do not treat repeated copies of the same complaint as independent evidence. Where possible, check whether reported problems are acknowledged or contradicted by the operator or public authority. Record the date and context of driver reports so older experiences are not presented as proof of current conditions. | ||||
| 100% | 0% | 0% | 0% | |
How to passSelect representative sites where official information and driver evidence reveal potential discrepancies and investigate them in depth. Determine whether the apparent conflict is caused by outdated availability data, site-specific restrictions, payment or access requirements, maintenance, parking rules, vehicle compatibility, temporary faults, or differences in what the operator means by availability. Separate genuine contradictions from situations where the sources describe different conditions. Identify which source is more reliable for each question and preserve unresolved uncertainty where the evidence cannot establish the cause. | ||||
| 100% | 0% | 0% | 0% | |
How to passApply the evidence to a finite set of realistic charging scenarios within the same geographic market(s) selected for the shortlist: a driver who refuses to install multiple charging apps, a driver needing rapid charging during a long trip, a driver using contactless payment only, a driver arriving when a site is busy, a driver whose preferred connector is unavailable, and a driver relying on a charger listed as available but finding it unusable. For each scenario, determine which networks or charging arrangements remain practical, what restrictions or additional costs arise, and where the evidence is insufficient. Use market-appropriate evidence and do not combine pricing, access rules, or network comparisons across different geographic markets unless the geographic difference is explicitly identified. Use the findings to expose differences that a simple charger-count comparison would miss. | ||||
| 100% | 100% | 0% | 0% | |
How to passChallenge the emerging conclusions using a finite set of failure modes relevant to public charging: a network with many chargers but poor practical reliability, a highly reliable site with unusually restrictive payment requirements, a driver complaint that appears systemic but is actually isolated, an official availability status that conflicts with multiple recent user reports, and a charging network whose apparent price advantage disappears after parking or idle fees. Check whether network-level conclusions are being incorrectly inferred from individual sites and whether driver anecdotes are being given more weight than the evidence supports. Correct any unsupported conclusions and identify important evidence gaps. | ||||
| 100% | 0% | 0% | 0% | |
How to passSynthesize the research into a reusable framework for evaluating public EV charging networks and individual charging sites in other cities or regions. Define the evaluation criteria, evidence hierarchy, method for handling official versus user-reported information, rules for distinguishing isolated incidents from recurring problems, treatment of missing or conflicting information, and a practical scoring or classification approach only where it can be justified without false precision. The framework should allow a future researcher to add a new network or site without changing the underlying methodology. | ||||
| 100% | 0% | 0% | 0% | |
How to passPresent the final comparison using the framework developed above. For each shortlisted network or representative charging option, explain its strongest practical advantages, important limitations, access and payment requirements, pricing risks, reliability evidence, and confidence level. Clearly separate verified official facts from driver-reported experiences and inference. Identify credible alternatives and explain which circumstances would make an alternative preferable. Conclude with practical guidance for different types of EV drivers rather than declaring one universal winner. Include the source pages supporting important factual claims and identify what future evidence could change the conclusions. | ||||
Failure modes on this task
25 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | 1 | 4 | 1 | 5 |
| Standing up the evidence1 mode | — | — | 1 | 1 |
| No signal in the judge’s words1 mode · the grader | — | 1 | 7 | 4 |