I want to understand Germany’s vehicle industryVehicles
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
71%
Claude Opus 5.5
68%
Claude Opus 5
64%
GPT-6 sol
61%
GPT-5.6 sol
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
78m
Claude Opus 5
45m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
110
Claude Opus 5
48
Claude Opus 5.5
48
GPT-6 sol
34
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (28) | ||||
| 100% | 100% | 100% | 100% | |
How to passBegin by establishing the structure of Germany’s vehicle industry. Use authoritative sources such as: https://www.gtai.de/en/invest/industries/mobility/automotive-industry https://www.vda.de/en https://bmdv.bund.de/en https://www.kba.de/EN/Home/home_node.html https://www.bundeswirtschaftsministerium.de/Navigation/EN/Home/home.html Map the industry into: Passenger cars Premium/luxury vehicles Sports/performance vehicles Commercial vehicles Trucks Buses Electric vehicles Plug-in hybrids Hydrogen/fuel-cell vehicles Automotive software Autonomous-driving technology Batteries Charging infrastructure Automotive electronics Powertrain technology Vehicle components Mobility services Motorsport Vehicle testing/certification Aftermarket and servicing Create an initial Germany Automotive Ecosystem Map. | ||||
| 100% | 100% | 100% | 100% | |
How to passResearch the country's major automotive clusters. At minimum investigate: Bavaria Baden-Württemberg Lower Saxony North Rhine-Westphalia Saxony Hesse Saarland Thuringia For each region determine: Major manufacturers Major suppliers Manufacturing plants R&D centers Universities/research institutes EV/battery activity Automotive technology specialization Major automotive cities Create: Region Major Companies Major Plants/R&D Main Vehicle Segment Technology Strength Do not assume that the location of a company's headquarters means all of its manufacturing occurs there. | ||||
| 100% | 100% | 100% | 100% | |
How to passResearch the major German vehicle manufacturers. At minimum investigate: Volkswagen Group BMW Group Mercedes-Benz Group Porsche Audi Opel MAN Truck & Bus Daimler Truck Use the list as a starting point and identify additional relevant German manufacturers. For each company research: Headquarters Major brands Vehicle segments Major markets Manufacturing footprint R&D footprint EV strategy Software strategy Battery strategy Commercial-vehicle strategy Current major vehicle platforms Major technology priorities | ||||
| 100% | 100% | 100% | 100% | |
How to passThis is an important verification step. Create a structure showing: Parent Group → Brand → Vehicle Segment For example, determine which brands belong to Volkswagen Group and distinguish them from independent manufacturers. Do the same for other groups where relevant. For each brand identify: Ownership/group Positioning Vehicle categories Major markets Technology focus Do not count a parent group and each of its brands as completely independent manufacturers when calculating industry presence. | ||||
| 100% | 100% | 100% | 100% | |
How to passCreate a detailed brand comparison covering German passenger-car brands. Include relevant brands such as: Volkswagen Audi BMW Mercedes-Benz Porsche Opel MINI where German-group ownership is relevant Smart where relevant to the German automotive ecosystem For each brand compare: Brand Positioning Main Segments EV Range Performance Focus Major Markets Technology Focus Clearly distinguish German ownership from German manufacturing where the distinction matters. | ||||
| 100% | 100% | 100% | 100% | |
How to passCreate a dedicated EV research track. Investigate: Battery-electric passenger cars Electric SUVs Electric sedans Electric hatchbacks Electric performance cars Electric vans Electric trucks Electric buses For major German brands identify current EV models and record: Model Body style Battery capacity where officially stated Claimed range Charging capability Drive configuration Performance specifications Starting price where officially available Model availability Official product page Do not compare range figures without checking whether the measurement standard is comparable. Where possible, distinguish: WLTP from other testing standards. | ||||
| 100% | 100% | 100% | 100% | |
How to passDo not make the vehicle analysis exclusively about EVs. Investigate: Mild hybrids Full hybrids Plug-in hybrids Diesel-electric applications where relevant For selected models compare: Powertrain Electric-only capability Battery size Fuel type Claimed efficiency Intended use Advantages Limitations Use manufacturer specifications and official regulatory information. | ||||
| 100% | 100% | 100% | 100% | |
How to passCreate a dedicated performance-vehicle section. Investigate: Porsche BMW M Mercedes-AMG Audi Sport Volkswagen performance models Other German specialist manufacturers Research: Sports cars Performance sedans Performance SUVs Track-focused vehicles High-performance EVs For selected vehicles compare: Model Engine/Powertrain Power 0–100 km/h Drive Performance Positioning Official Source Use manufacturer specifications wherever available. Do not treat claimed acceleration times as independent performance testing. | ||||
| 100% | 100% | 100% | 100% | |
How to passInvestigate Germany's commercial-vehicle ecosystem. Cover: Light commercial vehicles Vans Trucks Buses Long-haul vehicles Urban delivery vehicles Electric commercial vehicles Hydrogen/fuel-cell vehicles Research: Volkswagen Commercial Vehicles Mercedes-Benz Vans MAN Daimler Truck Other relevant German manufacturers For each segment identify: Major models Powertrain technologies Payload/capacity Intended applications Electrification strategy Major markets | ||||
| 100% | 100% | 0% | 100% | |
How to passGermany's automotive ecosystem is not limited to vehicle manufacturers. Investigate major suppliers such as: https://www.bosch.com https://www.zf.com https://www.continental.com https://www.schaeffler.com https://www.mahle.com https://www.webasto.com/en-int.html https://www.hella.com https://www.brose.com Identify additional significant suppliers. For each determine: Major components Electronics Software Powertrain Braking Steering Thermal management Interior systems ADAS EV components Battery technologies Create a supplier map: Supplier → Component/Technology → Vehicle Application → Major Manufacturers Served Do not claim a specific OEM-supplier relationship unless supported by a reliable source. | ||||
| 100% | 0% | 100% | 100% | |
How to passConstruct the broader German vehicle supply chain: Raw Materials → Components → Electronics → Powertrain → Battery → Vehicle Manufacturing → Software → Distribution → Service → Recycling For each stage identify representative German companies. Then create separate pathways for: Internal-combustion vehicles Battery-electric vehicles Commercial vehicles Software-defined vehicles Identify which components are changing most rapidly because of electrification and software. | ||||
| 100% | 100% | 100% | 100% | |
How to passInvestigate Germany's transition toward software-defined vehicles. Research: Vehicle operating systems Infotainment OTA updates Connected cars Cloud services Digital dashboards ADAS Autonomous driving AI in vehicles Cybersecurity Vehicle data Software development platforms Investigate Volkswagen Group's software activities as well as technology initiatives from BMW, Mercedes-Benz and major suppliers. For every major technology: Identify the company Technology name Purpose Current deployment Vehicle models where publicly documented Future direction Official source Separate production technology from announced future technology. | ||||
| 100% | 100% | 0% | 0% | |
How to passResearch Germany's work in: Adaptive cruise control Lane keeping Automated parking Highway assistance Traffic-jam assistance Automated driving Level 2 systems Level 3 systems Higher automation research Use: KBA Federal government sources Manufacturer documentation Reputable technical sources Clearly distinguish SAE automation levels where applicable. Do not call a Level 2 driver-assistance system "self-driving" without qualification. For each significant system record: System Company Automation Level Current Availability Vehicle/Market Main Capability | ||||
| 100% | 0% | 0% | 100% | |
How to passInvestigate Germany's battery ecosystem. Cover: Battery cell production Battery packs Battery management systems Battery materials Recycling Second-life batteries Gigafactories Battery R&D Research relevant German companies and major manufacturing projects. For each facility/company determine: Location Technology Production status Planned capacity where officially disclosed Vehicle/customer connection where verified Current operating status Do not treat an announced factory as an operating factory. Clearly classify: Operating → Under construction → Announced → Cancelled/Delayed | ||||
| 100% | 100% | 100% | 100% | |
How to passInvestigate Germany's EV charging ecosystem. Use: Bundesnetzagentur — Charging Infrastructure Bundesministerium für Digitales und Verkehr Major charging-network operators Vehicle manufacturers Research: Public chargers Fast charging Ultra-fast charging Highway charging Urban charging Home charging Workplace charging Charging-network interoperability Payment methods Do not use a single charging-network map as proof of nationwide availability. | ||||
| 100% | 100% | 100% | 0% | |
How to passUse KBA data to understand the German vehicle market. Investigate: New passenger-car registrations EV registrations Plug-in hybrids Diesel Petrol Commercial vehicles Market shares Manufacturer/brand registrations Use: KBA — Vehicle Statistics When using current statistics: Identify the reporting period. Record the source publication date. Distinguish registrations from sales. Distinguish brands from parent groups. Do not compare incomplete periods with full-year data. Create a market-data table. | ||||
| 0% | 0% | 0% | 0% | |
How to passInvestigate Germany's used-vehicle ecosystem. Research: Major used-car platforms Manufacturer-certified used vehicles Private sales Dealer sales EV used-car market Used diesel market Used petrol market Potential platforms include: mobile.de AutoScout24 Germany Compare: Vehicle selection Price ranges Age Mileage Powertrain Dealer/private availability Warranty information Financing where relevant Do not treat listing prices as final transaction prices. | ||||
| 0% | 0% | 0% | 0% | |
How to passBuild a practical ownership-cost framework. Investigate: Purchase price Registration Vehicle tax Insurance Fuel Electricity Maintenance Tires Inspection Repairs Depreciation Parking Charging Use official German sources for taxes and inspection requirements. Investigate: TÜV Association Federal Ministry of Finance — Vehicle Tax Do not produce a single universal ownership cost. Instead create example ownership profiles for: Small petrol car Diesel family car Battery-electric car Premium car Commercial vehicle Clearly label assumptions. | ||||
| 0% | 0% | 0% | 0% | |
How to passInvestigate the practical regulatory environment. Cover: Vehicle registration Periodic inspection TÜV/HU Emissions requirements Winter tires Vehicle modifications Imported vehicles EV-specific considerations Used-car documentation Use KBA and official German government sources. Separate: Legal requirement from Recommended practice and Industry convention. | ||||
| 100% | 100% | 100% | 100% | |
How to passIdentify major German vehicle manufacturing and technology facilities. For selected companies, investigate: Plant location Vehicles produced Production status EV production Battery production R&D activities Employment where publicly disclosed Future investment Create: Company Facility City Main Production/R&D EV Activity Status Official Source Do not infer that a facility manufactures every model sold by the company. | ||||
| 0% | 0% | 0% | 0% | |
How to passInvestigate Germany's contribution to automotive culture. Cover: Nürburgring German touring-car racing Formula 1-related German manufacturers/teams Porsche motorsport BMW M Motorsport Mercedes-AMG motorsport Audi motorsport Automotive museums Driving culture Performance-car communities Use official track, museum, manufacturer and motorsport sources. Distinguish: Road-car technology from Motorsport technology. Do not assume racing technology is directly available in consumer vehicles. | ||||
| 0% | 0% | 0% | 0% | |
How to passIdentify major automotive museums. Research Mercedes-Benz Museum, BMW Museum, Porsche Museum, Audi museum mobile, and Volkswagen Auto museum. For each investigate: Location, Main collection, Major historical vehicles, Current exhibitions, and Factory tours where available. Verify current information from official sources. | ||||
| 0% | 100% | 0% | 100% | |
How to passCreate a comprehensive comparison of at least 10 major brands. Compare: Brand Segment Key Models EV Strategy Performance Technology Main Market Position Include a mixture of: Mass-market Premium Luxury Performance Commercial vehicles Do not rank brands simply by prestige. | ||||
| 0% | 0% | 0% | 0% | |
How to passSelect at least 20 current or recently relevant German-market vehicles across categories. Cover: Hatchbacks Sedans SUVs Sports cars EVs Plug-in hybrids Vans Commercial vehicles For each compare: Starting price Powertrain Claimed efficiency/range Size Performance Cargo capacity where relevant Charging capability where applicable Safety/ADAS Warranty where available Use official manufacturer specifications. Do not mix different model years without explicitly identifying them. | ||||
| 0% | 0% | 0% | 0% | |
How to passCompare: Manufacturer dealerships Authorized dealers Online manufacturer sales Used-car platforms Private sellers Certified pre-owned programs For each channel investigate: Price transparency Warranty Financing Trade-in Vehicle inspection Return/cancellation rights where applicable Consumer protection Use official German consumer-protection sources when discussing legal rights. | ||||
| 100% | 100% | 100% | 100% | |
How to passIdentify major changes affecting Germany's automotive industry. Research: Electrification Software-defined vehicles AI Autonomous driving Battery manufacturing Charging infrastructure Chinese competition Supply-chain localization Automotive manufacturing automation Hydrogen Sustainable materials Recycling Connected vehicles For every trend identify: Current status Major German companies involved Government/industry initiatives Evidence of investment Major challenges Announced future developments Separate current reality from future announcements. | ||||
| 100% | 100% | 100% | 100% | |
How to passPerform a dedicated verification pass. For every major company verify: Ownership Headquarters Major brands Vehicle segments Manufacturing locations Technology claims For every major vehicle verify: Model name Powertrain Current availability Official specifications For every market statistic verify: Source Reporting period Definition Whether it measures registrations, production or sales Prioritize: KBA → German government → VDA → manufacturer → major supplier → reputable industry source when applicable. | ||||
| 100% | 0% | 100% | 100% | |
How to passProduce a comprehensive report containing: 1. Executive Summary Explain: Germany's automotive strengths Major manufacturers Major suppliers Leading technology areas EV transition Current market direction 2. Germany Automotive Ecosystem Map Show: Manufacturers → Suppliers → Technology → Infrastructure → Mobility Services 3. Major Manufacturer & Brand Directory Include at least 10 major brands/groups. 4. Vehicle Database Include at least 20 representative German-market vehicles across different categories. 5. EV & Hybrid Guide Compare: BEVs PHEVs Hybrid systems Charging Range Ownership considerations 6. Commercial Vehicle Guide Cover: Vans Trucks Buses Electric commercial vehicles Hydrogen/fuel-cell developments 7. Automotive Supplier Directory Include at least 15 major German automotive suppliers, with their technology and supply-chain roles. 8. Technology Guide Cover: Automotive software ADAS Autonomous driving Batteries Charging AI Automotive electronics Connectivity 9. Automotive Region Map Compare Germany's major automotive clusters and cities. 10. Market Data Use current KBA data to show: Vehicle registrations EV adoption Powertrain trends Major brands/groups 11. Used-Car Market Compare major platforms and explain how to evaluate German used vehicles. 12. Ownership & Regulation Guide Cover: Vehicle tax Insurance Inspection Maintenance Registration Charging Other important ownership considerations 13. Motorsport & Automotive Culture Cover major German motorsport institutions, tracks, museums and enthusiast culture. 14. Automotive Visitor Guide Automotive Heritage Guide Explain the industry significance of major museums, factory tours and automotive experiences. 15. Future Technology & Industry Outlook Identify the most important developments in: EVs Batteries Software AI Autonomous driving Hydrogen Charging Manufacturing Clearly distinguish current products from announced/future projects. 16. Master Company & Vehicle Tables Provide consolidated tables for: Manufacturers Brands Suppliers Vehicles Technology companies Charging providers Museums/experiences Do not fabricate vehicle specifications, prices, production locations, ownership relationships, market statistics, charging availability, factory status, or technology capabilities. Verify current specifications and availability against official sources. Treat prices, vehicle availability, market statistics, charging availability and regulatory information as time-sensitive. Clearly distinguish German companies from companies that merely manufacture vehicles in Germany. | ||||
Failure modes on this task
38 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Standing up the evidence1 mode | 2 | 3 | 5 | 2 |
| Delivering the result1 mode | 1 | 2 | 2 | 1 |
| No signal in the judge’s words1 mode · the grader | 5 | 5 | 4 | 6 |