I’m interested in French bread and wantFood and Drink
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
86%
Claude Opus 5.5
71%
GPT-6 sol
57%
GPT-5.6 sol
43%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
22m
Claude Opus 5
17m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
118
Claude Opus 5
75
Claude Opus 5.5
64
GPT-5.6 sol
55
GPT-6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (7) | ||||
| 100% | 0% | 100% | 100% | |
How to passEstablish the criteria that will be used consistently when comparing French traditional breads: bread name and recognised alternative names, regional association, documented historical or cultural background, ingredients where explicitly documented, preparation or fermentation method where documented, shape or physical characteristics where stated, traditional serving or culinary context, official quality or geographical designation where applicable, legal or specification requirements where applicable, source type, and documented limitations. Distinguish between a legally protected product, a recognised regional tradition, and a general bread style. Do not assume that a bread's popularity or association with a French region means that it has an official protected designation. | ||||
| 100% | 100% | 100% | 100% | |
How to passFind at least five publicly accessible authoritative sources from French government or cultural institutions (such as INAO — inao.gouv.fr — for French designations), official European Union food-quality resources, recognised French culinary organisations, museums, or other established institutions explaining French food heritage, geographical indications, traditional specialities, or bread-related regulations and cultural practices. Open the actual pages and record what they explicitly state about protected designations, geographical origin, traditional recipes or specifications, and the distinction between protected products and ordinary culinary traditions. Record exact URLs and use "not stated" for requested information that a source does not explicitly provide. Do not treat a general description of French culinary heritage as proof that a particular bread has legal protection. | ||||
| 100% | 100% | 0% | 0% | |
How to passFind at least eight French breads or bread styles with publicly documented regional or historical associations. Include breads from different regions where reliable sources permit. For each candidate, record the exact name, regional association, documented origin or historical context, ingredients where explicitly stated, preparation characteristics where documented, traditional use or serving context where stated, and any official designation or specification if one exists. Open the actual source pages and prefer institutional or recognised culinary sources over unsourced food blogs. If sources disagree about the history or regional association, preserve the disagreement rather than choosing a version without evidence. | ||||
| 100% | 100% | 100% | 100% | |
How to passFor the strongest candidates, cross-check whether the bread or its associated product name has an official French or European protection, such as a recognised geographical indication or traditional-speciality designation, using the relevant official database (such as INAO's search tool or the EU's eAmbrosia geographical indications register) where applicable. Record the exact protected name, registration or specification information where publicly available, geographical scope, and documented requirements. For breads without such protection, explicitly record that the researched sources did not establish an applicable protection rather than claiming that no protection exists anywhere. Distinguish official specification requirements from descriptions provided by culinary or historical sources. | ||||
| 100% | 100% | 100% | 0% | |
How to passCompare the selected breads using documented ingredients, fermentation or preparation methods, flour or grain information where stated, physical characteristics, and regional context. Identify meaningful differences between the breads without assigning arbitrary authenticity scores. Where multiple sources describe different preparation methods or ingredients for the same bread, preserve those differences and explain whether they appear to relate to regional variation, historical practice, source context, or a protected specification. Do not convert a modern recipe into a historical fact or treat one baker's recipe as the universal traditional method. | ||||
| 100% | 100% | 0% | 0% | |
How to passSearch publicly readable French baking, food, or France-related communities for discussions where multiple distinct contributors discuss traditional French breads, regional differences, baking methods, ingredient substitutions, or what they consider important when making or buying traditional breads. Prefer discussions that provide enough context to identify the bread or regional tradition being discussed. Open the actual discussions and record their exact URLs. Summarise recurring practical observations and disagreements, clearly labelling them as "community-reported observations rather than authoritative historical or regulatory information." Do not use community comments to establish protected status, legal definitions, or historical facts when authoritative sources provide different information. | ||||
| 0% | 0% | 0% | 0% | |
How to passCreate a concise Markdown document named french_traditional_bread_guide.md containing the following sections: Research Scope and Definitions; Official Protection and Heritage Framework; Traditional Bread Profiles; Regional Comparison; Protected vs Traditional Examples; Preparation and Ingredient Differences; Community-Reported Observations; Reference Checklist; and Sources. The Traditional Bread Profiles section must contain at least eight researched breads or bread styles from multiple French regions where reliable sources support the regional association. The Protected vs Traditional section must clearly distinguish official geographical or traditional-speciality protection from cultural or culinary tradition and must not infer legal status where official evidence is unavailable. The Regional Comparison should identify documented similarities and differences without assigning arbitrary authenticity scores. The Reference Checklist should help a reader evaluate a claim about a traditional French bread by checking its region, source, documented ingredients or method, historical evidence, and official designation where relevant. Clearly separate government or institutional information, culinary sources, and community observations. Include the exact URL for every source used and add a "Researched On" field. | ||||
Failure modes on this task
10 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | — | 2 | 1 | 3 |
| Standing up the evidence1 mode | 1 | — | 2 | 1 |