I want to explore France’s visual artsArts & Entertainment
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
87%
Claude Opus 5.5
47%
GPT-6 sol
13%
GPT-5.6 sol
0%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
28m
Claude Opus 5
7m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
117
Claude Opus 5
87
Claude Opus 5.5
51
GPT-6 sol
12
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (15) | ||||
| 100% | 100% | 0% | 0% | |
How to passBegin by identifying the major institutions and networks that shape visual arts and design in France. Research across: Fine arts Contemporary art Photography Graphic design Industrial/product design Decorative arts Architecture/design Digital/new-media art Illustration Sculpture and installation Fashion-related design where it overlaps with visual/design practice Start with authoritative sources such as: https://www.culture.gouv.fr/ https://www.louvre.fr/ https://www.musee-orsay.fr/fr https://www.centrepompidou.fr/en/ https://palaisdetokyo.com/ https://www.grandpalais.fr/fr Identify at least 20 significant institutions, organizations, museums, galleries, schools, fairs, or networks. For each, record: Name City/region Main discipline Historical/contemporary/design focus Primary audience Major programs Official website Whether it is primarily a museum, gallery, educational institution, government body, fair, foundation, or cultural organization Output: Create a structured map of France's visual-arts and design ecosystem. | ||||
| 100% | 100% | 100% | 0% | |
How to passDo not limit the research to Paris. Identify at least six French cities or regions with documented visual-arts or design activity. Include Paris and select the remaining locations from places such as Lyon, Marseille, Lille, Bordeaux, Strasbourg, Nantes, Saint-Étienne, Nice, or other locations supported by the research. For each location, record major museums, contemporary-art spaces, design institutions, galleries, art or design schools, recurring events, artist or residency opportunities, public-art initiatives, and notable cultural districts where these are publicly documented. Record “not stated” rather than forcing every category to be present for every location. Summarize what distinguishes each location using observable factors such as concentration of institutions, disciplinary strengths, schools, recurring events, artist-support infrastructure, or documented programmes rather than subjective claims that one city is inherently better. Output: Produce a city comparison table showing the evidence-backed reasons different users might visit, study, work, or participate in each location. | ||||
| 100% | 0% | 0% | 0% | |
How to passOpen the official websites of at least ten major museums identified through the ecosystem research. For each institution, investigate its permanent collections, collection strengths, current or upcoming exhibitions at the time of research, contemporary-art programming where applicable, design or decorative-arts holdings where applicable, photography or new-media programming where applicable, educational programmes, public events, ticketing and access information, and official collection databases or online archives. Record the research date for changing information such as exhibitions, events, prices, and access arrangements. Use the institution’s own pages for collection descriptions rather than tourism or comparison sites where official information is available. Record “not stated” for requested fields that cannot be verified. Output: Create a museum comparison table covering at least ten major institutions. | ||||
| 100% | 100% | 0% | 0% | |
How to passIdentify at least 15 contemporary-art venues across France, using a mixture of national institutions, independent art centres, commercial galleries, artist-run spaces, nonprofit organizations, and regional contemporary-art centres where reliable public information permits. For each venue, use its official website or an authoritative institutional directory to record venue type, city, disciplinary focus, current or recently documented programme, artists exhibited or represented where stated, admission or access information, and institutional partnerships or art-fair participation where documented. Record the research date for changing programme information. Where the venue explicitly positions itself around emerging artists, established artists, or both, record that description; otherwise use “not stated” rather than inferring the category. Cnap directories may be used for discovery, but facility-specific claims should be checked against the venue or organizer where possible. Output: Produce a contemporary-art venue shortlist with source links and evidence limitations. | ||||
| 100% | 100% | 0% | 0% | |
How to passResearch the French design ecosystem separately from fine-art institutions. Investigate institutions and organizations involved in: Industrial design Product design Furniture Graphic design Interior design Digital design Design research Craft/design Collectible design Include major institutions and schools such as relevant design programs identified through: https://www.culture.gouv.fr/ https://www.ensci.com/ https://beauxartsparis.fr/fr Research what each institution actually offers rather than assuming that an institution with "design" in its name covers every discipline. For each institution, record: Discipline Programs Research Exhibitions/events Notable alumni/designers where officially documented Admissions or participation requirements Location Official source The French Ministry of Culture also operates initiatives supporting research in accredited art and design schools, including the RADAR program. Output: Produce a French design-institution comparison. | ||||
| 100% | 0% | 100% | 0% | |
How to passUsing official institutional and organizer calendars, identify up to 20 exhibitions or major visual-arts or design events that are current or upcoming at the time of research, covering multiple cities and disciplines where possible. For each event, record the event or exhibition name, institution or organizer, city, opening date, closing date where stated, discipline, featured artist or designer where stated, main theme, ticket or admission information where published, official event page, and research date. Verify dates on the official event or organizer page rather than relying only on third-party calendars. Exclude events that had already ended before the research date. If fewer than 20 qualifying events can be verified publicly, report all qualifying events rather than filling the list with expired or weakly sourced entries. Output: Create a chronological research-time exhibition and event calendar. | ||||
| 100% | 0% | 0% | 0% | |
How to passInvestigate major France-based fairs relevant to visual art and design, including Art Paris, Paris Photo, the current Paris fair or successor associated with major contemporary-art-market activity, Design Miami Paris where currently operating, and other significant fairs identified through the research. For each fair, open its current official website and record the latest announced or most recent completed edition, dates, location, disciplinary focus, participating galleries or designers where published, public programming, audience orientation, ticket or access information, source URL, and research date. Clearly distinguish a forthcoming edition from a past edition and record “not yet announced” where future dates or participants are unavailable. Output: Produce a fair comparison table based on information verified at the time of research. | ||||
| 100% | 0% | 0% | 0% | |
How to passResearch documented routes for discovering emerging or recently highlighted artists and designers in France, using sources such as Cnap programmes, art-school graduate exhibitions, institutional exhibitions, gallery programmes, residencies, prizes, supported projects, design-school showcases, and publicly documented calls or awards. Identify at least 15 artists or designers whose recent connection to the French visual-arts or design ecosystem can be verified. For each person, record name, discipline, current or recent institutional, educational, gallery, residency, prize, or supported-project connection, relevant city or country connection where stated, official artist/gallery/institution page, notable project where documented, and the specific evidence that justifies including them as emerging or recently highlighted. Do not classify someone as emerging purely because of age or subjective reputation. Output: Produce an evidence-backed emerging-talent discovery list. | ||||
| 100% | 100% | 0% | 0% | |
How to passInvestigate publicly documented opportunities available to artists or designers working in or with France, starting with Cnap, the Ministère de la Culture, DRACs, regional cultural institutions, and other authoritative programme providers. Identify up to ten current, recurring, or recently documented opportunities spanning categories such as grants, project support, residencies, fellowships, exhibition calls, production support, publishing support, prizes, professional development, or studio/workspace opportunities. For each opportunity, record provider, discipline, eligibility, nationality or residency requirements where stated, application requirements, funding or support offered, current deadline or documented application cycle, duration where stated, official application page, status, and research date. Do not present an expired one-time opportunity as currently open. If fewer than ten qualifying opportunities can be verified, report all qualifying opportunities and identify the gap. Output: Create a dated opportunity tracker. | ||||
| 0% | 0% | 0% | 0% | |
How to passIdentify at least ten French institutions offering meaningful education in visual arts, design, or related creative disciplines, using a mixture of fine-arts schools, design schools, public institutions, specialised schools, and postgraduate or research programmes. For each institution, investigate undergraduate and postgraduate programmes where applicable, major disciplines, documented language of instruction, international-student information, admissions requirements, portfolio requirements where stated, tuition or fees where publicly stated, application timeline, location, official admissions page, and research date. Do not infer that a programme is taught in English merely because an English-language website exists. Record “requires confirmation” where language, fees, or admissions information is unclear. Output: Produce an education comparison table. | ||||
| 100% | 0% | 0% | 0% | |
How to passResearch how someone can experience French visual arts remotely. Identify at least 10 official digital resources, such as: Online museum collections Digital archives Artist databases Virtual exhibitions Digitized catalogs Design archives Public collections For each resource: Identify the institution. Explain what can be accessed online. Record whether the collection is searchable. Identify whether images can be viewed at high resolution. Record any usage/copyright restrictions. Provide the official access page. Cnap, for example, maintains an online collection database containing tens of thousands of works acquired by the French state since 1791. Output: Create a digital-resource guide. | ||||
| 0% | 0% | 0% | 0% | |
How to passExpand beyond museums and galleries. Research French creative industries where visual arts and design intersect with entertainment, including: Film production design Animation Illustration Photography Fashion imagery Stage/set design Digital art Video games Music visual identity Festivals Immersive experiences Identify at least 10 relevant institutions, festivals, studios, or cultural events. For each, determine: Creative discipline Role in the French entertainment ecosystem Notable programs/projects Public accessibility Opportunities for artists/designers Official website Keep the focus on the user's visual-arts/design niche rather than turning the task into a general French entertainment guide. | ||||
| 100% | 100% | 0% | 0% | |
How to passUsing the previous research, create four practical pathways aligned to the task: A. Visitor / art enthusiast — identify relevant museums, exhibitions, contemporary spaces, design destinations, fairs, and public programmes. B. Student / prospective learner — identify relevant schools and programmes, admissions routes, portfolio expectations, language information, and international-student considerations. C. Artist / designer — identify grants, residencies, calls, exhibitions, professional networks, development opportunities, and routes to portfolio visibility. D. Independent researcher / emerging-scene explorer — identify artist databases, graduate shows, regional institutions, digital collections, artist programmes, contemporary directories, and other channels useful for discovering less-established artists, designers, or regional activity. For each pathway, explain why the selected resources are useful for that objective using evidence from the earlier research. Do not force one institution into every category or declare one option universally best. | ||||
| 100% | 100% | 0% | 0% | |
How to passSelect the 15 most important recommendations from the research and perform a final verification using the relevant institution’s or organizer’s current official website. Where appropriate, cross-check with another authoritative source such as the Ministry of Culture, Cnap, an institutional network, official event organizer, or school admissions page. Re-check current status, dates, location, eligibility, admissions or application requirements, programme availability, official URL, and research date. If sources conflict, record the discrepancy and determine whether it results from different editions, programme years, definitions, or publication dates. Prefer the source that is most directly authoritative and applicable to the specific claim rather than automatically selecting whichever page is newest. Do not preserve outdated dates or requirements as current facts. | ||||
| 100% | 0% | 0% | 0% | |
How to passProduce a comprehensive final dossier containing: 1. Executive overview — explain the structure of France’s visual-arts and design ecosystem and summarize the strongest evidence-backed options for different objectives without declaring one city or institution universally best. 2. Major institutions — include at least 20 verified institutions with name, city, discipline, type, key documented strength, current or recently documented programme where relevant, and official source. 3. City comparison — compare at least six cities or regions using documented museums, contemporary art, design, galleries, schools, events, artist-support infrastructure, and other relevant ecosystem evidence. 4. Exhibition and event calendar — include up to 20 verified exhibitions or major events that are current or upcoming at the research date. If fewer than 20 qualify, report all qualifying entries rather than using expired events. 5. Art and design fairs — compare relevant France-based fairs by discipline, audience, timing, location, public access, and current edition status. 6. Emerging artists and designers — include at least 15 people with evidence showing their recent institutional, educational, residency, prize, gallery, or supported-project connection to the French scene. 7. Grants, residencies, and professional opportunities — include up to ten verified current, recurring, or recently documented opportunities with eligibility, support, application status or cycle, and official source. 8. Education guide — include at least ten relevant schools or programmes with admissions, portfolio, language, fee information where stated, and official admissions links. 9. Digital resources — include at least ten official online collections, archives, databases, or other digital visual-arts/design resources. 10. Cross-disciplinary visual/design resources — include relevant connections to areas such as animation, illustration, photography, scenography, fashion imagery, or digital/immersive visual practice without expanding into a general entertainment-industry guide. 11. Four practical pathways — provide separate evidence-backed pathways for Visitor / Art Enthusiast, Student / Prospective Learner, Artist / Designer, and Independent Researcher / Emerging-Scene Explorer. 12. Verification and caveats — clearly distinguish official institutional facts, institution-reported claims, independently corroborated information, secondary-source context, and information requiring confirmation. For dynamic information such as exhibitions, fair dates, ticket prices, application deadlines, admissions, and programme availability, record the research date and use the latest applicable official information available at that time. Final deliverable: A verified, practical France Visual Arts & Design Guide connecting institutions, cities, exhibitions, contemporary artists and designers, design events, education, professional opportunities, digital resources, and cultural experiences across France. | ||||
Failure modes on this task
38 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | 2 | 5 | 8 | 12 |
| Standing up the evidence1 mode | — | 1 | 1 | 1 |
| No signal in the judge’s words1 mode · the grader | — | 2 | 4 | 2 |