Research publicly available information on Carnival in Br…Community and Society
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
50%
GPT-5.6 sol
33%
Claude Opus 5.5
33%
GPT-6 sol
33%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
38m
Claude Opus 5
17m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
79
Claude Opus 5
77
Claude Opus 5.5
53
GPT-5.6 sol
31
GPT-6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (6) | ||||
| 100% | 100% | 100% | 0% | |
How to passNavigate to https://visitbrasil.com/en/festivity/carnival-en/. Record the description of Brazilian Carnival, its main cities of celebration, the relative timing linked to Easter, and the listed festivities such as street blocks and samba school parades. Note the exact page URL and the key cultural statements. | ||||
| 0% | 0% | 0% | 100% | |
How to passNavigate to https://turismo.prefeitura.rio/rio-carnaval/. Record any official statements on Rio Carnival, its cultural or economic importance, and related initiatives or publications. Capture the full page URL and the concrete descriptive points provided. | ||||
| 0% | 0% | 0% | 0% | |
How to passNavigate to https://www.carnavalderua.rio/. Record available details on street blocos, pre-Carnival activities, and any general programme structure or categories of events. Note the exact page URL and the listed event types or locations mentioned. | ||||
| 0% | 0% | 100% | 0% | |
How to passNavigate to https://prefeitura.rio/riotur/prefeitura-divulga-o-balanco-do-2o-dia-de-desfiles-na-sapucai-e-a-programacao-deste-domingo/. Record the characterisation of Carnival in Rio de Janeiro, references to its scale or recognition, and notes on street versus formal celebrations. Capture the page URL and the main overview passages. | ||||
| 0% | 0% | 0% | 0% | |
How to passNavigate to https://www.gov.br/turismo/pt-br/assuntos/noticias/carnaval-do-rio-de-janeiro-e-axe-music-recebem-reconhecimentos-federais-pela-importancia-cultural-no-brasil. Record any statements on the cultural status of Rio Carnival or related Brazilian Carnival traditions and their national significance. Note the exact page URL and the recognition details given. | ||||
| 100% | 100% | 100% | 100% | |
How to passUsing only the data recorded in the prior steps, create a single written document organised under headings for cultural significance, relative timing and duration, main celebration locations, parade and street elements, and official resources. Place the exact source URL beside every factual claim and keep all timing references relative to the Easter cycle, leaving the document ready for review. | ||||
Failure modes on this task
15 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Standing up the evidence1 mode | 1 | 1 | — | 3 |
| Reaching the page1 mode | 1 | — | — | — |
| No signal in the judge’s words1 mode · the grader | 2 | 3 | 3 | 1 |