I am studying for a public contestScience and Education
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5.5
100%
GPT-6 sol
100%
GPT-5.6 sol
75%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
56m
Claude Opus 5
17m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
118
Claude Opus 5
64
Claude Opus 5.5
56
GPT-6 sol
20
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (8) | ||||
| 100% | 100% | 100% | 100% | |
How to passSearch official Caixa Econômica Federal and official organizing-body sources to determine whether a current public-selection edital for the requested Caixa Econômica Federal contest is available. If no current edital is available, identify the most recent completed Caixa contest edital that is relevant to the intended position. Record the contest year, position, organizing institution, publication date, and official edital source. Do not assume that a rumored or upcoming contest has an edital; clearly distinguish an officially published edital from news, expectations, or preliminary announcements. | ||||
| 100% | 100% | 100% | 0% | |
How to passOpen the official edital identified in Step 1 and verify that it is the correct and most recent available document for the selected Caixa contest/position. Extract the examination structure, number of questions, subjects, scoring rules, minimum requirements, elimination criteria, and any other information that directly affects preparation. If multiple versions, amendments, or corrigenda exist, check the official sources and use the latest applicable version, recording which version was used and noting any material amendments. Do not mix requirements from different Caixa contests or positions without clearly identifying the source. | ||||
| 100% | 100% | 100% | 100% | |
How to passResearch the institution responsible for administering the selected Caixa contest using official sources and reputable examination-related sources where appropriate. Identify the organizer, its role in the selection process, and the characteristics of the examination format used for the relevant Caixa contest. Based on the verified edital and available previous-test information, describe the typical question style, subject emphasis, wording patterns, difficulty characteristics, and any relevant scoring or elimination rules. Clearly distinguish facts stated in the edital from observations based on previous questions or independent analysis. Use this information to provide practical preparation suggestions specific to the examination rather than generic study advice. | ||||
| 100% | 100% | 100% | 100% | |
How to passCreate a document named “Study - Caixa” and use it as the central deliverable for the research. Structure the document so that the information can be updated if a newer official edital becomes available. Include the verified contest identification, official edital source, organizer information, examination structure, subject list, test characteristics, and study recommendations. Preserve source links and identify the date or version of the material used. | ||||
| 100% | 100% | 100% | 100% | |
How to passUsing the verified information from Steps 2 and 3, add a concise but useful debrief to “Study - Caixa” explaining what candidates should expect from the Caixa examination. Cover the usual question format, subject distribution or emphasis where officially documented, common question characteristics supported by available previous tests, scoring and elimination rules, and practical preparation tactics. Explain which subjects should receive greater attention based on the actual edital rather than assuming that every Caixa position has the same syllabus. Clearly label recommendations and analytical observations separately from verified examination requirements. | ||||
| 100% | 100% | 100% | 100% | |
How to passParse the latest applicable edital and add a subject-by-subject study syllabus to “Study - Caixa”. For every subject explicitly listed in the editalsuch as Portuguese, Mathematics, Banking Knowledge, Law, Information Technology, Economics, or other subjects applicable to the selected position—summarize the actual topics and subtopics that candidates are expected to study. Preserve the edital's terminology and do not add subjects or topics merely because they appeared in another Caixa examination. Include the official edital source and identify the specific edital/version used for the summary. | ||||
| 100% | 100% | 100% | 100% | |
How to passResearch at least three online courses specifically intended to prepare candidates for the selected Caixa contest or the same Caixa position/examination profile. Prefer established educational providers and verify the course details on their current official sales or course pages. For each course, record the provider, course name, applicable contest/position, included subjects or material, format, access period where stated, and current lowest publicly listed price that can be verified without purchasing or enrolling. Compare the courses on syllabus coverage and practical usefulness rather than price alone. Do not purchase, enroll, or enter payment information. If a course price requires login or cannot be verified publicly, mark the price as unavailable rather than estimating it. | ||||
| 100% | 100% | 100% | 0% | |
How to passResearch the most recent completed Caixa contest for the selected position and determine the latest reliably documented candidate-to-vacancy ratio. Use official Caixa, organizer, or other reputable sources and explain exactly how the ratio was obtained, including the number of registered candidates and vacancies when available. Add the calculation and source to the end of “Study - Caixa.” Then review the entire document to ensure that the edital summary, organizer/test debrief, subject-by-subject syllabus, three course options and prices, and candidate-to-vacancy ratio all refer to the same contest/position unless a different historical contest is explicitly identified. Clearly distinguish verified facts from recommendations, assumptions, and unavailable information. Do not submit applications, purchase courses, or enroll in any preparation program. | ||||
Failure modes on this task
2 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Standing up the evidence1 mode | — | — | — | 2 |