Research publicly available information on heavy industryHeavy Industry and Engineering
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
88%
Claude Opus 5.5
63%
GPT-5.6 sol
50%
GPT-6 sol
0%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
59m
Claude Opus 5
29m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
108
Claude Opus 5
36
Claude Opus 5.5
34
GPT-6 sol
34
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (8) | ||||
| 100% | 100% | 100% | 0% | |
How to passNavigate to https://ncc.abcb.gov.au/. Record the stated purpose of the National Construction Code, the responsible body that produces and maintains it, the performance-based nature of the code, and the available editions or volumes listed. Note the exact page URL and the key descriptive statements. | ||||
| 0% | 0% | 0% | 0% | |
How to passNavigate to https://www.abcb.gov.au/editions/ncc-2025/adopted/volume-one/b-structure/part-b1-structural-provisions or the equivalent current structural provisions page linked from the NCC site. Record the main structural design actions referenced, the Australian Standards listed for timber, concrete, masonry and steel, and any performance requirements described. Capture the full page URL and the concrete technical references. | ||||
| 100% | 0% | 100% | 0% | |
How to passNavigate to the current Safe Work Australia construction-work Code of Practice and related construction resources. Record the current construction-work guidance, high-risk construction requirements, and relevant safety controls. For each material requirement, state whether it is a model-code recommendation or an approved code with legal effect, and identify the relevant state or territory regulator where jurisdictional approval must be checked. | ||||
| 100% | 0% | 100% | 0% | |
How to passReview the official Construction resources page and linked current resources covering construction work, demolition, excavation, falls, electrical risks, asbestos, hazardous manual tasks and other construction hazards. Record the resource title, purpose, latest update date and exact source URL. | ||||
| 100% | 0% | 0% | 0% | |
How to passNavigate to https://austroads.gov.au/infrastructure/technical-specifications. Record the purpose of the Austroads Technical Specifications, the subject areas covered across the series, and examples of individual specifications with their latest update indicators if shown. Note the exact page URL and the organisational statements. | ||||
| 100% | 100% | 0% | 0% | |
How to passNavigate to https://www.abs.gov.au/statistics/industry/building-and-construction/building-activity-australia/latest-release. Record the latest available summary figures for total building work done, residential versus non-residential components, and any notes on the scope of the data collection. Capture the full page URL and the key statistical claims presented. | ||||
| 100% | 100% | 100% | 0% | |
How to passUse current Australian Government and ABS sources to summarize relevant engineering/construction-sector context. Prioritize ABS Engineering Construction Activity for engineering construction statistics and use relevant Australian Government industrial-sector material only where it directly informs heavy industry, materials, infrastructure or engineering practice. Record the scope and limitations of each source. | ||||
| 100% | 100% | 100% | 0% | |
How to passUsing only the data recorded in the prior steps, create a single written document organised under headings for National Construction Framework, Structural and Technical Standards, Work Health and Safety in Construction, Infrastructure Specifications, Building Activity Statistics, and Industry Sector Context. Place the exact source URL beside every factual claim and ensure all references remain relative to the latest published editions or data releases, leaving the document ready for review. | ||||
Failure modes on this task
16 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Standing up the evidence1 mode | 1 | 1 | 2 | 5 |
| Delivering the result1 mode | — | 1 | — | — |
| No signal in the judge’s words1 mode · the grader | — | 2 | 1 | 3 |