I am starting a master’s in Advanced SoftwareComputers Electronics and Technology
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
GPT-6 sol
100%
GPT-5.6 sol
50%
Claude Opus 5.5
0%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
53m
Claude Opus 5
27m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
113
Claude Opus 5
100
Claude Opus 5.5
48
GPT-6 sol
23
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (8) | ||||
| 100% | 100% | 100% | 0% | |
How to passDefine the three workloads, two editable budgets and one comparison schema before collecting prices. Cover model identifier and version, context/output limits, input, output and cached-token rates, cache writes or storage, batch pricing, long-context tiers, tool charges, rate limits, structured output, usage reporting and source date. Separate published facts from workload assumptions and derived calculations; use “not stated” where a required field is absent. | ||||
| 100% | 100% | 100% | 0% | |
How to passUse the current official OpenAI developer pricing/model pages, Anthropic pricing/model documentation and Gemini API pricing/model pages. Choose exactly one cost-optimised and one higher-capability text model from each provider that have public API prices. Record exact model IDs, eligibility or preview status, units, context limits, supported capabilities and exact URLs. Do not use third-party comparison sites to fill first-party gaps, and explain any model excluded after initial consideration. | ||||
| 100% | 100% | 100% | 0% | |
How to passReconcile each provider’s billing rules for uncached input, cached reads, cache writes or storage, output/reasoning tokens, batch requests, long-context thresholds and separately charged tools. Preserve provider currency and add a dated, editable conversion to GBP from a public source; if a reliable rate cannot be accessed, keep GBP as “not stated” rather than inventing one. Show the normalised per-million-token table and retain exceptions as separate columns instead of forcing false equivalence. | ||||
| 0% | 100% | 100% | 0% | |
How to passCreate `ai_api_cost_calculator.csv` with editable model rates, exchange rate, budgets, token volumes, cache hit rate, retry rate, batch share, requests and sessions. Include formulas for one request/session, each monthly workload, total monthly spend, cost per successful result, evaluation overhead and budget variance. Treat unsupported caching or batch modes as zero use, not a discount. Add one manually worked calculation per provider to verify the spreadsheet formulas and flag taxes or tool fees whose applicability is not stated. | ||||
| 0% | 100% | 100% | 0% | |
How to passCreate exactly 18 synthetic evaluation cases, six per workload, without calling an API. Extraction cases must have expected fields and deterministic schema checks; RAG cases must test answer correctness, grounding and refusal when evidence is missing; coding cases must include requirements, tests or expected behaviours and tool-use constraints. Define pass/fail checks, a short human rubric and allowed variance. Model-graded scoring may be optional only if it is calibrated against human ratings on at least six cases and its extra cost is counted. | ||||
| 0% | 100% | 100% | 0% | |
How to passCalculate the estimated cost of evaluating all six models on all 18 cases using each model’s applicable rates. For calculator scenarios, assume three repetitions and a 10% rerun allowance. Then calculate the three monthly workloads under baseline, caching/batch where eligible, and illustrative stress scenarios with doubled output and doubled retries. Compare cost and expected evidence needs without claiming unrun quality or latency results. Recommend a workload-specific shortlist and state what measured result would justify moving from a cheaper to a higher-capability model. | ||||
| 100% | 100% | 100% | 0% | |
How to passUse current public provider documentation for spend limits or alerts, usage/cost reporting and rate limits, including OpenAI’s spend-limit and Costs/Usage API guidance, Anthropic’s Usage and Cost API documentation, and Google Cloud billing-budget guidance where applicable. Design 50%, 75%, 90% and 100% budget responses, per-request token caps, retry ceilings, timeouts, cache/batch rules, fallback conditions and required log fields. Distinguish an alert from a hard stop and mark controls requiring account access as implementation steps, not completed actions. | ||||
| 0% | 100% | 100% | 0% | |
How to passCreate `ai_api_cost_evaluation_system.md` with methodology, source log, six-model evidence table, pricing reconciliation, formulas, workload and stress results, evaluation design, routing policy, guardrails, refresh procedure and unresolved questions. Deliver the two CSV files with matching model/workload names and include a blank row/template for adding future models. Every material claim must have an exact link and access date; label facts, assumptions and calculations separately, report contradictions, and state clearly that no paid evaluation was run. | ||||
Failure modes on this task
12 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | 3 | — | — | 4 |
| Standing up the evidence1 mode | 1 | — | — | 2 |
| No signal in the judge’s words1 mode · the grader | — | — | — | 2 |