I need to procure a fleet of 5Computers Electronics and Technology
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
96%
Claude Opus 5.5
92%
GPT-5.6 sol
64%
GPT-6 sol
52%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
37m
Claude Opus 5
26m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
119
Claude Opus 5
97
Claude Opus 5.5
63
GPT-6 sol
63
GPT-5.6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (25) | ||||
| 100% | 100% | 100% | 100% | |
How to passResearch official processor and RAM specifications. Define minimum hardware criteria in working_dossier.md. | ||||
| 100% | 100% | 100% | 100% | |
How to passInspect official refurbishing process documentation for Lenovo Outlet, Dell Refurbished, and Amazon Renewed. | ||||
| 100% | 100% | 100% | 100% | |
How to passReview corporate/refurbished warranty terms for each candidate and verify whether at least 1 year of hardware warranty coverage is provided. Clearly flag any candidate that has less coverage or where the warranty terms are not publicly disclosed. | ||||
| 100% | 100% | 100% | 0% | |
How to passReview each vendor's publicly available quantity, business, or volume-discount policies for an order of 5 laptops. Record any published discount or threshold, and mark discounts that require a sales quote or account as not publicly disclosed. | ||||
| 100% | 0% | 100% | 100% | |
How to passCreate a scoring framework based on processor, 32GB RAM, 1TB SSD, refurbished/new status, warranty, battery condition or guarantee, return policy, price, availability, and suitability for a 5-unit fleet. | ||||
| 100% | 100% | 100% | 100% | |
How to passSearch Lenovo Outlet for certified refurbished ThinkPad models meeting hardware specs. Record findings in scratchpad. | ||||
| 100% | 0% | 0% | 100% | |
How to passVerify that each Lenovo candidate meets the mandatory processor class, 32GB RAM, and 1TB SSD requirements. Record the exact processor model rather than imposing an additional processor-generation requirement. | ||||
| 100% | 100% | 100% | 100% | |
How to passSearch Dell Refurbished store for Latitude or Precision laptops meeting baseline specs. Record items in scratchpad. | ||||
| 100% | 0% | 100% | 100% | |
How to passConfirm that each Dell candidate meets the mandatory processor class, 32GB RAM, and 1TB SSD requirements, and record screen resolution as an additional comparison detail rather than a pass/fail requirement. | ||||
| 100% | 100% | 100% | 100% | |
How to passSearch a reputable US vendor's current new laptop inventory for models meeting the mandatory processor, 32GB RAM, and 1TB SSD requirements, keeping the new-inventory candidates separate from refurbished listings. | ||||
| 100% | 0% | 100% | 100% | |
How to passVerify that each new-model candidate meets the mandatory processor class, 32GB RAM, and 1TB SSD requirements. Record benchmark information and port selection only as comparison details where publicly available. | ||||
| 100% | 100% | 100% | 0% | |
How to passRead each vendor's published refurbished grading and battery-health policies. Record any stated battery-capacity guarantee or condition requirement, and clearly flag vendors that do not publish a specific battery-health threshold. | ||||
| 100% | 100% | 100% | 0% | |
How to passCheck publicly stated shipping and delivery estimates for the shortlisted configurations and determine whether they appear compatible with the 30-day procurement window. | ||||
| 100% | 100% | 100% | 100% | |
How to passCheck publicly available business, volume, or promotional discounts that could apply to a 5-unit purchase. Do not require account-only or unpublished coupon codes; record those as unavailable for public verification. | ||||
| 100% | 0% | 100% | 0% | |
How to passCalculate the per-unit and five-unit merchandise cost for each qualifying configuration using published prices and discounts. Add publicly determinable shipping and estimated sales tax using the stated representative US destination, and clearly identify any amount that cannot be determined without checkout. | ||||
| 100% | 100% | 100% | 0% | |
How to passEvaluate each 5-unit configuration against the $6,500 total budget using the best publicly supported estimate for merchandise, discounts, shipping, and tax. Clearly distinguish confirmed totals from estimates and flag configurations that cannot be conclusively verified as under budget. | ||||
| 0% | 0% | 0% | 0% | |
How to passCompare reputable independent reviews or reliability evidence for shortlisted models, focusing on thermal performance, chassis durability, and long-term business use. | ||||
| 100% | 100% | 100% | 100% | |
How to passTest the vendor-search fallback rule by documenting when a portal search is unsuccessful twice and switching to an alternative vendor or source without repeatedly retrying the same failed search. | ||||
| 100% | 100% | 100% | 0% | |
How to passCompare the published return windows, restocking fees, exclusions, and return-shipping terms for shortlisted vendors, without assuming a specific minimum return period. | ||||
| 100% | 100% | 100% | 100% | |
How to passIf publicly available, summarize longer-term ownership costs such as warranty extensions, battery replacement coverage, or other recurring costs as a secondary consideration; do not make a three-year TCO a mandatory ranking criterion. | ||||
| 100% | 0% | 100% | 0% | |
How to passRank top 3 fleet options based on performance scores, price savings, and warranty coverage. | ||||
| 100% | 100% | 100% | 0% | |
How to passRe-verify all mandatory constraints: 5 units, Intel i7/Ryzen 7 class processor, 32GB RAM, 1TB SSD, and a total estimated cost strictly below $6,500, while separately flagging any tax/shipping amount that cannot be publicly determined. | ||||
| 100% | 0% | 100% | 0% | |
How to passCreate a detailed price table containing unit prices, five-unit subtotals, discounts, estimated tax, shipping, refurbished/new status, warranty, and direct listing links. Clearly label estimates and unavailable values. | ||||
| 100% | 0% | 100% | 0% | |
How to passWrite a concise procurement summary comparing the strongest fleet configurations, their estimated total costs, key warranty/return considerations, and the main reasons for the recommended configuration. | ||||
| 100% | 100% | 100% | 0% | |
How to passWrite and save the comprehensive purchasing dossier in working dossier.md, including the candidate comparison, verified specifications, pricing calculations, tax and shipping assumptions, warranty and return findings, budget gate, recommended configurations, and direct listing links. | ||||
Failure modes on this task
24 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Standing up the evidence2 modes | — | 2 | 1 | 4 |
| Delivering the result1 mode | — | 1 | — | 1 |
| Reaching the page1 mode | — | 1 | — | — |
| No signal in the judge’s words1 mode · the grader | 1 | 5 | 1 | 7 |