Build a reusable Contract Obligation Clause-Comparison & …Law and Government
How each model performed on this task. Every chart states its N, and the reward carries a 95% confidence interval once there is more than one run to measure.
Core metrics
Check Pass %
Mean share of the task's checks passing, averaged over runs. N = 1 run per model.
100%
Claude Opus 5.5
100%
GPT-6 sol
100%
GPT-5.6 sol
0%
Claude Opus 5
Extended metrics
Median Time
Wall-clock minutes, median across runs. N = 1 run per model.
83m
Claude Opus 5
22m
GPT-5.6 sol
Tool Calls
Tool invocations, median across runs — counted from the trajectories. N = 1 run per model.
112
Claude Opus 5
60
Claude Opus 5.5
24
GPT-5.6 sol
23
GPT-6 sol
Verifier × model pass rate
How often each verifier passed, per model. Green is passing, red is failing — a red band across a row is a verifier nothing clears.
| Verifier | Claude Opus 5.5n=1 | GPT-6 soln=1 | GPT-5.6 soln=1 | Claude Opus 5n=1 |
|---|---|---|---|---|
| checks (15) | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a structured intake method for determining what exactly needs to be reviewed before examining any agreement. Separate objectives such as understanding obligations, comparing two contract versions, preparing for renewal, checking termination requirements, identifying operational responsibilities, locating financially significant provisions, or deciding which clauses deserve professional review. Record the agreement type, parties, document version, amendment status, effective period where known, related documents, review purpose, and information that is intentionally unavailable. Research authoritative guidance on contract interpretation, document review, and identifying the operative version of an agreement. Clearly establish that the system is a research and organization tool rather than a substitute for qualified legal advice. Add a field identifying the decision that the review is supposed to support. Distinguish between reviewing the contract itself and researching external rules that may affect it. Include a boundary rule for situations where the available document is incomplete, unsigned, poorly scanned, or missing referenced schedules. The intake should prevent the researcher from beginning clause analysis before understanding what document and decision are actually being investigated. | ||||
| 100% | 100% | 100% | 0% | |
How to passDevelop a source hierarchy showing which sources should be preferred for different types of claims encountered during contract research. Include the actual agreement and its amendments, official government guidance, legislation or regulatory materials where relevant, court or tribunal materials, recognized legal-information institutions, universities and law schools, professional legal publications, established legal reference books, reputable explanatory publications, and weaker secondary sources such as forums or informal commentary. Research how authoritative legal-information sources explain the difference between contractual language, external legal rules, commentary, and practical interpretation. Create fields for source title, issuing organization, publication or update information, claim supported, accessibility, and limitations. Explain that an authoritative source for explaining a legal concept may not prove what a particular contract means. Include fictional cases where a popular online explanation conflicts with the wording of the agreement or a primary source. The system should require the researcher to identify which source actually supports each important conclusion. Also create an escalation rule for claims that are too jurisdiction-specific, legally consequential, or ambiguous to resolve confidently through general online research. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a system for establishing exactly which contractual documents belong together and which version controls the review. Record agreement title, parties, document date, effective date where available, signature status, amendment number, schedules, appendices, exhibits, referenced policies, attachments, and later modifications. Research reliable document-management and contract-administration practices from authoritative sources. Explicitly distinguish the original agreement from amendments, renewals, addenda, side letters, incorporated policies, and replacement agreements. Include a method for recording whether a document is complete and whether referenced material has actually been obtained. Create fictional situations where an older agreement and a later amendment both appear authoritative, where an attachment is referenced but unavailable, and where two versions contain identical titles but different operative provisions. Add a confidence field for document identity. Do not assume that the newest file automatically replaces every previous provision. The register should allow a reviewer to reconstruct the contractual document set and understand which evidence supports the conclusion about the operative version. If a required document is missing, the system should preserve that gap rather than silently assuming what it contained. | ||||
| 100% | 100% | 100% | 0% | |
How to passDevelop a reusable taxonomy for classifying contract language. Separate obligations, prohibitions, permissions, conditions, representations, warranties, definitions, payment provisions, confidentiality requirements, intellectual-property provisions, termination rights, renewal mechanisms, dispute procedures, notices, compliance requirements, service levels, liability provisions, and informational language. Research authoritative legal and contract-management resources explaining how contractual provisions function, while avoiding unsupported claims that a particular wording always has one universal legal effect. Create fields for clause location, exact subject, responsible party, trigger, required action, deadline, consequence, exception, dependency, and evidence source. Distinguish an obligation from a statement describing an existing situation. Include fictional examples where language appears mandatory but is qualified by an exception elsewhere, and another where a seemingly minor definition changes the scope of a major obligation. The framework should help a stranger translate complex contractual language into structured review information without pretending that the structured interpretation is itself a definitive legal conclusion. Add a “requires professional interpretation” flag for clauses whose meaning cannot safely be resolved from the available evidence. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a detailed tracker for obligations that arise from the agreement. Each entry should record the responsible party, action required, trigger event, deadline or timing rule, frequency, dependency, exception, evidence location, consequence of non-performance, and confidence. Research contract-administration guidance on managing recurring and event-triggered obligations. Distinguish fixed deadlines from obligations triggered by events such as notice, delivery, breach, approval, renewal, termination, or receipt of another document. Include fictional cases where an obligation becomes active only after another party takes an action, where two clauses appear to establish different deadlines, and where a recurring requirement is easy to overlook because it appears in an appendix. Do not invent missing dates or assume that a deadline exists when the document does not establish one. Add fields for “known,” “conditional,” “unclear,” and “requires confirmation.” The register should support both one-time and recurring obligations. It should also make dependencies visible so that a reviewer can understand why a seemingly distant clause may affect an important future action. The final tracker should allow obligations to be prioritized without converting uncertain legal interpretation into false certainty. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a structured process for comparing two or more versions of an agreement. The system should identify additions, deletions, wording changes, moved provisions, changed definitions, altered exceptions, changed notice mechanisms, and changes that appear minor but materially affect obligations. Research authoritative document-comparison and contract-management practices. Separate purely formatting or editorial changes from changes that may affect meaning. Create fields for original wording, revised wording, change type, affected party, practical effect, confidence, supporting evidence, and review priority. Include fictional examples where one word changes an obligation, where a definition is changed rather than the main clause, where an exception disappears, and where a provision moves to another section without obvious wording changes. Do not decide that a change is legally material merely because it looks substantial; explain the reasoning and identify when professional review is appropriate. The comparison system should also detect when two apparently different clauses actually impose the same requirement. The final methodology should produce a prioritized change log rather than an unstructured list of every textual difference. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a specialized framework for reviewing provisions governing renewal, expiration, termination, notice, cure periods, and related procedural requirements. Research authoritative sources explaining contract-administration concepts such as notice, automatic renewal, termination rights, and timing requirements, while recognizing that actual legal effects can depend on the agreement and applicable law. Record who can act, what event triggers the provision, how notice must be delivered, when it becomes effective, what deadlines apply, whether exceptions exist, and which other clauses interact with the provision. Include fictional cases involving automatic renewal, a notice period that begins after another event, different notice requirements for different termination grounds, and a contract where renewal language appears in one section but termination language creates a dependency elsewhere. Do not assume that a “renewal date” alone tells the user what action is required. The system should highlight timing-sensitive provisions and identify information that must be confirmed before acting. It should also separate “the contract appears to require this” from “external law may affect this,” ensuring that the final result does not overstate what can be concluded from the contract alone. | ||||
| 100% | 100% | 100% | 0% | |
How to passDevelop a method for finding clauses that interact with one another. Identify references to definitions, schedules, appendices, policies, other sections, amendments, exceptions, and incorporated documents. Research contract-drafting and document-review guidance on internal consistency and cross-references. Create fields for the originating clause, referenced provision, relationship, apparent dependency, conflict type, evidence, and resolution status. Include fictional cases where a definition narrows a term used elsewhere, where one clause establishes a general rule and another creates an exception, where an amendment changes one provision but leaves an inconsistent reference in another, and where two sections appear to establish different obligations for the same event. Do not automatically choose the clause appearing later in the document as controlling. Instead, record the conflict and research what authoritative sources can and cannot establish about resolving it. The system should distinguish a true contradiction from a difference in scope or timing. It should also flag unresolved conflicts for human or professional review. This framework should become one of the strongest safeguards against reading each clause in isolation and missing how the agreement operates as a connected document. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate an uncertainty framework for situations where contractual wording is incomplete, unclear, internally inconsistent, poorly scanned, or dependent on missing external information. Research authoritative materials on ambiguity, contract interpretation, document quality, and evidence limitations. Record the exact uncertainty, possible interpretations, evidence supporting each interpretation, information that could resolve it, and practical consequence of choosing incorrectly. Include fictional cases where a key term is undefined, where a date is missing, where an attachment referenced by the agreement is unavailable, and where a scanned clause contains an unreadable word. Do not silently fill gaps using common practice or general knowledge. Explicitly distinguish “not found,” “not stated,” “unclear,” and “contradicted.” Create a confidence scale that reflects evidence quality rather than subjective certainty. The system should also identify when researching more online material is unlikely to resolve the ambiguity because the missing evidence is inside a private document or depends on facts unavailable to the reviewer. This prevents the research process from producing a confident-looking answer when the underlying contract simply does not provide enough information. | ||||
| 100% | 100% | 100% | 0% | |
How to passDevelop a transparent method for deciding which clauses deserve attention first. Consider factors such as deadline sensitivity, financial consequence, operational impact, termination exposure, dependency, uncertainty, scope, recurrence, evidence quality, and difficulty of correcting a mistake after the fact. Research reputable contract-management and risk-prioritization methodologies and explain which elements are evidence-based versus judgment-based. Avoid creating arbitrary numerical scores without defining what each score means. Build a priority classification such as immediate review, high priority, routine review, monitor, or informational. Include fictional cases where a short clause creates a major deadline risk, where a lengthy clause has little practical impact, and where a seemingly important provision is already superseded by an amendment. Require the reviewer to explain why each high-priority item ranks above lower-priority items. Include a “what could change this priority?” field. The system should also account for uncertainty: an unclear provision with potentially serious consequences may deserve earlier escalation even when its ultimate interpretation is unknown. The final priority model should support decisions rather than simply generating a visually impressive list of clauses. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate a traceability system connecting every important review conclusion to its underlying evidence. For each conclusion, record the exact clause or source, the external research used, the interpretation made, competing interpretations considered, confidence level, and unresolved issue. Research authoritative approaches to legal research, source citation, document review, and auditability. Distinguish direct textual evidence from external explanation and analyst inference. Include fictional cases where two independent sources support a conclusion, where an online explanation repeats another source without adding independent evidence, and where the contract wording itself is stronger evidence than a generic explanation. Add a field identifying whether the conclusion is reproducible by another reviewer. The audit trail should make it possible to work backward from a recommendation to the exact contractual evidence supporting it. It should also preserve rejected interpretations when they were reasonably plausible, explaining why they were not selected. This prevents the final ranked output from becoming a black box. A stranger should be able to inspect the evidence trail, understand how the priority was assigned, and identify exactly what information would need to change before the conclusion should be revised. | ||||
| 100% | 100% | 100% | 0% | |
How to passApply the completed framework to at least eight clearly fictional stress-test cases, using invented agreement facts only to test the system rather than presenting them as real legal situations. Include: an automatic-renewal clause with an easy-to-miss notice requirement; two conflicting deadlines; an amendment that changes a definition affecting another obligation; a missing referenced schedule; an obligation triggered by another party's action; an inconsistent termination provision; an ambiguous or incomplete clause; and a version-control problem involving an apparently superseded agreement. For each case, record the document identity, relevant clauses, source hierarchy, obligations, triggers, deadlines, dependencies, conflicts, uncertainty, priority ranking, evidence trail, and escalation decision. Deliberately introduce misleading but plausible information into several cases. Include at least one case where the correct outcome is “insufficient information,” rather than a forced interpretation. Change one important fact in several cases and verify that the priority ranking changes appropriately. The purpose is to determine whether the system responds to evidence and contractual structure rather than simply assigning the same high priority to familiar words such as “termination,” “payment,” or “renewal.” | ||||
| 100% | 100% | 100% | 0% | |
How to passAttempt to break the system using realistic weaknesses in document review. Test incomplete agreements, outdated amendments, copied clause summaries, misleading section headings, inconsistent definitions, hidden exceptions, cross-reference errors, missing schedules, duplicated obligations, contradictory deadlines, and clauses whose apparent importance changes after another provision is considered. Also test whether the system incorrectly treats generic legal commentary as proof of what a specific agreement means. Identify at least five failure modes and show which part of the methodology catches each one. Test the opposite failure as well: excessive caution. If the agreement clearly states a straightforward requirement and authoritative evidence supports the interpretation, the system should not manufacture uncertainty merely because every contract can theoretically be disputed. Check whether the system can distinguish a genuine contractual conflict from two provisions addressing different circumstances. Also test whether a reviewer could accidentally prioritize a dramatic-looking clause while overlooking a simple but time-sensitive obligation. Document every detected weakness and modify the methodology where necessary. The final system should be rigorous without becoming impractical, and cautious without becoming incapable of reaching useful conclusions. | ||||
| 100% | 100% | 100% | 0% | |
How to passCreate the final method for ranking contract-review priorities based on the completed research. The ranking should consider consequence, urgency, uncertainty, dependency, recurrence, evidence quality, correction difficulty, and relevance to the reviewer's actual objective. Produce separate ranking logic where appropriate for different situations, such as preparing to sign, preparing for renewal, comparing amendments, preparing for termination, or monitoring an active agreement. Do not assume that one universal ranking works for every contract. Each ranked item should include: what requires attention, why it ranks at that position, the supporting contractual evidence, confidence, unresolved questions, and what new information could change the ranking. Include an escalation category for items where professional legal review may be appropriate. Test the methodology against several of the fictional cases from Step 12 and verify that materially different facts produce materially different rankings. Keep the ranking traceable to evidence rather than subjective impressions. The final system should allow someone to answer not merely “What does this contract contain?” but “What should I examine first, why does it matter, what evidence supports that priority, and what information would change my decision?” | ||||
| 100% | 100% | 100% | 0% | |
How to passAssemble the completed system into a practical reusable tracker that contains the intake register, document/version register, clause taxonomy, obligation tracker, trigger and deadline register, clause-comparison log, renewal and termination tracker, cross-reference conflict log, uncertainty register, evidence-to-conclusion audit trail, risk-priority model, eight fictional stress tests, adversarial audit results, and final ranking methodology. Include reusable fields and instructions so another person can apply the tracker to a new agreement without redesigning it. Clearly distinguish contractual text, authoritative external information, interpretation, inference, uncertainty, and professional-review escalation. Every high-priority item should contain a concise rationale, supporting evidence, confidence, and “what could change this?” field. Include instructions for preserving the research trail and updating the tracker when an amendment or new document appears. Finally perform the stranger test: a person unfamiliar with this project should be able to identify the operative documents, extract obligations, trace triggers and deadlines, compare versions, detect conflicts, handle uncertainty, verify supporting sources, prioritize review items, and understand when further professional review may be appropriate without asking follow-up questions. | ||||
Failure modes on this task
15 tagged verifier failures, by the stage they were assigned to. Higher is worse.
| Failure stage | Claude Opus 5.5 | GPT-6 sol | GPT-5.6 sol | Claude Opus 5 |
|---|---|---|---|---|
| Delivering the result1 mode | — | — | — | 10 |
| Standing up the evidence1 mode | — | — | — | 2 |
| Reaching the page1 mode | — | — | — | 1 |
| No signal in the judge’s words1 mode · the grader | — | — | — | 2 |