- Every contract arrives with clauses extracted and mapped to the playbook before a lawyer opens it
- Suggestions cite both the contract text and the playbook position they rely on
- Lawyers accept, edit or reject each suggestion, and those decisions feed evaluation
- Playbook positions maintained as structured data rather than a document nobody updates
Anonymised composite. Client names, identifying details and confidential figures are withheld; public facts are cited in the references.
The situation
A legal team handled a steady flow of commercial agreements: supplier contracts, NDAs, services agreements, licences. Each one was checked against an internal playbook that set out preferred positions, acceptable fallbacks and red lines for clauses such as liability caps, indemnities, termination, governing law and data protection. The work was skilled but repetitive, and it bunched up at quarter end when everything was urgent.
The team had tried generic AI chat tools. The answers sounded plausible, cited nothing, and didn't know the playbook. Nobody trusted them, rightly.
What had to be true
- The lawyer decides. The system prepares and proposes; it never sends a redline on its own.
- Every suggestion is cited to the exact contract text and the playbook position it relies on.
- Consistent structure, so outputs can be validated and compared, not just read.
- Measured accuracy per clause type, on the team's own contracts, before anyone relies on it.
- Confidentiality: contract text handled under the firm's existing data controls.
Contract review is a well-studied task: the CUAD dataset, built with legal experts, frames it as finding the clauses a reviewer needs to see across dozens of categories1. That framing, extraction first and judgement second, shaped the design.
Architecture
Extraction used structured outputs, where the model's response is constrained to a supplied JSON Schema2. Each clause came back with its type, section reference, verbatim quote and confidence. A deterministic check then confirmed that every quote actually appeared in the source document. See structured outputs for reliable LLM integration.
The data flow, end to end
- UploadContract parsed with its numbering and structure preserved
- ExtractClauses returned as schema-valid records with verbatim quotes
- VerifyQuotes checked against the source; mismatches flagged, not hidden
- RetrieveRelevant playbook positions and fallbacks for each clause type
- CompareClause against playbook; propose redline citing both texts
- ReviewLawyer accepts, edits or rejects; decision logged for evaluation
The hard part: long context versus retrieval
A single contract usually fits within a modern model's context window, so the temptation is to put everything in: the contract, the whole playbook, and a pile of precedents. Research on long contexts shows models use information at the start and end of the input more reliably than information in the middle3, which is a poor property when the clause that matters is on page fourteen.
The design split the jobs:
- The contract is read whole for extraction, because cross-references and definitions matter.
- The playbook is retrieved per clause, in the spirit of retrieval-augmented generation4, so each comparison sees only the relevant positions.
- Comparison runs clause by clause, with small, focused prompts that are easier to evaluate.
The trade-offs are covered in long context vs RAG.
Contracts from counterparties are also untrusted input. Prompt injection tops the OWASP Top 10 for LLM applications5, so contract text was always treated as data, the model had no tools that could act on it, and outputs were constrained by schema. See prompt injection defence.
A suggestion a lawyer can verify in ten seconds is worth more than a fluent paragraph they have to check from scratch.
How the engagement ran
- Turn the playbook into positions
- Label a test set
- Review and decide
- Extract and verify
- Compare and cite
- Log every decision
- Run evaluations per clause type
- Gate releases on results
- Tune prompts and schema
- Clause taxonomy
- Positions and fallbacks
- Red lines
- Schema design
- Quote verification
- Labelled test set
- Per-clause retrieval
- Cited suggestions
- Review UI
- Evaluation gate
- Decision feedback
- More contract types
Key decisions and trade-offs
| Decision | Alternative | Why |
|---|---|---|
| Structured extraction first | Free-text review in one prompt | Validatable, comparable, displayable output |
| Verbatim quotes checked in code | Trust the model's quotes | Catches fabricated or paraphrased citations |
| Playbook as structured positions | Playbook document in the prompt | Precise retrieval; easier to maintain |
| Evaluation per clause type | One overall accuracy figure | Strong clause types ship; weak ones wait |
| Lawyer decides every suggestion | Auto-apply high-confidence redlines | Accountability stays with the profession |
Building evaluation sets that hold up in production is covered in evaluation sets that predict production.
What I'd do again, and differently
Again: turn the playbook into data before writing a prompt, and treat lawyer decisions as the evaluation signal. Differently: involve lawyers in schema design from the start. The clause taxonomy is a legal decision, and the earlier they own it, the fewer rework cycles follow.
If you're considering AI for review work like this, the AI architect page describes how I run these engagements, or analyse your challenge and bring the result to a call.
- 1CUAD: An Expert-Annotated NLP Dataset for Legal Contract ReviewHendrycks et al. (arXiv; NeurIPS 2021 Datasets and Benchmarks) · 2021
- 2Structured outputsAnthropic documentation
- 3Lost in the Middle: How Language Models Use Long ContextsLiu et al. (arXiv; TACL) · 2023
- 4Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksLewis et al. (arXiv; NeurIPS 2020) · 2020
- 5OWASP Top 10 for LLM Applications 2025OWASP GenAI Security Project