- The fastest LLM tested completed contract reviews 276 times faster than LPOs and 77 times faster than junior lawyers, per Onit’s AI Center of Excellence study.
- LLM-based review can cut per-document costs by close to 99.97% compared to junior lawyers, though a hybrid workflow with attorney oversight will narrow that gap in practice.
- GPT4-1106 matched LPO accuracy on issue determination (F-score 0.87) but fell short on localisation, pointing to an AI-first-pass, human-narrowing architecture rather than full replacement.
Onit’s AI Center of Excellence put several LLMs against junior lawyers, senior lawyers and Legal Process Outsourcers on contract review accuracy, speed and cost, and on issue determination, the models held their own. The paper, titled “Better Call GPT: Comparing large language models against lawyers,” found GPT4-1106 matching LPO accuracy scores while completing reviews in under five minutes versus the human average of nearly an hour.
Where LLMs Stand on Accuracy
The study used senior lawyer judgments as ground truth, then measured how well each model identified legal issues in a contract dataset. Onit‘s GPT4-1106 reached an F-score of 0.87 on issue determination, placing it level with LPOs and slightly ahead of junior lawyers on that specific task.
Issue localisation is a different story. LPOs scored around 0.77 on pinpointing the exact contract text containing a problem; GPT4-32k came in at roughly 0.74. The gap is not large, but it is consistent: models flag issues reliably but locate them with less precision than experienced practitioners. For legal teams, that points to a workflow where AI handles the first pass and a human reviewer narrows to the flagged sections, rather than reading end-to-end. Most legal operations leaders, however, underestimate the preparation this architecture requires.
Speed: Minutes vs Hours
Palm2 Text-bison averaged 0.73 minutes per contract. GPT4-1106 averaged 4.7 minutes. Junior lawyers in the study averaged 56.17 minutes; LPOs averaged 201 minutes; even senior lawyers, the fastest human group, averaged 43.46 minutes. At the extremes, the fastest model was roughly 276 times quicker than LPOs and 77 times quicker than junior lawyers.
For legal departments managing high-volume commercial contracting, that throughput gap changes what is operationally possible. Backlogs that accumulate over weeks can clear in hours, and the time savings legal AI can deliver shift the constraint from review capacity to attorney judgment on the flagged items.
The Cost Case
The study calculated per-contract cost using commercial LLM pricing and average hourly rates. Those figures assume the model’s output needs no substantive human correction, a ceiling condition rather than a typical one. In practice, a hybrid model with attorney review of AI-flagged items will carry higher per-contract costs than 2 cents. The study does not quantify that blended rate, so legal operations teams building a business case should model their own review overhead against the baseline figures rather than applying the raw number directly.
Deploying LLMs for Contract Review
The study’s findings point to four operational decisions legal teams need to make before deploying LLMs for contract review.
Model selection matters for localisation. GPT-4, Claude 2.0 and other models tested showed varying precision on locating issues within contracts, even when their issue-determination scores were similar. Teams should pilot against their actual contract types before committing to a single model. Aggregate F-scores across a generic dataset will not predict performance on a specific category of supply chain or IP licensing agreements.
Prompt engineering is non-trivial. Each model in the study required different prompting to complete the review task reliably. Organisations that treat this as a one-time setup rather than an ongoing refinement process tend to see accuracy drift as contract language and legal standards evolve. Dedicated prompt governance is worth formalising early.
Human oversight stays in the architecture. The ground truth in the Onit study was set by senior lawyers, not by the models themselves. LLMs carry no professional liability for incorrect outputs, and the study’s scope was limited to issue identification in existing contracts, not drafting, negotiation or advice. The oversight layer is not optional; the question is how to size it efficiently.
Hallucination risk needs a validation process. LLMs can produce confident, plausible-sounding output that is factually wrong. For contract review, a false negative on a material clause is a legal exposure. Validation protocols, human sign-off on AI outputs before they are acted on, must be designed before deployment, not after the first error surfaces.
What the Numbers Don’t Settle
The Onit paper is built around a single dataset reviewed to a single ground truth. How well the accuracy figures generalise across contract types, jurisdictions and legal standards is an open question the study acknowledges but does not resolve. The cost figures rest on commercial API pricing, which varies by volume tier and changes as vendors update their models. The speed advantage, while substantial, becomes less decisive as the proportion of attorney review time in a hybrid workflow rises.
None of that undercuts the core finding: at F-score parity with LPOs and junior lawyers on issue determination, LLMs have cleared the accuracy bar for high-volume, first-pass contract review. The remaining variables are operational, not capability-based.
