Legal AI Accuracy Claims: How to Read Them Like a Skeptic

The Deck That Said 50% Faster

Legal AI accuracy claims tend to arrive the same way: a vendor deck with one number on the cover and a footnote in six-point type. Last spring, a general counsel I know forwarded exactly that, up to 50% faster contract review, with a promise of better dispute-risk identification buried in the appendix. No sample size. No contract types. No definition of "faster", faster than what, exactly, and measured on whose paper?

She had two weeks to decide. Her procurement team, reasonably, wanted to know whether the number was real.

I'm not picking on that vendor. Claims like this get published all the time, and the publicly available article behind hers genuinely does promise up to 50% faster review. The problem is what's missing: the snippet doesn't identify the benchmark, the sample size, the contract types, the jurisdictions, the baseline, or what counts as an error. A number without a method is a mood, not a measurement.

Legal teams are buying document AI the way they once bought e-discovery software, under deadline, with thin technical support, and a lot of faith. The stakes are higher now. Contract review touches indemnities, liability caps, and termination rights, and a tool that misses one of those doesn't produce an inconvenience. It produces exposure that surfaces 18 months later, usually during a dispute, usually in front of someone who asks why nobody caught it.

So how do you tell a real capability from a well-designed slide? You don't need a data science degree. You need six or seven questions and a willingness to be annoying in a sales call. This piece is the framework I'd hand to that general counsel, and to anyone else staring down a procurement deadline.

What "Accurate" Actually Measures

Accuracy in document review isn't a single number. It's at least two numbers pulling against each other, plus a definition of what a mistake even is. Vendors who quote one figure are usually quoting the flattering one.

The pair that matters is precision and recall. Precision asks: of everything the system flagged, how much was real? Recall asks: of everything real, how much did it catch? A tool that flags every clause in the document has perfect recall and useless precision. A tool that flags nothing has the reverse. Both can be described as "accurate" if you only report the number you like.

For contract work, false negatives are the expensive failure. A false positive costs you five minutes of looking at a clause that turned out fine. A false negative costs you the clause. One direction of error is annoying; the other is a lawsuit.

There's a third layer that marketing decks skip entirely: what counts as a correct extraction. If the tool pulls a liability cap of "12 months of fees" and the contract says "the greater of 12 months of fees or $250,000," is that a hit or a miss? Technically it found the clause. Practically, it dropped the carve-out that makes the cap meaningful. Clause-level scoring rewards that. Outcome-level scoring doesn't. If you want a plain-language explanation of how precision and recall get calculated, scikit-learn's model evaluation documentation is more readable than most vendor whitepapers.

And here's the question I'd ask before any of it: who decided what the right answer was? That's ground truth, and in legal work it's rarely as objective as vendors imply. Two experienced lawyers reading the same indemnity clause often disagree about whether it's standard. If the humans disagree, the AI's "accuracy" score against their labels is measuring something fuzzier than the slide suggests.

Why a Demo Is Not a Benchmark

Demos are curated. Benchmarks are adversarial. The gap between them is where most bad purchases live.

A team I spoke with ran a flawless demo on a commercial lease, clean scan, standard form, one amendment. Everyone left impressed. Then they ran the tool on 40 of their own vendor agreements, a mix of legacy paper, redlined Word files, and one scanned exhibit with a stamp across the warranty section. The tool normalized party names inconsistently across the set and missed a liability cap that appeared in a schedule rather than the body. Nobody had tested a schedule in the demo, because the demo contract didn't have one.

That's not a knock on the technology. It's a knock on the test design. And it's exactly why a demo tells you whether the tool works on the demo.

A credible benchmark names five things: the sample and how it was selected, the contract types and jurisdictions covered, the baseline it's being compared against, the definition of an error, and the distribution of results rather than a single average. Most published claims name one or two. The 50% faster figure floating around the market names roughly zero.

Consider what "faster" even means as a baseline. Faster than a first-year associate? Faster than a partner who's seen 300 of these? Faster than the same team using a previous-generation tool? Those are wildly different comparisons, and the same system can honestly beat one and lose to another. Without the baseline, the percentage is decoration.

Jurisdiction matters just as much. Contract conventions that read as standard in one market are nonstandard in another, and research on legal AI consistently flags jurisdiction, traceability, and bias sensitivity as areas where results vary. A tool tuned on US commercial paper may read a UK or EU agreement confidently and wrongly. Ask which jurisdictions were in the training and test sets, then ask again when the answer is vague.

Five Questions That Expose a Weak Claim

Most of the work happens in the first ten minutes of a conversation. Here's what to ask, in order.

1. Faster than what baseline, and measured how? If the answer involves a time study, ask for the raw times. If it involves an estimate, you've learned something.

2. Which contract types and which jurisdictions? "Commercial contracts" is not an answer. NDAs, MSAs, leases, and license agreements have different clause structures and wildly different failure modes.

3. Who labeled the ground truth, and did they agree with each other? A single annotator is a data point. Three annotators with an agreement score is a benchmark.

4. What counts as an error? Missed clause, wrong extraction, wrong classification, missed cross-reference? Ask them to show you one real error from the test set. Vendors who've done the work can do this in seconds.

5. What happens to output nobody can verify? Every system produces some. The question is whether it flags uncertainty or produces a confident summary of something it half-understood.

That fifth question is the one I'd push hardest on. Research on AI-assisted legal work keeps landing in the same place: the strongest operating model pairs automated extraction and drafting with human judgment, auditing, and approval. Systems designed for contract review are built to spot missing or nonstandard language and route those findings to a person. But routing only helps if the flagged items are real and the unflagged ones are safe. Confidence without calibration is just a nicer-looking guess.

Run these five questions past two vendors and you'll notice something immediately. The weak ones answer with adjectives. The strong ones answer with numbers, caveats, and a willingness to say "we don't perform well on that contract type yet."

The Numbers a Credible Claim Carries

You don't need to design the study. You need to recognize one when it's shown to you.

A real evaluation usually includes a sample in the hundreds, not dozens, enough to survive the awkward question of which contract types dominate the mix. It reports the contract mix and the jurisdictions. It names the baseline explicitly. And it reports a distribution: median performance, worst decile, and where the failures cluster. A system that's excellent on 80% of contracts and dangerous on the other 20% has an average that hides the part you care about.

It also separates error types. Missing an entire indemnity clause and mislabeling a notice period are not the same failure, and a single accuracy percentage flattens them into one comfortable number. The useful version looks like a table: clause type, recall, precision, and the business cost of a miss.

That last column is the one legal teams should insist on, because it reframes the whole conversation. A missed governing-law clause is usually recoverable. A missed liability cap, an unspotted auto-renewal, or an indemnity that survived a redline is not. Once you're cost-weighting errors, vendor claims built on raw counts start looking thin.

The same skepticism applies to risk scoring. Several tools now assign clause-level risk and recommend redlines by comparing incoming contracts against preapproved standards. That's genuinely useful, and it means the important question shifts from "how accurate is the model?" to "how current is the standard it's measuring against?" A stale standard automates outdated guidance at scale. Accuracy against the wrong baseline is just speed in the wrong direction.

Governance frameworks are catching up to this. NIST's AI Risk Management Framework lays out expectations around measurement, documentation, and monitoring that translate reasonably well to legal document work, even if it wasn't written for contract review specifically. Vendors who can map their claims to that vocabulary are usually the ones who did the measurement in the first place.

How to Run a Pilot That Answers the Question

Skip the beauty contest. Build a test set from your own paper.

Pull 40 to 60 executed contracts from the last two years, deliberately oversampling the messy ones, the scanned amendments, the multi-schedule agreements, the deals where legal got dragged in late. Have two reviewers independently extract the clauses you care about most. Reconcile their disagreements and keep the unresolved ones as a separate category. That reconciled set is your ground truth, and it's worth more than any vendor benchmark.

Then run the tool blind. Don't tell it which document types are hard. Measure four things: time per contract, recall on your priority clause list, the number of items routed for human review, and the share of routed items your reviewers disagreed with.

That last metric is the one most pilots forget. It's your false-positive load, the noise the tool generates that a person still has to clear. A system that catches everything but flags 200 items per contract hasn't saved anyone time. It's moved the work.

A workable pilot also needs an honest time baseline. Stopwatch your current process on ten contracts before the pilot starts. Not an estimate from memory, an actual measurement. Teams routinely discover their "manual" process is faster than they remembered on standard NDAs and much slower on the contracts that matter. That split is the real business case.

Finally, run the pilot on documents you'd actually send. Sanitized demo contracts hide the things that break extraction: unusual defined terms, inconsistent party names, cross-references to deleted sections, deadlines expressed in three different formats. Those aren't edge cases in real portfolios. They're Tuesday. Cornell's Legal Information Institute is a useful refresher on how much of contract interpretation turns on definitions and their interactions, which is precisely where automated extraction needs a human check.

The Metric Vendors Rarely Put on the Slide

Documents processed per hour is a throughput number. It tells you the machine is busy. It says nothing about whether the output is safe.

As automation handles routine extraction and comparison, human value moves toward resolving ambiguous, high-impact, cross-functional exceptions. That's review by exception, and it only works if the exception queue is honest. The metrics that actually predict whether this model holds up are less photogenic: average review time per contract, the percentage of contracts routed for escalation, unresolved exceptions at signing, override rates, and post-signature disputes.

Override rate deserves its own paragraph. It's the share of the system's recommendations a human changed. A low override rate on a well-designed pilot means the tool and the reviewer agree. A suspiciously low override rate can mean the opposite, reviewers rubber-stamping output because checking it feels redundant after the fourth confident-looking summary. That's the failure mode nobody demos.

Post-signature disputes are the slowest and most honest metric of all. They take a year or two to show up, which is exactly why they're absent from procurement decks. But if your contract AI is missing indemnity carve-outs, the evidence will eventually arrive in the form of a disagreement about who pays.

The operational discipline here is unglamorous. Track the escalation queue weekly. Look at what's sitting in it. Sample closed items and check whether the resolution was right. The teams doing this well treat human-in-the-loop as an actual workflow with owners and review cycles, not a phrase in a security questionnaire.

Good Vendors Should Want This Scrutiny

Here's the part that surprises people: the best vendors in this category don't flinch at hard questions. They volunteer the limits.

They'll tell you which contract types they handle well and which ones still need full human review. They'll show you a failure case without being asked twice. They'll describe their evaluation set, including the awkward detail that it was built by their own team and therefore isn't fully independent. That kind of honesty is a buying signal, not a weakness.

You can spot the difference in about fifteen minutes. Vendors who did the measurement talk about distributions, error types, and jurisdictional variance. Vendors who didn't talk about speed, magic, and the future of the profession. The confidence is inversely proportional to the evidence, almost every time.

The broader trend supports this. Legal AI is moving from search toward workflow, classifying contracts, comparing terms against playbooks, scoring risk, proposing redlines, routing work to the right reviewer. Every step in that direction raises the cost of an unmeasured claim, because the system is now making decisions rather than returning results. Research and professional guidance keep converging on the same operating model: automated extraction plus human oversight, with auditing and transparency built in and jurisdictional review done before deployment.

Which brings me back to that general counsel and her two-week deadline. She didn't need a better deck. She needed a bigger test set and permission to be skeptical in the room. She got both, ran a 50-contract pilot, and ended up choosing a vendor whose headline number was smaller and whose methodology section was longer. Her review time on standard NDAs barely moved. Her review time on the messy, high-value agreements dropped by a third, and that's where the money was all along.

The next generation of buyers will ask these questions by default. Until then, ask them early, ask them twice, and write down the answers. The vendor who can survive that conversation is the one worth signing with.