The AI Productivity Myth: Faster Isn't the Same as Done

The Spreadsheet Said 47 Hours. The Missed Deadline Said Otherwise.

I keep a time log that borders on obsessive. Every task, every timer, every little "the AI saved me 12 minutes" note. On March 31, the sheet totaled 47 hours saved that month through drafting, summarizing, and triaging documents. Two days later, my internet provider auto-renewed a 24-month business plan at a rate I'd been meaning to cancel for weeks. The notice window closed on a Tuesday I never circled, and the early-termination fee was $600.

So which number was real? The 47, or the $600?

I spent the next six months auditing myself. Every AI-assisted task went in the log. Every minute of checking, correcting, and re-reading went in the log too. The final tally: 212 hours "saved" across six months, and fewer finished outcomes than the same stretch a year earlier, when I had no AI in the loop at all.

That's the AI productivity trap in one sentence. It isn't that the tools are bad. It's that we measure them wrong. Speed is easy to log. Correctness is easy to forget. And when you only log the easy half, every automation looks like a win, right up until the renewal notice lands in your inbox.

I'm not anti-AI. I run documents through summarization and extraction every single day. But I had to rebuild how I count what it gives me, and the rebuild was more embarrassing than the original mistake.

Why "Time Saved" Is the Wrong Unit of Measure

Time saved counts minutes that stopped happening. It says nothing about whether the work got done correctly, on time, or at all. A tool that turns a five-minute draft into a 25-minute verification exercise hasn't saved you 30 minutes. It's cost you 20.

The number survives because it's flattering and almost impossible to disprove. Nobody audits "hours saved." It shows up in vendor decks, team retrospectives, and my own spreadsheet for three straight months.

Ergonomics researchers have been pointing at this problem since long before large language models existed. In her 1983 paper "Ironies of Automation," Lisanne Bainbridge described how automating a task doesn't remove the human, it changes the human's job into monitoring and exception handling. Those jobs are often harder, less engaging, and more error-prone than doing the task by hand. Airline pilots still practice hand-flying for exactly this reason: when the automation drops out, the manual skill has to be there.

Watch what happens to document work under the same pressure. You stop reading contracts closely because the summary is "usually right." Six months later, your close-reading muscle has atrophied. And the one agreement that actually matters is the one you skim.

Here's what my March log tracked versus what it ignored:

• Logged: documents summarized, drafts generated, claimed hours saved
• Ignored: verification minutes, corrections, rework, missed deadlines, client-visible mistakes

See the pattern? Every logged item is something that happened to a file. Every ignored item is something that happened to my actual results.

A productivity number that can't survive an audit isn't a metric. It's a mood.

The Review Overhead Nobody Puts in the Case Study

Every AI output that reaches a client, a court, a landlord, or a payment carries a verification cost. If that cost isn't in your log, your savings figure is fiction. In my six-month audit, review overhead ate 61% of the time AI supposedly saved me.

Some of that is pure clock time. A chunk of it is the invisible tax of context switching. Gloria Mark's research at UC Irvine famously found it takes roughly 23 minutes to fully refocus after an interruption, and every half-checked summary is a self-inflicted interruption that costs far more than the 90 seconds it took to generate.

The nastier cost is false confidence. A wrong summary is worse than no summary, because no summary forces you to read, and a wrong summary tells you there's nothing to read. That's not a time savings. That's a liability with a progress bar.

A scenario from my own audit: I ran 60 short agreements through a summarizing tool in one afternoon, feeling extremely efficient. Verifying them took just under nine hours over the following two days. Net result was a wash on time and a small disaster on attention, because the work I'd displaced was the client project that actually paid that week.

My hidden cost list, for anyone building their own audit:

1. Verification time against the source document
2. Correction time when the output is wrong
3. False confidence that suppresses normal skepticism
4. Context switching every time you bounce between summary and original
5. Downstream cleanup when a bad number gets quoted in an email that's already sent

Verification isn't overhead sitting on top of the work. Verification is the work.

The Summary Was Right. It Still Nearly Cost Me $18,000.

A 38-page vendor agreement came back with a clean summary: the customer may terminate for convenience with 30 days' written notice, and fees are non-refundable. Neat, readable, wrong in the way that matters.

The actual clause set was stranger. Termination for convenience wasn't available during the initial 12-month term at all. Notice had to be received by the 10th of the month, not merely sent. And a cross-reference buried in Section 11.4 pulled in an early-termination fee equal to 50% of remaining fees. On our contract value, that worked out to roughly $18,000.

Was the summary lying? Not exactly. "May terminate for convenience" was technically true. It had simply been stripped of its conditions, which is what compression does. This is the verification gap, and it's predictable: summaries preserve topics and lose conditions. Names, numbers, dates, negations, exceptions, and conditional obligations are the first casualties of every pass.

The uncomfortable part is that the summary read better than the original. One page of clean sentences beats nine pages of defined terms in small font. Fluency feels like accuracy. Stanford's HAI researchers and plenty of others have documented how model errors tend to arrive confidently, in perfect grammar, which makes them harder to catch than an obvious glitch.

The fix was mechanical, not philosophical. I now require source anchoring: every extracted point has to point back to a page or paragraph. If a tool can't do that, its output is a table of contents, and I treat it like one.

A summary without a source link is a rumor with good formatting.

Faster Feels Better, and Adoption Data Won't Save You

Speed is visible and immediate. Quality failures are invisible and delayed. That asymmetry explains why adoption keeps climbing even when the results stay flat.

The numbers back this up. According to Thomson Reuters, 41% of law firms and 47% of corporate legal departments said their legal teams were using generative AI, up from 28% and 23% the year before. Legal research and document summarization were the leading uses, each cited by 74% of respondents. And the same professionals are demanding guardrails: 96% wanted safeguards for confidential data, 94% wanted outputs grounded in authoritative content, and 90% wanted explainable reasoning.

Sit with that combination for a second. Usage is exploding, and the people using the tools are openly asking for the features that make them trustworthy. Adoption isn't evidence of value. It's evidence of hope.

There's a second-order effect nobody warns you about. When a client learns that a 60-page agreement can be summarized in 20 minutes, their expectation for turnaround drops. But the document's risk didn't drop. Speed compresses the review window while the exposure stays exactly the same size, which is precisely how a renewal deadline slides past you.

Which is why the sane default is human-in-the-loop work: machine does the first pass, human owns the last word. Not because humans are smarter, but because someone has to be accountable when the sentence in Section 11.4 turns into an $18,000 invoice.

The Four-Stage Loop That Fixed My Week

I split every document task into four stages: extract, classify, question, approve. AI owns the first two with citations attached, and I own the last two. The rule is simple: nothing reaches a decision without a human signature on it.

Stage one, extract. Pull obligations, dates, monetary terms, termination rights, and defined terms into a table where every row carries a source reference. This is the single highest-value thing document analysis does, and it's also the part people trust too much. Keep the references.

Stage two, classify. Score each item on five factors, one to three points each: financial exposure, duration, termination difficulty, personal-data sensitivity, and dispute cost. Twelve points or higher gets a full human read. A risk score prioritizes your attention. It never provides a legal conclusion, and it never closes a file.

Stage three, question. I run the same constraint-heavy prompts every time, because constrained outputs are easier to audit:

• "List every obligation imposed on each party."
• "Identify exceptions to the termination right."
• "Separate explicit statements from inferences."
• "Quote the source location for each answer."

That last one does more for reliability than any clever phrasing, because it makes the model show its work or admit it can't.

Stage four, approve. I read the flagged sections myself. Escalation rules are non-negotiable: unusual clauses, conflicting provisions, unclear source material, or any penalty with a number attached gets a full manual read, regardless of how tidy the summary looks.

The whole loop adds maybe 40 minutes to a contract review. It has saved me from two expensive mistakes in five months. I'll take that trade every week.

What I Track Now, and Where AI Actually Earns Its Keep

My dashboard has four numbers on it, and none of them are hours:

• Completed outcomes per week, each with a name and a due date attached
• Rework rate, the percentage of AI-assisted work I had to redo
• Escalation count, how many documents needed a full human read
• Deadline misses, the only metric that has never once lied to me

I dropped "documents processed" entirely. For one glorious week that number hit 68, and it meant absolutely nothing. Nobody pays for processed.

Where the tools genuinely win: first-pass organization, mapping defined terms across a long agreement, comparing two drafts, extracting obligations into a table, and building a navigation map of an 80-page document. Where they lose: final judgment, arithmetic on penalty formulas, jurisdiction-specific questions, and anything where being wrong costs money or rights. The NIST AI Risk Management Framework is worth reading on this, especially its emphasis on measuring risk and keeping humans in oversight roles rather than treating automation as a finish line.

Use AI to find the pages that matter. Then read those pages yourself.

The cost of reading is falling fast. That means the bottleneck shifts from "can I find the problem in 200 pages" to "do I know what to do about it." Judgment is about to become the most valuable thing in white-collar work, which is either great news or terrifying, depending on how much of yours you've been quietly outsourcing to a summarizer.