AI Productivity Paradox: Feeling Faster, Shipping Slower
Developers report feeling faster with AI; measured outcomes tell a messier story. What the research actually found, and how to test it on your own team.
Source: METR RCT3 receipts in this article ↓
TL;DR: The best available controlled evidence says developer perception of AI speedup cannot be trusted: in METR's randomized trial, experienced developers were 19% slower with AI access while believing they were 20% faster. That result doesn't make AI useless. It makes "the team loves it" worthless as a delivery metric. Test the paradox on your own repo before renewing seats on vibes.
What is the AI productivity paradox?
The AI productivity paradox is the measured gap between how much faster developers feel with AI coding tools and what delivery data actually shows. Self-reported speedup is consistently, confidently positive; controlled measurement is messier: sometimes flat, sometimes negative, always smaller than the vibes suggest. The paradox isn't that AI never helps. It's that the people using it can't feel the difference between help and drag.
What the METR study actually found
METR ran the cleanest experiment in the field: a randomized controlled trial with 16 experienced open-source developers completing 246 real tasks - bug fixes, features, refactors - on mature codebases they'd contributed to for years. Each task was randomly assigned AI-allowed or AI-disallowed; the allowed condition used the era's frontier tooling (primarily Cursor Pro with Claude 3.5/3.7 Sonnet).
Three numbers, in the study's own units:
- Before starting, developers expected AI to make them 24% faster.
- With AI allowed, they were 19% slower than without.
- Afterward - having just lived the slowdown - they still estimated AI had made them 20% faster.
Sit with that third number. These were not AI tourists. They were expert developers on home turf, and their perception missed the sign of the effect, not just the size. Whatever your team reports in the next retro about AI speedup is drawn from the same well of self-perception.
The honest caveats, stated plainly because the study states them: this was early-2025 tooling, expert developers, and mature high-standards codebases. It doesn't tell you about juniors, greenfield work, or this year's models. What it does tell you survives all of those caveats: perceived speedup is not evidence.
Why do developers feel faster when they aren't?
No one is lying. The miscalibration has mechanics:
- Effort is not throughput. Prompting feels lighter than typing, and the brain reads "less strain" as "more speed." The clock disagrees.
- Generation time feels like progress. Watching plausible code stream in registers as work happening. Review, correction and integration, where the minutes actually went, register as friction. Friction is forgotten first.
- Output volume is visible; rework is deferred. The 400 generated lines land today and feel like a sprint's work. The rewrites land over the following weeks, itemized nowhere. The research covered in AI code churn shows exactly where that deferred bill shows up in git.
- The demo anchors the memory. The one task where AI nailed it in a minute becomes the reference experience; the five tasks of prompt-wrangling median out of memory.
Is AI actually making your team faster? How to test it
A survey won't settle this, and neither will a vendor's acceptance-rate dashboard. Acceptance measures how agreeable the suggestions felt to the person accepting them. You settle it the way METR did, scaled to your reality: outcomes, measured on your own repo, before versus after.
The short honest checklist:
- Cycle time, end to end. First commit to merge to deploy. If AI is buying real speed, this is where it shows up.
- Review load and pickup. Did PR volume and size rise while review depth fell? Then some of your "speedup" is unreviewed risk relocating downstream.
- Short-window churn on shipped code. The treadmill check. GitClear's 623M-change dataset measured two-week churn up 15% and duplication up 81% across the AI era; Faros measured bugs per developer up 54% at high adoption. Check whether your repo is in that pattern before assuming you're the exception.
- Attribution you can defend. Compare AI-assisted vs non-assisted work only via explicit signals: trailers, bot accounts. Guessed attribution produces confident nonsense, which is the failure mode this whole exercise exists to escape. The slop-detection playbook in AI slop code covers the signal set.
Run the comparison over months, not sprints, and judge trends. One measurement ritual, honestly kept, beats any quantity of enthusiasm.
What this doesn't mean
This is not a case for banning the tools. A ban would just replace one unmeasured belief with another. METR's result is scoped, later tools differ, and some workflows genuinely benefit. The case is narrower and harder to argue with: your team's feelings about AI, including yours, are not data. The organizations that win this era will be the ones that can tell the difference, because they measured their own delivery instead of outsourcing the verdict to vibes or vendors.
Feeling faster is free. Knowing is cheaper.
Frequently asked
Did the METR study prove AI coding tools don't work?
No. It measured 16 experienced open-source developers on mature codebases they knew deeply, with early-2025 tools. In that setting, AI access made them measurably slower while they believed the opposite. The result generalizes as a warning about trusting perception, not as a verdict on every team, task, or later tool generation.
Is developer self-report useless for judging AI tools?
Not useless, miscalibrated. In METR's trial, developers expected a 24% speedup, experienced a 19% slowdown, and still estimated a 20% speedup afterward. Surveys tell you how the tools feel, which matters for retention and adoption. They cannot tell you what the tools did to delivery. You need both, and only one comes from asking.
What should we measure instead of asking developers if AI helps?
Delivery outcomes from your own repo: cycle time from first commit to merge, review pickup and depth, short-window churn on shipped code, and defect flow. Compare before and after adoption, or between explicitly attributed AI-assisted and non-assisted work. Vendor acceptance-rate dashboards are marketing, not measurement.