One number has done more than any other to make people suspicious of AI coding tools: 19 percent slower. It comes from a real randomized trial, it was reported accurately, and the researchers who produced it have since said they think it no longer describes the present. Almost nobody who quotes the number quotes that part.
If you run a small software shop, or you are the one founder who also writes the code, the honest version of this story is more useful than either the hype or the backlash. Here is what was actually measured, what the authors said about it, and what changed.
What the study measured
In July 2025, the research group METR published a randomized controlled trial of 16 experienced open-source developers working through 246 real issues on repositories they already knew well, codebases averaging more than 22,000 GitHub stars and over a million lines of code. Tasks were randomly assigned to allow or forbid AI tools. Developers took 19 percent longer to complete issues when they were allowed to use AI (METR).
The striking part was never the slowdown. It was the perception gap. Those developers expected AI would speed them up by 24 percent going in, and after finishing the tasks and being measured, they still believed it had sped them up by 20 percent (METR). People who had just been slowed down could not feel it.
That is the finding worth carrying around. Not “AI makes developers slower,” but “developers are unreliable narrators of their own speed.” Those are very different claims, and only one of them survives.
The caveats were in the original, and they were skipped
METR was unusually direct about what the study did not show. The researchers explicitly stated it was not evidence that AI fails to speed up most developers, since 16 people on mature open-source projects is not a representative sample, and not evidence that current AI cannot be used more effectively to get a real speedup. They also flagged that participants had limited experience with the specific tooling, and that learning effects past roughly 50 hours of use were outside what the study could see (METR).
The tools matter too. The trial ran on Cursor Pro with Claude 3.5 and 3.7 Sonnet, which were the frontier models at the time (METR). A result about early-2025 models is a result about early-2025 models.
The follow-up broke, and how it broke is the story
METR ran a new version of the experiment and published an update in February 2026 saying the design had failed them. Their words: “we believe that the data from our new experiment gives us an unreliable signal of the current productivity effect” (METR).
The reason is the interesting bit. Developers would not cooperate with the no-AI condition. METR reports that 30 to 50 percent of developers told them they were choosing not to submit some tasks because they did not want to do those tasks without AI (METR). The people most attached to the tools quietly removed themselves from the sample, which biases the measured effect downward.
Sit with that for a second. A productivity study became unrunnable because a large share of professional developers would rather skip paid research tasks than do them unassisted. That is not a controlled measurement, but it is real behavioral evidence, and it points the opposite direction from the headline. METR’s own current position is that “it is likely that developers are more sped up from AI tools now, in early 2026, compared to our estimates from early 2025,” while cautioning that this is only very weak evidence about the size of any increase (METR).
What a small shop should actually do with this
The lesson is not which way the arrow points. It is that you cannot answer this question by asking your team how it feels, because the one thing the research established firmly is that the feeling is wrong in both directions.
So measure something small and real instead. Pick one repetitive category of work you do often, such as writing tests for existing code, or a first pass at a migration script. For four weeks, note the date, the task, and elapsed time for each one. Do half with the assistant and half without, alternating, and do not decide in advance which kind of task gets which. Then compare the two columns.
Four weeks of a plain spreadsheet will tell you more about your codebase than any published trial will, because the published trials keep finding that the answer depends on the codebase. Mature, unfamiliar, or heavily idiomatic code was where METR’s developers lost time. Greenfield and boilerplate is where most people report gains. Your repo is one of those or the other, and only you can check which.
Watch the review side too. If assistant-written changes take longer to review or come back more often, that cost is real and lands on someone other than the person who felt fast. Small teams that have gone furthest with this, including the ones replacing expensive SaaS with code they now maintain themselves, tend to discover the bill arrives in maintenance rather than in the build, which is the pattern behind The AI Tools for Small Business That Are Firing Salesforce.
What it costs to run that test
Finding out is cheap, which is the strongest argument for measuring rather than arguing. GitHub Copilot has a free tier at $0, with Pro at $10 per user per month and Pro+ at $39 per user per month (GitHub Copilot plans). Cursor, the tool used in the original METR trial, offers a free Hobby tier, Pro at $20 per month, and Teams at $40 per user per month (Cursor pricing).
For a four-week experiment on a two-person team, that is a rounding error against the cost of guessing wrong for a year in either direction. If you are earlier than that and still deciding where AI belongs in the business at all, the broader version of this question is covered in AI for Small Business: A Practical Guide to Getting Real Results.
The part the argument keeps missing
Most of the public fight over these tools is really a fight about whether developers are about to be replaced, which makes both sides read the evidence badly. The METR results do not speak to that at all. What they describe is a tool that changes where an experienced person’s time goes, sometimes for the better and sometimes not, on work that still requires someone who understands the system to judge whether the output is right.
The version of this that works for a small team is the one where a two-person shop takes on the kind of project that used to need five people, not the one where a five-person shop becomes two. The second story gets more attention. The first is the one that shows up in the repos.
Run your four weeks. Then you will have a number that is about your code rather than someone else’s, which is the only kind that should change how you work.
Frequently asked questions
Is the 19 percent slowdown finding wrong?
No. It was a properly conducted randomized trial and the result stands for what it measured: 16 experienced developers on mature open-source repositories using early-2025 tools. METR itself now believes developers are likely more sped up in early 2026 than that estimate suggests (METR). The finding is dated, not discredited.
Why can’t developers tell whether AI is helping them?
The trial found participants believed AI had sped them up by 20 percent while measurement showed the opposite (METR). Waiting on a generated suggestion feels less effortful than typing, even when the total time including review and correction is longer.
How long should my own test run before I trust it?
Four weeks on one repeated task type is enough to see a direction. Alternate assisted and unassisted work rather than choosing per task, since choosing is how bias gets in, which is exactly what went wrong with METR’s second experiment.
Does this apply to non-developers using AI at work?
METR states plainly that its results do not extend beyond software development (METR). The transferable part is the perception gap, which is a good reason to measure any AI workflow rather than survey people about it.
