A lot of discussion about AI coding tools jumps between two extremes: benchmark wins on one side, glowing user stories on the other. Both are useful, but neither directly answers a simple practical question: if experienced developers work on code they know deeply, do current AI tools actually help them finish faster? 🧪
A randomized controlled trial tackled exactly that question by studying seasoned open-source contributors making real changes in repositories they had worked on for years. The result was unexpected: with early-2025 AI tools available, developers did not speed up. They took 19% more time on average.
That finding matters because it comes from a setting much closer to production software work than most benchmark suites. At the same time, it should be read carefully. It is a historical snapshot of early-2025 tooling, not a timeless verdict on all AI-assisted development. By February 2026, newer results had already been published for later-generation tools, and these older findings were explicitly described as outdated for understanding the current frontier.
⚠️ Why This Study Was Worth Running
Coding benchmarks and agent evaluations are useful, but they usually trade realism for scale. Tasks are often isolated, compact, and easy to grade automatically. That makes them convenient to run, yet it can distort what they tell us.
There are at least two ways that distortion can go:
Benchmarks may make models look better than they are in day-to-day engineering by focusing on cleanly scoped tasks with crisp scoring.
They may also make models look worse than they are in practice, because a real human can step in, unblock a stuck workflow, and correct small failures during normal use.
So benchmark scores do not map cleanly to impact in the field. 🚧
The broader motivation here is not just software productivity for its own sake. Understanding whether AI can meaningfully accelerate development work also informs a bigger question: how much could AI accelerate AI research and engineering itself? If that acceleration becomes strong enough, it could affect oversight, safeguards, and the pace of capability growth.
🛠️ How the Experiment Was Set Up
The researchers recruited 16 experienced developers from substantial open-source repositories. These were not tiny side projects. On average, the repositories had more than 22k stars and over 1 million lines of code, and the participating developers had already spent multiple years contributing to them.
The workflow looked like this:
Developers submitted lists of genuine issues they considered worthwhile for their projects.
Across all participants, this produced 246 real tasks.
Those tasks included bug fixes, new features, and refactors.
Each issue was randomly assigned to one of two conditions:
AI allowed
AI not allowed
When AI use was permitted, developers could choose whatever tools they wanted. In practice, they mostly used Cursor Pro with Claude 3.5 Sonnet and Claude 3.7 Sonnet, which were frontier systems at the time. When AI use was disallowed, they worked without generative AI assistance.
The tasks were not trivial one-minute prompts. They averaged about two hours each. Developers recorded their screens and then reported the total implementation time they spent. Participants were paid $150 per hour. ✅
🚀 The Main Outcome: Slower, Not Faster
The headline number is blunt: when AI tools were available, developers needed 19% longer to complete issues.
That result cuts against both participant intuition and common expectations. Before the experiment, developers predicted AI would make them about 24% faster. Even after going through the study and actually experiencing the slowdown, they still believed AI had improved their speed by about 20%. 🤯
That gap between perceived productivity and measured productivity is one of the most interesting parts of the study. It suggests that “this feels useful” and “this reduces elapsed implementation time” are not the same thing.
📌 What This Result Does Not Prove
The study authors were careful not to over-claim. The evidence does not establish any of the following:
That AI fails to help most software developers
That these developers represent the majority of software work
That the result extends beyond software engineering
That future models in the same setting will also slow people down
That no better workflow with existing tools could produce a speedup
Those caveats matter. The study covered experienced maintainers working in codebases they already knew very well, under quality expectations consistent with real pull requests. That is a specific environment, not the entire software industry. ⚠️
It is also possible that better prompting, better scaffolding, more repository-specific context, or stronger fine-tuning could change the outcome in this exact setting.
🔍 Looking for Explanations
The researchers examined 20 possible factors that might explain why AI use was associated with slower completion times. They found evidence that 5 likely contributed.
Just as important, they ruled out several common objections:
The developers were using frontier models for that period
Participants generally followed their assigned condition
There was no clear sign of skew from selectively dropping harder tasks
Pull request quality looked similar with and without AI
The slowdown remained visible across multiple outcome definitions and estimation approaches
So this was not a fragile artifact that disappeared the moment the analysis changed. 🧭
🧩 Why This Clashes with Benchmarks and Anecdotes
At first glance, the result looks hard to reconcile with two other familiar observations:
Benchmark systems sometimes perform impressively on coding tasks
Many developers report that AI tools are meaningfully helpful
The tension becomes easier to reason about once you separate what is being measured.
🧱 Real repository work in the trial
Task type: pull requests in large, mature open-source codebases
Success criterion: the human developer is satisfied the change would pass review, including style, tests, docs, and implicit expectations
AI mode: chat, Cursor agent mode, autocomplete
Observed pattern: AI slowed people down on realistic tasks lasting roughly 20 minutes to 4 hours
🧪 Benchmark-style evaluations
Task type: things like open-source PR tasks with author-written tests, or manually designed research problems with automatic scoring
Success criterion: algorithmic evaluation, usually via tests or similar machine-checkable outcomes
AI mode: often more autonomous systems with complicated scaffolding and potentially millions of sampled tokens
Observed pattern: models can succeed on tasks that are genuinely difficult for humans
💬 Anecdotes and broad adoption
Task type: highly varied
Success criterion: a human finds the result useful
AI mode: mixed usage across models and tools
Observed pattern: many people, though certainly not everyone, say AI is valuable for software tasks taking more than 1 hour
These are not identical measurement systems. They answer different questions. 📎
🧠 Three Plausible Ways to Reconcile the Evidence
A helpful mental model is to treat the three evidence streams as potentially measuring overlapping but non-identical things.
1. 🛑 The trial may understate real capability
Under this view, benchmark scores and user reports are mostly right, while the experiment misses some important ingredient. Maybe the methodology introduces hidden distortion, or maybe the chosen setting is unusually difficult for AI.
For example, the developers in the study typically did not sample hundreds or thousands of trajectories per problem. If strong performance requires much heavier elicitation, then the trial naturally undershoots what models can do in a maximally optimized setup.
There could also be long learning curves. Tools like Cursor may only pay off after several hundred hours of use, while these developers usually had only a few dozen hours with Cursor before and during the experiment. 🕒
2. 📉 Benchmarks and anecdotes may overstate usefulness
Under this interpretation, the randomized trial is closer to the truth for this kind of serious engineering work, while benchmarks and self-reports are overly generous.
That is plausible because benchmarks often reward success on neatly bounded tasks, and because self-reported productivity is now backed by strong evidence of being error-prone. People can feel faster while actually taking longer.
3. ⚖️ All three are valid, but for different slices of reality
This mixed explanation is likely the most intuitive. Benchmarks, anecdotes, and controlled trials may each be accurate for the subset of tasks they represent.
AI may look strong on tasks that are tightly scoped, automatically graded, or tolerant of rough edges. It may be less effective where quality bars are high and many requirements are implicit: documentation standards, test quality, lint rules, project conventions, backward compatibility, and reviewer expectations. 🛠️
🧪 A Few Important Nuances from the FAQ
Several common objections were addressed directly.
One obvious question is: if developers could choose whether to lean on AI, how did they still end up slower? The answer is that they misjudged the effect. On average, after the study, they still believed AI had sped them up by 20%. It is also possible they valued AI for reasons beyond pure throughput, such as enjoyment or future skill-building.
Another concern is sample size. Yes, 16 developers sounds small, but each participant completed issues in both conditions, and the study included 246 completed issues total. The empirical strategy reported enough statistical power to reject the null of zero effect, though representativeness remains a legitimate concern.
There is also the “maybe they were AI beginners” argument. The study says most participants had substantial prior experience with LLM prompting, often ranging from dozens to hundreds of hours. That does not rule out further learning effects past 50 hours of Cursor use, but it does weaken the idea that the sample was made up of total novices. 🔧
Finally, the result does not imply AI is useless in software engineering. The authors explicitly leave room for the possibility that AI helps more in different contexts, such as for less experienced developers or for engineers working in unfamiliar codebases.
🔭 What This Means Going Forward
No single measurement approach captures the whole story. Real software work is messy, diverse, and difficult to study cleanly. That means we should expect tradeoffs:
Controlled trials bring realism, but only within a narrow setting
Benchmarks scale well, but often simplify the problem too much
Anecdotes capture lived experience, but can be unreliable
The practical takeaway is not “trust benchmarks” or “ignore benchmarks.” It is that multiple evaluation methods are necessary if we want a serious picture of AI capability. 📌
The study also points to an important policy and forecasting angle. If AI eventually begins to deliver large speedups for expert developers in realistic, high-context environments, that could signal broader acceleration in AI R&D itself. That would have implications for risk, governance, safeguards, and concentration of power.
For now, the historical finding is straightforward: in this specific experiment, with early-2025 tools, experienced open-source maintainers working on their own repositories became 19% slower, not faster. That does not settle the future. But it does puncture the assumption that impressive demos and confident user impressions automatically translate into real productivity gains. ✅
🔍 TL;DR Summary
🧪 A randomized trial with 16 experienced open-source developers and 246 real issues found that early-2025 AI tools made them 19% slower on average.
🤔 Developers expected a 24% speedup beforehand and still believed they got about a 20% boost afterward, despite measured slowdown.
📌 The result applies to a narrow but important setting: experienced contributors working on familiar, large repositories with real PR-quality expectations.
⚖️ The study does not show that AI is broadly useless for software development, only that this specific workflow did not speed up this specific group.
🧩 Benchmarks, anecdotes, and controlled trials may each be accurate for different task distributions, which is why all three matter.
🔭 By 2026, these findings were already considered historically limited, but they remain valuable as evidence that perceived usefulness and actual productivity can diverge sharply.