A pattern shows up in almost every AI coding pilot we've watched from the outside: the team runs a trial on a new service, gets a genuinely impressive speed-up, greenlights a wider rollout, then points the same tools at the platform the business actually depends on and can't reproduce the number. This isn't inconsistent execution, and it isn't the tool underperforming. Recent research shows it's structural, and it's worth understanding before you scope your next AI-assisted modernisation effort around the wrong benchmark.
The headline studies on AI coding speed are real, and the gains are real too. One widely cited trial had developers build an HTTP server in JavaScript, from scratch, with an AI assistant. They finished 55.8% faster. That's the number that ends up in a slide deck, and it isn't wrong. It's just describing a specific kind of work: greenfield, no history, nothing to integrate with.
A second study, published in 2025, tells a different story. Sixteen experienced open-source developers worked 246 real tasks inside repositories averaging over a million lines of code, systems they'd each spent roughly five years in. Half the tasks allowed AI tools, half didn't. Allowing AI tools didn't speed anything up. It slowed completion time by nineteen per cent.
What makes the result worth noting is what the developers believed. Beforehand, they predicted AI would make them 24% faster. Afterward, having just been measurably slower, they believed they'd gained 20%. Economists and machine learning researchers asked to forecast the outcome in advance guessed even higher. Almost everyone in the room was wrong, in the same direction, including the people the slowdown happened to.
If a rollout decision is being made on the basis of how fast engineers say AI feels, that's not a safe input. The research shows the perception gap holds even among experienced developers who are paid to notice.
The two studies aren't in tension. They're measuring different halves of the same job. Writing code and integrating it into an existing system used to be one activity, done by the same senior engineer in the same sitting. AI is very good at the first half and does almost nothing for the second, which means the two costs are now separable, and only one of them has actually gotten cheaper.
Three long-standing ideas in software engineering explain why the second half resists. Fred Brooks distinguished the accidental complexity of software, the labour of expressing a design in code, from its essential complexity: the underlying logic, its fit with everything else the system already does, and how it needs to change over time. AI attacks the accidental part. It leaves the essential part exactly where it was. Peter Naur went further: a working system isn't really the code, it's the theory the team holds about how the system maps onto the real-world problem it solves. A model that generates text without holding that theory produces code nobody fully understands yet, and the bill for that arrives at the next change, not the first one. Melvin Conway's observation that systems mirror the communication structure of the organisations that built them adds the last piece: an accumulated codebase is, in effect, a record of years of coordination. Integrating into it means paying that coordination cost again, regardless of who or what wrote the new lines.
The practical implication is specific: don't scope a modernisation programme's expected AI gains off a greenfield pilot. The pilot tells you what AI does to a blank page. Your platform is not a blank page, and the more valuable and more accumulated it is, the smaller the gain you should expect from tooling alone.
What actually moves the number on a legacy system is making the essential complexity easier to work with in the first place: decision records that capture why the system is built the way it is, module boundaries that let an agent's blast radius be scoped and reviewed, and test coverage that lets a change be verified quickly instead of taking someone's word for it. That's not a side effect of AI adoption. On a mature system, it's the actual project, and AI-generated code is only as safe as the structure it's landing on.
This is also where AI engineering assurance earns its keep. The gap between "the model produced a plausible-looking change" and "someone verified it's correct for this specific system" is exactly where the nineteen per cent went. Closing that gap deliberately, with review discipline and monitoring built for agent-generated output specifically, is what turns a fast prototype into a fast production system, rather than a fast prototype and a slow, quietly deferred integration bill.
AI's speed gains are real, but they're concentrated in exactly the place a benchmark study measures them: new code, with nothing to integrate against. On an accumulated production system, the constraint shifts from how fast the model can write to how well the organisation has prepared that system to absorb the change safely. A modernisation programme that treats those as the same problem will hit the nineteen per cent before it hits the fifty-five.
If you're planning a modernisation programme and want a clearer read on what AI will and won't accelerate on your specific systems, we're happy to talk it through. Contact us to explore what fits your systems.
.jpeg)
Because writing code and integrating it into an existing system are different costs. AI reduces the first one sharply. It does almost nothing for the second, which is dominated by undocumented context, dependencies, and coordination history that only shows up once you try to change something that's already in production.
No, it means the gain won't come from the tool alone. It comes from pairing the tool with the groundwork that makes a system easier to integrate into: clearer module boundaries, documented decisions, and stronger test coverage. Without that groundwork, AI-generated changes still need the same manual verification they always did.
Benchmark against similar-maturity systems, not against a greenfield pilot. A speed-up measured on a brand-new service will not transfer to a ten-year-old platform, and setting a rollout target based on the wrong number is one of the more common ways these programmes lose credibility internally.
