AI coding assistants and engineering skills: the evidence
In one trial of AI coding assistants, developers estimated 20% less task time and measured 19% more. What other controlled studies add on speed and skill.
Sixteen experienced developers forecast that AI tools would cut their task time by 24%. After the study, they estimated the tools had cut it by 20%. The measured result was that tasks took 19% longer when AI was allowed. That gap, from a randomized trial by METR, gets quoted as proof that AI tools do not work, and other studies get quoted as proof they make developers twice as fast. Both readings skip the part that makes the research useful: what each study measured, on whom, and what its authors say it does not show.
This post covers only studies we opened and read. It is a sample, not a systematic review, and the tools in most of them are older than what you can install today.
Four speed results, four different questions
Peng et al., 2023
Freelance developers build one HTTP server in JavaScript
55.8% faster with Copilot
Limit: One standardized task. Code quality not examined.
Cui et al., 2025
4,867 developers at three companies, normal work, code completion
26.08% more completed tasks among users, IV estimate (SE 10.3%)
Limit: Counts pull requests. Each experiment is noisy on its own.
METR, 2025
16 experienced contributors on mature repositories they know
19% longer with AI, early-2025 tools
Limit: Small sample, one setting. METR now calls it out of date.
METR, 2026 update
57 developers, late-2025 tools; similar design, lower pay, wider repos
18% faster for 10 returning, 4% for 47 new; intervals include zero
Limit: Selection effects. METR calls it an unreliable signal.
An often-cited positive number comes from Peng and colleagues, in a 2022 experiment. They recruited 95 programmers on Upwork and randomly split them, 45 with Copilot and 50 without, to build an HTTP server in JavaScript as fast as they could. The paper reports that "thirty-five developers from both the treated and control groups completed the task" and its survey, and only finishers enter the comparison: 71 minutes on average with Copilot against 161 without, 55.8% less, with a 95% interval of 21% to 89%. Three of the four authors were at GitHub or Microsoft Research; the fourth was at MIT. It was one standardized task, and the paper says it did not examine code quality.
Cui and colleagues looked at real work. Microsoft, Accenture and a Fortune 100 company each gave a random subset of developers access to Copilot, 4,867 people in total. The pooled instrumental-variable estimate, for developers who actually used it, was 26.08% more completed tasks per week, counted as pull requests (standard error 10.3%). On the same basis, commits rose 13.55% (SE 10.0%) and builds rose 38.38% (SE 12.55%). The authors note that each experiment is noisy alone. At Microsoft, the one site with tenure and seniority data, gains were significant for recent hires and junior staff but not for long-tenured and senior developers. Two authors work at Microsoft. Pull request counts also measure output, not value, and the authors raise the possibility that more builds reflect more trial and error, with weak support for that reading at Accenture only.
METR's trial sits at the other end of the setting. Its 16 participants averaged 5 years of prior experience on the mature open-source projects they worked in, and they completed 246 tasks from their own issue lists, with each task randomly allowed or disallowed AI and the time self-reported alongside screen recordings. METR's own summary says it provides no evidence that AI fails to speed up many or most software developers, or that no more effective way of using existing tools exists, and that it can't rule out learning effects beyond 50 hours of Cursor use. The page now carries a note that the results are out of date.
The February 2026 update is a good lesson in why this is hard to measure. With 57 developers on late-2025 tools, the estimates were an 18% reduction in task time for the 10 returning participants (95% interval -38% to +9%) and 4% for the 47 new ones (-15% to +9%). The design was similar, but pay fell from $150 to $50 an hour and the new developers came from more diverse repositories. METR calls this an unreliable signal, because many developers now decline to work without AI, and 30% to 50% reported withholding tasks they did not want to do without it. METR says developers are likely faster now than in early 2025, and that its data is only very weak evidence of by how much.
Read together, these results fit a plausible pattern: larger gains on unfamiliar, self-contained work and for less experienced developers, and smaller or uncertain gains for experts in code they know well. That is our reading across different studies; no single one tested it. Self-assessment doesn't settle it either. METR's developers overestimated their speedup, while Peng's participants estimated 35% against a measured 55.8%.
Far fewer studies test whether you can still do the work afterward.
Shen and Tamkin, both working with Anthropic, ran a randomized experiment in which 52 crowd-recruited developers with Python experience learned Trio, an async library they had not used. Both groups had web search and instructions, and one could also use a chat assistant running GPT-4o. In a quiz afterward, taken without assistance, the AI group scored 4.15 points lower out of 27 (Cohen's d = 0.738, p = 0.010). The largest gap was on debugging questions, and the AI group was not significantly faster. Within the AI group, the authors sorted 24 sessions into patterns: clusters that delegated the code or leaned on the AI to debug averaged under 40%, while clusters that asked conceptual questions or checked their understanding of generated code averaged 65% or higher. Those patterns are descriptive, with only two to seven people in each, so they suggest a direction and prove no cause. The study covers about an hour with one library, and participants had no job reason to learn it.
Kazemitabaar and colleagues reached a less worrying result in a very different setting. Across 10 sessions, 69 learners aged 10 to 17 with no text-based programming experience worked through Python tasks, half with Codex for the writing tasks. Each one was followed by a modification task done by hand. The Codex group scored 1.8 times higher on the writing tasks during training and performed no worse on the manual modifications. On tests without AI a week later it scored slightly higher (not statistically significant), though it also hit significantly more errors on writing tasks. Children learning first concepts tell you little about a professional learning a library, and the tools differ. Still, that design built something in: after generating code, every learner had to change it unaided.
That comparison can't bear much weight, since the studies differ in age, length, tool and task. Still, manual engagement with the generated code is the visible difference, in both the study design here and the high-scoring patterns there.
Within one AI group, clusters that delegated averaged under 40% on the quiz and clusters that worked to understand averaged 65% or higher, with two to seven people in each.
Perry and colleagues gave 47 participants, mostly university students, security related coding tasks, with 33 using an assistant built on codex-davinci-002. On a JavaScript task to insert a row, 36% of the assistant group wrote code open to SQL injection against 7% of the control group. Assistant users also rated their answers as more secure, and participants with secure solutions reported less trust in the AI than those with insecure ones. The caveats are substantial: the sample was mostly students, the model was an early one, group sizes were small, and the overall group effect in the regression came in at p = 0.056, just short of the usual threshold.
The failure is easy to reproduce by hand:
// Passes a demo with friendly input. Fails on O'Brien, and worse on hostile input.
var sql = $"INSERT INTO coupons (code, owner) VALUES ('{code}', '{owner}')";
// Same behavior on honest input, and the values stay data.
await using var command = new NpgsqlCommand(
"INSERT INTO coupons (code, owner) VALUES ($1, $2)", connection);
command.Parameters.AddWithValue(code);
command.Parameters.AddWithValue(owner);
await command.ExecuteNonQueryAsync();Against a disposable PostgreSQL, the first version failed with a syntax error for the owner O'Brien and the second
stored it correctly. Both look complete at a glance, as injection bugs tend to.
None of these habits was tested as a package, so weigh them as reasonable steps from the findings above.
When learning something new, ask for explanations before code. This follows the high-scoring pattern in Shen and Tamkin, who found conceptual questioners were also the fastest of the high scorers. Try your own version first and let the errors happen. That group hit many errors and resolved them itself, while the delegators met few.
Change generated code by hand before you accept it. Add a case or remove a branch without help. If you can't, you have found out what you don't yet understand, at a much cheaper time than during an incident.
Write the check yourself on code that can hurt you. Treat generated code as a pull request from someone capable who knows nothing about your system. The code review checklist applies unchanged, and writing your own boundary tests forces you to state the rule the code is supposed to follow. On security-sensitive paths, distrust your own confidence, since that is what Perry's participants got wrong.
Keep unassisted repetitions on the skills you need when the tool can't help. Debugging is the one to protect, given where Shen and Tamkin saw the largest gap, and production incident work depends on reading evidence about your own system. Deliberate practice means short, hard repetitions with feedback, and no study here tested it directly for AI users.
Since self-assessment was off in both METR's and Peng's data, borrow METR's idea at a tiny scale. Before each task, write down a time estimate and flip a coin to decide whether you may use AI, then record the actual time. After a few weeks, compare:
SELECT ai_allowed,
count(*) AS tasks,
percentile_cont(0.5) WITHIN GROUP (
ORDER BY actual_min::numeric / predicted_min
) AS median_actual_over_predicted
FROM task_log
GROUP BY ai_allowed;A few dozen tasks from one person will not produce a publishable effect, and a coin flip will annoy you on tasks you would rather delegate, which is exactly the selection problem METR ran into. It will still tell you more about your own work than your impression does, and it leaves out quality, so note rework too.
Shen and Tamkin's debugging gap can be tested on yourself without a study. "Two Amounts of Tax on One Line" gives you an invoice that disagrees with itself by a cent, and hidden tests judge the fix instead of your impression. The C# secure coding exercises put attack payloads against your code in the same way.
Practice what you just read
- Open in the editor: Two Amounts of Tax on One Line
Algorithm Easy Free, no account needed
Two Amounts of Tax on One Line
A customer rejected an invoice whose only line disagrees with its own total by a cent. How does one number come out as two?
- Open the Pro kata: Sort Column Allowlist
Secure coding Medium Katabench Pro
Sort Column Allowlist
Build an ORDER BY clause from a user-chosen column without opening a SQL injection hole.
More like this: C# coding challenges →