You Don't Learn What You Delegate

A rare thing: a pre-registered randomized controlled trial of how AI assistance changes what workers learn. Shen and Tamkin (Anthropic), How AI Impacts Skill Formation. Fifty-two professional and freelance Python programmers were randomized, asked to learn a library they had never used (Trio, an async I/O library), complete two coding tasks, and then take a quiz on what they actually learned. One group had a GPT-4o chat assistant that could write the entire correct solution; the other had no AI at all.

The headline, in two numbers: the AI group scored 17% lower on the quiz — about two grade points (Cohen's d = 0.738, p = 0.010) — and gained no significant time in return. The productivity miracle measured in prior work (55.5% faster with Copilot; 26.8% more pull requests) did not show up when the task required learning. That is the paper's contribution, and the thesis of this post: AI assistance doesn't just complete the task — it removes the part of the task that teaches. You don't learn what you delegate, and the trial shows exactly which behaviors turn delegation into learning and which into a 17% toll.

The experiment

Participants made this a hard test of "AI helps novices most": more than a year of Python, weekly use, prior AI experience, never Trio (Fig 17). A warm-up task, then 35 minutes for two Trio tasks: a concurrent timer and a record-retrieval function with error handling. Then a 14-question, 27-point quiz over 7 Trio concepts in three skill types — conceptual understanding, code reading, debugging (Fig 20); code-writing was excluded, since syntax is what AI fixes cheapest.

Fig 3 - Experiment interface with AI assistant panel

Fig 4 - Learning task and comprehension check flow

The treatment assistant was no toy: GPT-4o with access to the participant's code, able to produce the full correct solution to both tasks. Four pilot studies tuned the design — the first platform had 35% non-compliance (controls using AI anyway); a later pilot (Fig 5) showed the effect so strongly (d=1.7) that the pre-registration assumed a conservative d=0.85 for power. Integrity controls included pledges and screen recordings of every participant (Figs 21–24; Figs 25–27 show both platforms).

Results: the 17% toll

The main results (Fig 6): a 4.15-point gap on 27 points — 17%, two grade points — with no significant difference in completion time; the effect survives controlling for warm-up speed (d=0.725, p=0.016). Fig 7 shows the control group scoring higher at every experience level, and Fig 8 shows where the damage concentrates: the largest gap is in debugging, the smallest in code reading. Both groups read code; only the control group had to fix it.

Fig 6 - Main study: task time and quiz score (n=52)

Fig 8 - Score breakdown by question type

That debugging gap is the paper's most important sentence in disguise. The industry case for human oversight of AI-written code rests on humans being able to debug what AI produces — and this trial suggests the mechanism that builds that skill is exactly what AI assistance removes.

Why there was no speedup

The qualitative analysis explains the missing speedup: heterogeneity. Watching every screen recording, the researchers found six AI-interaction personas (Fig 11), with completion times and quiz scores that barely overlap:

AI Delegation (n=4) — handed the whole task to the AI, finished fastest (19.5 vs 23 min), scored 24–39%.
Progressive AI Reliance (n=4) — started with a question, then delegated everything; lost the second task's concepts entirely.
Iterative AI Debugging (n=4) — used AI to fix code without understanding it; slowest of the low scorers.
Generation-Then-Comprehension (n=2) — generated code, then asked follow-ups to understand it. 65–86%.
Hybrid Code-Explanation (n=3) — asked for code with explanations, and read them.
Conceptual Inquiry (n=7) — asked only conceptual questions, hit errors, resolved them independently. Second-fastest overall.

Fig 11 - The 6 AI interaction personas with outcomes

The gap between worst and best usage is the real story: 24–39% vs 65–86% on the same quiz with the same assistant. The difference is not whether AI was used; it is what happened after the AI answered. The time side confirms it: some participants spent up to 11 minutes composing queries and asked 15 questions (Fig 12); debugging-heavy query mixes correlated with slower times and lower scores (Figs 18–19). And the paste-vs-type analysis (Fig 13) has a twist: pasting AI output was fastest, manually copying it was as slow as control — but the two scored the same on the quiz. Spending time typing doesn't build understanding.

Fig 13 - Pasting vs manual copying of AI output

Errors are the curriculum

The mechanism the paper lands on is the most old-school finding in it: errors teach. The median control participant hit three errors; the median AI participant hit one (Fig 15). Of the twelve error-free completers, only two were control. The errors that matter are the Trio-specific ones — RuntimeWarning (a coroutine never awaited), TypeError (a coroutine passed where an async function was expected) — because they force exactly the conceptual knowledge the quiz tests (Fig 14). The control group did not learn despite the errors; the errors were the lesson plan. Active coding time tells the same story from the other side (Fig 16): AI shifted time from writing to reading AI output. The AI group's own feedback confirms it — they felt "lazy" with "gaps in (their) understanding."

Fig 15 - Errors by condition: control meets the concepts

What this means

Three implications, stated as plainly as the paper allows.

First, the chat interface is the best case. A chat assistant at least forces the user to compose a query — some spent six minutes on a single one, and that thinking correlates with learning. An agentic tool that writes, runs, and fixes code itself removes even that. If this trial is the lower bound on cognitive offloading, agentic settings are below the floor.

Second, the supervision argument inverts. "AI writes code, humans verify" assumes humans have debugging skill to spend. This trial suggests that skill is built by the exact experience AI removes. Companies adopting AI-assisted onboarding for juniors are not just changing throughput; they may be choosing which generation holds the verification skill.

Third, the dark-factory argument applies to skills, not just complexity. When automation removed factory labor, complexity moved into supervision. When AI removes implementation, the question is where the learning goes — and this trial's answer is: out of the worker, unless the worker keeps engaging. Delegation is a productivity strategy and a learning strategy simultaneously; you cannot have both.

The six personas are the practical deliverable: the difference between treating the model as a chauffeur and as a tutor.

The productivity view asks: how much did AI finish? The skill view asks: what did you keep? This trial shows the two questions now have different answers. The 17% is what you pay when you confuse them.