Can AI Help You Do a Task Without Teaching You the Skill?

What counts as learning when AI is part of the workflow?

A correct, fast or polished result produced with AI available shows that the person-and-tool workflow can complete a task. On its own, it does not show that the worker learned to do it independently. A learning claim needs a measure beyond assisted output, and what that measure means depends on when and how it is taken. A practical measurement ladder starts with assisted output: what quality or speed did the person achieve with the tool? Next is immediate unaided performance, tested after assistance stops. That can indicate whether the worker can reproduce recently used knowledge, but not whether it lasts. A delayed unaided test asks about retention; a new but related task asks about transfer. Performance during a real workflow interruption can show whether work continues when a tool is unavailable, though an interruption is not necessarily a controlled learning test. Consider a worker preparing a written analysis with AI. A strong report may reflect useful collaboration: the worker framed the question, checked evidence and edited the result. It cannot by itself tell us whether that worker could construct the analysis unaided. An immediate no-tool task would test near-term independent performance; a comparable task later would speak to retention; changing the subject while keeping the reasoning demand would probe transfer. “How AI assistance impacts the formation of coding skills” uses an immediate quiz after a short learning task, while “Shifting Work Patterns with Generative AI” reports workplace time-use outcomes rather than an independent skill test. The first addresses near-term mastery in its coding exercise; the second can show changed work patterns, not whether workers gained or lost skill. Reserve learning, mastery, retention and transfer for outcomes actually assessed, rather than inferring them from speed or completion. This question concerns task performance and underlying skill, not whether an occupation will disappear.

Sources: How AI assistance impacts the formation of coding skills; Shifting Work Patterns with Generative AI

What does the customer-support study establish about learning on the job?

The customer-support field study offers credible but bounded evidence that repeated AI recommendations can coincide with workers learning useful approaches. During periods when the system was unavailable, agents handled conversations faster after more exposure, with larger gains among those who had followed its suggestions. That pattern is consistent with learning; it does not prove a general effect or show transfer to other jobs. “Generative AI at Work” followed 5,172 customer-support agents during the staggered introduction of a conversational assistant. It supplied suggested replies and guidance from internal support materials as agents handled recurring customer conversations. This live-work setting differs from a one-session tutorial: agents saw repeated cases, while people remained responsible for conversations. The paper reports access raised issues resolved per hour by 15% on average, with larger gains for less experienced and lower-skilled agents. Those are productivity results while assistance was available, not by themselves evidence of independent skill. The learning argument comes from a separate analysis of technical interruptions. Outages sometimes prevented the system from giving recommendations. Researchers compared chat duration around these periods and found the apparent benefit without suggestions grew with prior exposure. After about three months, agents handled outage-period chats faster than their pre-tool baseline; the estimated advantage was smaller soon after adoption. Agents with high initial adherence to recommendations showed faster handling during outages over time, while the low-adherence group showed no comparable reduction. A plausible mechanism is that an agent who tries a suggested way to diagnose or explain a problem can observe the customer’s response and carry that approach into later conversations. But this was not a randomized test of learning. Although rollout was staggered, outage timing was not a randomized lesson assigning some agents a skill and withholding it from others. Outages were occasional failures, lasted from minutes to hours, and did not always affect every worker. The paper notes they were rare and not necessarily random; chats during them may have differed. For this analysis, the outcome was chat duration because issue-resolution data were unavailable at that level. Faster handling without suggestions indicates changed performance, but does not directly test whether agents could explain the method, retain it later, or apply it to a new task. The result challenges a blanket claim that AI assistance necessarily prevents workplace learning. Repeated recommendations may expose workers to approaches they see less often in limited coaching, and the study found faster performance during some periods without the tool. Still, it covers one support operation and conversational workflow, using observational outage analysis rather than a purpose-built test of delayed mastery or transfer. “Generative AI at Work” supports calling the learning evidence suggestive, not concluding every agent acquired durable skill. For workers, the implication is conditional. Assistance may teach when it reveals useful examples and the worker remains engaged enough to try them and notice what happens. That is a reasoned interpretation, not a tested recipe for every workplace. If independent handling, explanation, or recovery matters in your role, the study gives a reason to preserve practice in those parts. If an approved supported workflow completes and checks the task reliably, this study alone does not establish a need to perform it unaided. It shows learning can accompany AI-supported work in one real setting, not how often this happens elsewhere or whether learning transfers.

Sources: Generative AI at Work; Generative AI at Work

What does a randomized coding task reveal—and what does it leave open?

The coding study supports a narrower warning: in one brief lesson about an unfamiliar Python library, participants assigned AI assistance scored lower on an immediate quiz than those who coded by hand. It does not show that AI permanently reduces developers’ skill or that all workplace assistance has the same effect. The distinction matters because finishing a task and learning how to do it are separate outcomes. The study tested what participants understood just minutes after practice, not whether they retained the material weeks later or could apply it in a different codebase.

In “How AI assistance impacts the formation of coding skills,” 52 mostly junior software engineers, each familiar with Python but unfamiliar with the Trio library, were randomly assigned to work with or without an AI assistant. They completed two short coding features using the library, then took a quiz covering debugging, reading code, writing code and core concepts. The AI group averaged 50% on the quiz, compared with 67% for the hand-coding group. The researchers describe the difference as statistically significant. The AI group finished about two minutes sooner on average, but that time difference was not statistically significant. So the experiment found a short-term mastery difference in this setup; it did not establish a dependable productivity gain alongside it.

Random assignment gives the comparison useful leverage: for this task and sample, the groups’ different quiz results can be more plausibly attributed to the assigned assistance condition than a simple comparison between people who choose different tools. But that causal reach is narrow. The sample was small, the work was a self-guided tutorial rather than an ongoing job assignment, the skill was a new library, and the test came immediately afterward. The result cannot tell us whether repeated use over months would widen, shrink or reverse the difference, whether the same pattern appears in familiar work, or whether it transfers to other occupations. The paper itself says that longer-term skill development remains unresolved.

A plausible explanation is that the kind of practice differed. When a learner asks a system to generate code or solve a debugging problem, the learner may perform fewer of the operations later tested: tracing code, locating an error and explaining why a solution works. The study’s screen-recording analysis found that some lower-scoring interaction patterns involved heavy delegation or relying on AI to debug. Higher-scoring patterns included asking conceptual questions and seeking explanations. Those patterns are clues about a possible learning pathway, not proof that a particular way of prompting causes better mastery: participants were not randomly assigned to those interaction styles, and the researchers explicitly say their qualitative analysis does not establish a causal link.

That limitation also keeps the finding useful without turning it into a rule against assistance. A short tutorial asks someone to acquire a new skill under time pressure. Repeated workplace tasks may provide further attempts, feedback and varied cases; they may also leave less time for deliberate practice. The coding experiment did not compare those conditions, so neither possibility can be claimed as its result. For work where a person must later explain, verify or debug code without help, a successful assisted output is not enough evidence of mastery. The proportionate conclusion is to distinguish immediate, unaided understanding from durable skill change, and to avoid calling one brief quiz a forecast of long-term deskilling.

Sources: How AI assistance impacts the formation of coding skills; How AI Assistance Affects Skill Formation in Coding: An Experiment

Why productivity experiments often cannot answer a skill question

A workplace trial can show that access to an AI tool changes time use or output without showing whether workers gain, retain or lose the ability to do the work unaided. To answer a skill question, researchers need to measure skill, such as by giving workers a comparable task without the tool and assessing the result against a stated standard. If a study measures only performance while assistance is available, its result belongs to the combined human-tool workflow. It cannot establish what the worker could do after assistance is removed. “Shifting Work Patterns with Generative AI” illustrates this boundary. The American Economic Review: Insights abstract describes a randomized field experiment across 66 firms and 7,137 knowledge workers over six months. Workers were randomly selected to receive access to a generative AI tool integrated into their email, meeting and writing applications. In the second half of the experiment, the 80% of treated workers who used the tool spent about two fewer hours per week on email and reduced work outside regular hours. The study reports no detected shift in the quantity or composition of workers’ tasks from individual-level AI provision. These are findings about access, use and work patterns. The abstract does not report an unaided skills test, delayed retention measure or transfer test, so it cannot tell us whether underlying skills improved or weakened. Random assignment matters, but its reach is specific. It helps estimate the effect of being offered access on outcomes. It does not turn an unmeasured outcome into evidence. Less time on email might reflect faster drafting, fewer messages, changed habits or a combination; without a skills measure, the experiment does not identify independent email-writing ability. Assignment to access and actual use are different events, as are individual use and organization-wide implementation. Saved time is not a direct measure of labor demand or displacement. The reported time-use outcome does not show whether employers increased output, reassigned work, reduced staffing or changed hiring. Those questions require separate evidence. Likewise, no detected task-composition shift here does not show that workplace redesign occurs nowhere else; it bounds what this study detected under its design and period. The practical research rule is to match the claim to the outcome measured. Cite this experiment as evidence that AI access changed some workers’ work patterns over six months. Do not cite it as proof of skill acquisition, deskilling or durable productivity because independent performance was not tested. For workers, a productivity dashboard can describe a supported workflow; it cannot substitute for a no-tool assessment when independent competence matters. Silence about skills is neither evidence that workers learned nor evidence that they forgot.

Sources: Shifting Work Patterns with Generative AI; Shifting Work Patterns with Generative AI

What changes when a study tests ability after AI is removed?

A study that tests people after AI is removed asks a more direct question about independent capability than one that records only the quality or speed of AI-supported work. It can show whether performance on a subsequent unaided task differs from performance with assistance available. Recent controlled reasoning experiments report that these outcomes can diverge. These tasks are not evidence about sustained workplace learning, and one post-assistance test cannot establish durable mastery. The 2026 preprint “How AI Assistance Affects Human Skill Development: A Study of Learning with Logic Puzzles” used a before, during and after design: participants completed logic puzzles before AI access, while on-demand assistance was available, and after it was removed. The researchers experimentally varied the cost of requesting help. Lower request costs led to more frequent use, and participants who requested help performed worse on the task after assistance ended than their earlier assisted performance would suggest. A Bayesian latent-ability model separated initial ability, post-AI ability and participant-specific change. Greater independent problem-solving was associated with larger estimated ability gains. That pattern makes a useful distinction visible: doing well with help and being able to do well alone are separate outcomes. The model estimates skill change; it does not observe general workplace skill. In addition, the association between independent effort and estimated gains does not by itself prove that effort caused the difference. People who chose to reason more independently may have differed in other ways. The task was controlled logic-puzzle solving, not recurring paid work, and delayed retention or transfer remain untested. Its result supports testing after removal; it does not settle how workers learn across months of AI-supported practice. A second 2026 preprint, “AI Assistance Reduces Persistence and Hurts Independent Performance,” reports a series of randomized human-AI interaction trials with 1,222 participants. Across controlled tasks including mathematical reasoning and reading comprehension, the researchers report that AI access improved short-term performance, while participants performed worse without it and were more likely to give up. The report says these effects appeared after brief interactions of about ten minutes. Randomized conditions and a no-help assessment add evidence that assisted success need not carry over to an unaided attempt. But the sample and tasks do not represent employees learning their jobs, Brief exposure cannot establish cumulative dependence, long-term retention or workplace transfer. The authors' explanation about expectations of instant answers is an interpretation of the findings, not a demonstrated account of long-run worker behavior. These removal designs answer something the coding study's immediate quiz answers differently. In “How AI assistance impacts the formation of coding skills,” software developers learned an unfamiliar Python library in a short randomized task and then took a quiz minutes later. The AI group scored lower on that immediate assessment than the hand-coding group. That is evidence about near-term mastery after practice, not delayed retention; the logic-puzzle and arithmetic/reading studies instead examine performance after assistance is withdrawn. Meanwhile, the customer-support study's temporary system interruptions occurred during real work, but outages were not randomized skill tests. Each measures a different point between supported performance and independent capability. The practical boundary is time horizon. A post-removal result informs unaided performance more directly than assisted output, yet an immediate test cannot show whether skill persists, transfers or matters in a role. Conclusions should match the measurement: these experiments raise a task-specific question about what happens when help disappears, not a general verdict on workplace learning.

Sources: How AI Assistance Affects Human Skill Development: A Study of Learning with Logic Puzzles; AI Assistance Reduces Persistence and Hurts Independent Performance; How AI assistance impacts the formation of coding skills

Why the way assistance is used may change the result

The evidence supports a conditional account: assistance that substitutes for the reasoning being learned may leave fewer chances to practice it, while informative support alongside independent reasoning can coincide with stronger short-run learning. Neither result establishes a universally effective prompt or workflow. The key question is what mental work remains the learner’s and what a later no-assistance measure can show. In “How AI assistance impacts the formation of coding skills,” a randomized study assigned 52 mostly junior software developers to learn an unfamiliar Python library with or without an AI assistant. The AI group scored lower on a quiz given minutes after the task: averages were 50% with AI and 67% with hand coding. Random assignment supports a causal conclusion about this brief learning task and immediate measure. It does not show that every kind of AI use impairs learning, or that the difference lasts. Researchers grouped participants by interaction pattern. Some lower-scoring patterns involved delegating code writing or debugging; higher-scoring patterns included asking conceptual questions, requesting explanations, or checking understanding after generated code. These are clues, not randomized comparisons: participants chose how to interact, so the analysis cannot establish that a query style caused better mastery. A controlled logical-reasoning experiment, “The Impact of AI Usage and Informativeness on Skill Development in Logical Reasoning,” made assistance quality part of the comparison. Its abstract reports that high-information AI improved short-run performance without reducing post-AI outcomes on average, though effects varied. Low-information assistance neither improved immediate performance nor preserved performance after removal, and was linked to weaker learning. The study also reports heavier use associated with weaker skill development, while light users performed similarly to matched non-users. This was a reasoning task, not a trial of employees learning their jobs. Associations between a participant’s use and skill development do not by themselves show that reducing use would cause an individual worker to learn more. These findings remain preliminary and task-specific. Together, the studies suggest a plausible mechanism: practice depends on whether the learner retrieves, attempts, explains, checks, and corrects the reasoning they hope to acquire. If assistance supplies a finished answer without making its basis clear, supported performance can improve while the worker does less of the underlying operation. If support provides useful information while leaving room to reason independently, learning may be preserved in some settings. This is an interpretation across studies, not a universal law or tested workplace recipe. For a skill needed in later independent work, make the interaction more like practice than delegation: form a prediction before asking, request an explanation when needed, and check the result against evidence or a known standard. Then test learning with a later unaided attempt at a comparable task. “How AI Assistance Affects Human Skill Development: A Study of Learning with Logic Puzzles” measured task performance before, during, and after access; its abstract reports that assisted performance overestimated later unassisted performance. That preprint supports measuring after removal, but puzzle results do not validate a particular work routine. The practical test is whether the worker can still perform, explain, or catch an error without help when the role requires it.

Sources: The Impact of AI Usage and Informativeness on Skill Development in Logical Reasoning; How AI assistance impacts the formation of coding skills; How AI Assistance Affects Human Skill Development: A Study of Learning with Logic Puzzles

How does a skill question differ from a job-change question?

Whether AI assistance teaches a task skill is separate from whether it changes a job. A learning study asks what a worker can do after support is removed. A job-change analysis must also ask whether an employer adopts the system, which tasks are reassigned, and whether output or staffing changes. Unaided performance is not a probability of job loss. The 2023 OECD report “The impact of AI on the workplace: Evidence from OECD case studies of AI implementation” draws on nearly 100 qualitative case studies of AI technologies in manufacturing and finance across eight OECD countries. It reports that job reorganization appeared more prevalent than job displacement in these cases, with automation prompting jobs to be reoriented toward tasks where people had a comparative advantage. The report also describes varied effects on skill requirements and job quality, including reports of increased work intensity. These cases show that implementation can change tasks and working conditions; they are not a representative estimate of generative AI use or a forecast for any occupation. The report does not test whether assistance teaches a skill. Consider a routine task where AI drafts a summary and a worker checks source material and handles exceptions. The system may change how the first draft is produced, while the remaining work depends on verification. Whether the worker learned to draft better is one question; whether the role’s task mix changed is another. The OECD cases show that implementation choices matter, but cannot predict an employer’s decision. “Shifting Work Patterns with Generative AI,” published in American Economic Review: Insights, studied 7,137 knowledge workers across 66 firms over six months. Workers were randomly selected for access to a tool integrated into applications used for email, meetings and writing. In the second half, the 80% of treated workers who used it spent about two fewer hours per week on email and reduced work outside regular hours. The researchers did not detect shifts in task quantity or composition from individual-level AI provision. These are work-pattern findings, not measures of unaided skill, retention or transfer. Random assignment strengthens conclusions about measured outcomes, but cannot answer an unmeasured learning question. Email time also does not establish a change in labor demand or staffing. For a worker, ask what competence a task requires independently. Verification, explaining a decision or recovering when a system is unavailable may remain consequential even when AI assists routine production. If quality can be maintained through a permitted, verified workflow, AI’s ability to perform part of a task alone does not justify broad retraining. A learning choice must also fit salary needs, location, credentials, available time, health and family responsibilities. Use skill evidence to decide whether focused practice is warranted, and separate evidence about employer implementation and labor demand to assess job change. No single signal shows that an occupation is safe or doomed, and none yields a personal job-loss probability. Keep the questions distinct.

Sources: The impact of AI on the workplace: Evidence from OECD case studies of AI implementation; Shifting Work Patterns with Generative AI

How should a worker check whether independent skill matters?

Choose one recurring task and ask what the work requires when assistance is unavailable or wrong. Compare an assisted result with a later similar attempt completed without AI, using the same quality criteria. This checks whether independent performance matters and whether a gap needs attention. Imagine a worker who uses AI to prepare a short analysis. They note what makes it usable: figures reconcile, conclusions follow evidence, and caveats are clear. They record what the tool supplied, what needed correction and the effort verification took. On a later request of similar difficulty, they try the relevant steps unaided. Can they explain the conclusion or spot an inconsistency? Does the work meet the same standard? This is an illustration, not a study or validated assessment. Keep the comparison modest. Tasks may differ in difficulty, information or time pressure, so a changed result cannot identify its cause. Choose similar cases and criteria in advance. One attempt can suggest a question for practice; it cannot establish durable mastery or show that AI caused a change. “How AI assistance impacts the formation of coding skills” describes an immediate quiz after a short assisted or unaided lesson. “How AI Assistance Affects Human Skill Development: A Study of Learning with Logic Puzzles” measures performance after access is removed. Both make independent performance visible, but brief, task-specific measures do not validate an informal comparison or establish long-term transfer. Success with a tool and later unaided performance are separate observations. Interpret a gap against the task’s requirements. If rules permit assistance, outputs can be checked reliably, and the worker is accountable for the verified result, supported performance may meet the need. Independent skill matters more when the worker must make a decision, explain its basis, catch an error, review output, or keep work moving during an interruption. They do not mean every task requires full independent mastery. If the comparison reveals a consequential weakness, choose a narrow response: repeat the step unaided, ask for feedback on the reasoning, or seek focused instruction. Fit the choice to available time, cost and responsibilities. A course or credential is not the default answer to one task-level gap, and one exercise is not a reason to change careers. If independent performance is not part of the work standard, evaluate the whole workflow, including verification effort, instead of creating a training burden without a work need. The next move should match the gap and the worker’s constraints. This small check can support continued tool use, protected practice or focused learning; it cannot estimate displacement or job-loss risk. After answering the skill question, a worker seeking broader task-level change signals can use the free checker at `/ai-job-risk-checker`. Its result is a transparent change-pressure signal, not a validated probability that a person will lose a job.

Sources: How AI assistance impacts the formation of coding skills; How AI Assistance Affects Human Skill Development: A Study of Learning with Logic Puzzles

What is the proportionate next step?

Do not infer skill growth from assisted output or skill loss from AI use alone. Match the evidence to the task, then check unaided performance only where it has a real purpose. The customer-support field study, Generative AI at Work, offers suggestive evidence that workers can learn through repeated recommendations,. But the randomized coding study, How AI assistance impacts the formation of coding skills, found lower immediate quiz mastery after one brief AI-assisted lesson. The controlled logic-puzzle study, How AI Assistance Affects Human Skill Development: A Study of Learning with Logic Puzzles, reports that assisted performance did not reliably predict performance after assistance ended. No study establishes a universal rule about AI and learning.

For one recurring task, record its quality standard and what must be independently checked. Compare an assisted result with a later similar attempt completed without assistance, using the same standard. This is not a validated assessment. If a consequential gap appears, target practice or instruction at that skill. Productivity trials without skill tests cannot settle it. For broader task change, the free task checker provides task-level change-pressure signals, not a validated probability of displacement. Consider the paid roadmap only when comparing broader paths against your constraints; it compares scenarios without guaranteeing outcomes. Firmer conclusions will require longer workplace trials testing delayed, transferable unaided skills.

Sources: Generative AI at Work; How AI assistance impacts the formation of coding skills; How AI Assistance Affects Human Skill Development: A Study of Learning with Logic Puzzles

Questions readers ask