Can AI Support Results Generalize to Experienced Workers?

Can customer-support results generalize to experienced workers elsewhere?

Only in a limited sense: customer-support results are a reason to expect AI effects to vary across tasks and workers, not a numerical forecast or a seniority rule for another occupation. For an experienced knowledge worker, the useful question is whether the actual task, tool, starting performance, required quality and surrounding workflow resemble the measured setting closely enough to learn anything from it.

The temptation to transfer a vivid average—or a novice-versus-expert split—is understandable, especially when the study observed real work at scale. But the same label, such as “knowledge work,” can cover different inputs, judgment, exceptions, checking and consequences of error. The article therefore examines what each result measured before drawing a comparison. Measured throughput or output in an experiment describes work under those conditions; it does not establish what will happen to another person's role, employment or income. Experience remains useful context for framing and checking work, but it cannot by itself predict whether assistance helps a different task.

What does the customer-support result actually measure?

In “Generative AI at Work,” the reported average is a throughput measure: issues resolved per hour, not a general measure of a worker’s total productivity or employment prospects. The study followed a staggered introduction of a conversational assistant in one customer-support operation, covering 5,172 agents. Across the observed operation, access was associated with an average increase of about 15% in issues resolved per hour. That is a meaningful field result because it concerns an implemented tool in ordinary support work, rather than performance on a detached laboratory prompt. Its unit, however, remains specific: completed support issues per hour in that system.

The average also conceals a split. The study reports that less experienced or lower-skilled agents improved both speed and quality, while the most experienced or highest-skilled agents recorded small speed gains alongside small declines in quality. This is evidence of variation within the studied workforce, not evidence that years of experience invariably reduce the value of AI. The source also reports that gains were larger for agents working on less familiar issues, a reminder that prior familiarity with the particular problem can differ from general seniority. The interaction between an assistant and a worker’s starting point and case mix matters to how the average should be read.

Throughput and answer quality are separate outcomes here. Resolving more issues per hour does not, by itself, mean that each answer became more accurate, complete or suitable; the study measured quality separately and found different patterns across agent groups. Nor does an observed quality change tell us whether downstream handling, customer outcomes or staffing changed. Those are further questions requiring their own measures. For a worker in another field, the study offers a useful comparison only after checking whether the task has similarly structured cases, accessible information, an output that can be checked and a workflow in which the tool can be used. It supplies no transferable productivity percentage for work that differs on those features.

A large sample and a live workplace make this stronger evidence about that support operation than a small, artificial exercise. They do not make the operation representative of every knowledge job: one employer’s tools, task mix, escalation rules and quality standards shape what its result means. The defensible lesson is to preserve the distinction between the measured rate and the broader job, and to ask what changed for which agents on which cases.

Sources: Generative AI at Work

How do developer and writing trials measure different gains?

The software and writing experiments broaden the evidence because they test AI assistance on different kinds of knowledge work, but their reported effects are not directly comparable with each other or with support throughput. Each study defines a different unit of success, uses a different population and asks a different question. A percentage or standardized effect from one design cannot be read as a common measure of how much more productive a worker becomes across occupations.

In “The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers,” researchers randomized access to a code-completion assistant across three company experiments involving 4,867 developers. The combined estimate was 26.08% more completed tasks for developers offered access, with a standard error of 10.3%; reported gains were greater among less experienced developers, who also adopted the tool more. The outcome was a count of completed tasks in those firms, not minutes saved per task or an independently harmonized quality score. Because the three experiments varied, the combined estimate summarizes those settings rather than establishing a fixed return for software work generally. The abstract supports the field setting and task-count result, but not a claim that all counted tasks were equally difficult, equally valuable or completed to a common quality standard across other occupations.

“Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence,” the MIT working paper, studied 444 college-educated professionals doing occupation-specific writing assignments online. Participants were randomly given ChatGPT access or not; the assignments, designed to resemble work in areas such as marketing, grant writing, consulting, analysis, human resources and management, generally took 20 to 30 minutes. The paper reports that access reduced task time by 0.8 standard deviations and increased blinded evaluator-rated quality by 0.4 standard deviations. Those are two distinct measured outcomes: elapsed time and ratings from experienced professionals who did not know the treatment condition. The study also reports larger gains for people with lower baseline performance. That is a comparison by starting performance on the assigned work, not a direct test that junior employees benefit more than senior ones. The document is a March 2023 working paper marked as not peer reviewed, and its short, incentivized tasks do not show whether gains persisted in ordinary jobs.

The contrast with the support study is useful precisely because it prevents a false ranking. That study counted issues resolved per hour in one operating support workflow; the developer experiments counted completed software tasks, while the MIT experiment separately timed assignments and rated their outputs. One measure cannot be converted into another from the reported evidence. A larger task count does not mean the same thing as faster completion, and a shorter completion time does not on its own establish the same quality or downstream value as a completed task count. The MIT quality ratings add information about the writing outputs under its evaluation procedure, but they do not create a common quality-adjusted productivity unit across the three studies.

Random assignment in real companies and realistic professional assignments gives these studies stronger causal evidence about access to assistance on the tested tasks than anecdotes or demonstrations. Yet their time horizons and work settings differ: company field experiments observed software-task completion in participating firms, while the writing experiment used brief online assignments rather than embedded work across a normal operating cycle. Neither tells us how an employer later changes responsibilities, staffing or the balance between production and review. Read together, the trials support a narrower conclusion: measured gains can appear in distinct kinds of work, but the size and meaning of each result depend on what was counted, who performed it and how the work was evaluated.

Sources: The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers; Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence

Does assisted performance tell us whether expertise transfers?

Assisted performance and unaided judgment are separate outcomes. “Does AI Assistance Enhance or Erode Expertise? Evidence from a Three-Month Field Experiment in Patent Drafting,” recorded on Google Research, adds a transfer question by measuring lawyers’ work with a custom assistant and then testing a later task without assistance. Its evidence concerns a particular patent-drafting workflow; it does not establish that AI generally teaches expertise or that assisted output predicts unaided capability in another profession.

The field experiment involved 133 practicing patent lawyers at 11 U.S. intellectual-property firms over three months. In the assisted drafting benchmarks, the study found higher quality with AI at both 10 and 90 days, with larger gains for junior lawyers. This repeated assisted-work result shows that performance differences were observed beyond a single initial encounter, within the trial’s specialized task and tool. It is still a measure of the quality of work produced with assistance at those test points, not a direct measure of what participants learned, retained or could do independently. An improvement while using a tool may reflect useful division of labor, improved access to relevant material, or other features of the workflow; the benchmark alone cannot distinguish these possibilities as general processes of skill acquisition.

Later, participants completed an unaided redlining task, and the reported pattern differed: senior lawyers showed gains while junior lawyers had no average gain. That result matters because it breaks the assumption that the group receiving larger assisted-work gains must also show the clearest subsequent unaided improvement. It makes transfer an empirical question of its own. The seniority pattern across these two outcomes cannot be collapsed into one statement that the tool either strengthened or weakened expertise: aided drafting quality and unaided redlining performance were measured under different conditions and at different points in the study. The result is specific to those tested tasks and should not be treated as a general seniority effect.

A three-month field trial and a later unaided task provide more relevant evidence about possible transfer than a one-off prompt exercise. But the population, custom assistant, legal materials and patent workflow remain unusually specific. The results do not establish broad skill formation, loss of expertise, or a reliable prediction for an individual lawyer. For other work, the key distinction is practical: first ask whether assistance changes the quality of the deliverable while it is in use; then, if expertise transfer matters, examine unaided performance separately over time.

Sources: Does AI Assistance Enhance or Erode Expertise? Evidence from a Three-Month Field Experiment in Patent Drafting

What did METR's early-2025 slowdown establish?

METR’s “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” found that its participants took 19% longer on AI-allowed issues than on issues where AI was disallowed. The estimate came from a randomized comparison within a small group of 16 experienced open-source developers, who worked on 246 real issues in repositories they knew well. METR reported a confidence interval from 2% to 39% longer. The result is a consequential counterexample to the idea that expertise automatically makes AI assistance faster: in this particular setting, experienced contributors expected to save time, yet the measured completion time went the other way.

The setting matters to what “completion” meant. The issues averaged roughly two hours of work and came from mature codebases familiar to the contributors, rather than isolated programming exercises with a short answer key. Issues were selected by participants and randomized to AI-allowed or AI-disallowed conditions, allowing each developer’s task times to be compared across both conditions. The study’s completion standard included review-oriented requirements, so the outcome was not simply how quickly someone could produce a plausible first patch. METR collected screen recordings and relied on developers’ reported implementation time, connecting the estimate to a work process that included making a change suitable for the repository’s expectations. Familiarity may reduce the time needed to learn a codebase, but it does not remove the integration and validation demands of changing established software.

The available tools also mark this as a dated result. METR says the February–June 2025 trial used tools primarily through Cursor Pro with Claude 3.5 or 3.7 Sonnet. It therefore describes that tool period, these contributors and their familiar repositories; it cannot establish how a later model or interface performs on the same work. Nor does the comparison identify why AI-allowed work took longer. Given the task setup, added time spent establishing context, integrating suggestions, running tests or checking a patch against review requirements is a plausible interpretation, but METR’s measured result does not prove any one of those as the cause.

This is stronger evidence about a narrow kind of experienced developer work than a benchmark demonstration, because it randomized real issues and judged completion against repository-oriented requirements. Its small, specialized sample still cannot settle performance for other occupations, developers unfamiliar with the codebase or later tools. The useful conclusion is conditional: expertise did not guarantee a speedup in this early-2025 workflow, while the experiment does not establish that experienced workers generally lose time with AI.

Sources: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

What changes in METR's later update?

In its February 24, 2026 update, “We are Changing our Developer Productivity Experiment Design,” METR reported later raw time estimates that point toward possible speedup among experienced open-source contributors. For the 10 developers returning from the earlier experiment, the estimate was 18% less time, with a 95% confidence interval ranging from 38% less to 9% more. For newly recruited developers, it was 4% less time, with an interval from 15% less to 9% more. Those ranges include no change, and METR explicitly judged the later data an unreliable signal, providing only very weak evidence about the magnitude of any real increase in speed. The point estimates should therefore not be read as established effects.

The update describes an experiment begun in August 2025 with 57 experienced contributors, working across 143 repositories and more than 800 tasks using later tools. That broader pool and METR’s conversations with participants make a genuine increase in speed plausible to the researchers. But who joined the experiment changed what could be measured: developers who were unwilling to work without AI were less likely to volunteer. METR also reports that 30–50% of participants said they withheld some tasks they did not want assigned to an AI-disallowed condition. The resulting task set was not simply a fresh, representative sample of the work developers would otherwise have done.

Pay and task comparability add further limits. METR says the later experiment paid $50 per hour, compared with $150 per hour in the earlier study; task types and quality changed as well. Some work assigned to the AI-disallowed arm was not completed. That creates a difficult comparison: if tasks that participants preferred to do with AI were withheld or left unfinished without it, the observed tasks and completed outcomes can differ by condition in ways that complicate the time estimate. The report’s concern is not merely that the new estimate is imprecise, but that selection into the study and into each task condition may distort what the raw difference represents.

Time measurement was also harder when developers used concurrent agents. When several agents worked at once, the researchers could not consistently determine how much human time belonged to each task; some relevant effort could be obscured by simultaneous work. This matters because a task-time estimate is only interpretable if the recorded time tracks the work being compared. METR treats these selection and measurement problems as substantial enough that readers should give its own “very weak evidence” judgment priority over the apparent size of either raw estimate.

The update changes how the 2025 slowdown should be used: it remains evidence about one early-2025 tool snapshot and task design, not a timeless counterexample to speed gains. But the later result does not show that the slowdown reversed, or cleanly measure a before-and-after tool effect. The tool vintage, volunteers, repositories, task selection, pay and time-recording conditions all changed. A real increase is plausible, as METR’s interviews and later raw results suggest; how large it is remains unresolved by this design.

Sources: We are Changing our Developer Productivity Experiment Design

Why can experience matter differently from task to task?

Novice catch-up and expert advantage can both be plausible because they describe different starting points and different parts of a task. The OECD review, “The Effects of Generative AI on Productivity, Innovation and Entrepreneurship,” frames results as dependent on both task properties and user experience, rather than one stable return attached to seniority. Read across the evidence discussed here, a person’s years in a profession, their initial performance on a particular task, their familiarity with its subject matter, and their ability to judge a proposed output are related but distinct characteristics. None is a reliable substitute for the others.

A lower starting performance can leave more room for improvement when the task is bounded, instructions are clear and outputs can be compared with a known standard. Assistance may supply structure, examples or a first pass that helps someone close part of an observed performance gap. That interpretation is consistent with novice catch-up patterns in some studied settings, but it is not proof that lack of experience causes larger benefits. A low baseline can reflect unfamiliarity with that task, limited access to useful methods, or other differences; the observed pattern alone does not identify which mechanism produced it. Nor does a stronger relative improvement necessarily mean the person reaches expert judgment or can handle exceptions independently.

Experienced task knowledge can matter at other points in the workflow. It may help a worker specify what a good answer must contain, notice when a fluent suggestion conflicts with context, choose what to verify, and take responsibility for how the result is used. These are plausible routes by which expertise could make assistance more useful, especially when the task contains exceptions or costly errors. They are an interpretation across distinct studies and OECD’s task-and-user framing, not a universal mechanism demonstrated by one experiment. They also depend on the expertise matching the task: years in a field cannot automatically help evaluate unfamiliar material or an output whose errors are hard to see.

There is a countervailing possibility. An expert may already complete a familiar task efficiently, leaving less routine time for a tool to save; additional review can then outweigh a faster first pass. Yet that same efficient baseline may include tacit checks that make the expert better at using a suggestion safely, or at recognizing when it should be rejected. The evidence permits both mechanisms and does not rank them for a particular worker. A useful comparison therefore records the specific task, starting quality, familiarity and review ability separately, then checks whether assistance changes accepted work and effort under those conditions. Seniority alone cannot answer that question.

Sources: The Effects of Generative AI on Productivity, Innovation and Entrepreneurship

Why can a faster draft still fail to improve the whole task?

A quicker first draft is only one component of productive work. To judge a consequential task, separate at least four outcomes: time to produce an initial draft, whether the deliverable meets its acceptance standard, the total human effort needed to check and correct it, and the value of the result in the next stage of work. A gain on the first measure can coexist with no change or a loss on another. Conversely, extra checking does not automatically erase a benefit if the final work is better, safer or more useful enough to justify that effort.

Consider a hypothetical customer-support response that arrives quickly and reads smoothly but omits a required policy condition. The agent must compare it with the current policy, find the omission, correct the wording and perhaps confirm that the change did not create a new inconsistency. If the response is sent before this review, downstream handling might require a correction or escalation; if it is reviewed carefully, the first-draft time saving may be partly or wholly consumed. This illustration is not a reported event from any study. It shows why apparent fluency is not the same as an accepted answer, and why review burden belongs inside the task boundary being measured.

A local test should therefore define the deliverable and quality bar before comparing assisted and unassisted work. Specify what counts as complete, which errors matter, what must be checked by a person, and where the task ends. Then record the time spent drafting, checking, correcting and handing off, alongside whether the output passed the agreed standard and whether any downstream correction was needed. If the output feeds a later decision, evaluate that use separately when feasible; do not silently count the draft as the whole workflow. Consistent criteria also help distinguish a genuine quality change from a difference in reviewer expectations. Where work volume changes, separate minutes saved per case from total capacity released: a faster unit may mean more cases, shorter queues, added review or no staffing change.

The boundary must be practical as well as rigorous. A small test may not capture rare failures, long-term effects or every benefit that appears after a handoff. That uncertainty is a reason to describe what the test covers and repeat it where stakes or task variation justify doing so, not to assume no benefit occurred. Likewise, extra review is not proof of failure: it may be an appropriate control that makes a valuable assisted workflow possible. The judgment should follow the combined evidence on accepted quality, total effort and downstream use, with unresolved outcomes named rather than folded into a speed claim.

What must change inside an employer before task gains affect a role?

A tool helping one person complete one task establishes a possibility under those conditions; it does not establish that an employer can integrate the tool across a workflow or will reorganize a role around it. In “New evidence for policy making in artificial intelligence,” the OECD describes implementation conditions reported by AI-adopting enterprises and in related interviews and institutional studies. The chapter points to data readiness, integration with existing systems, expense and uncertain returns, change management, and access to suitable vendors as practical issues. These are questions to investigate at a particular workplace: Are usable data available? Can the tool connect to approved systems? Who pays for setup and ongoing review, and what return would justify that cost? Which workers must change routines, and is there a provider the organization can rely on? The chapter does not make any one barrier decisive for every employer.

That evidence also has a boundary. The OECD material includes a selected group of firms already adopting AI across G7 countries and Brazil; it is not a representative census that estimates what all employers are doing. It can show that adoption involves organizational dependencies and reported obstacles, but it cannot establish the prevalence of adoption across every firm, predict a particular employer’s decision, or show that a task trial will scale. Conversely, successful integration in one team would be evidence about that implementation, not proof that other teams have the same data, systems, budgets, permissions or work patterns.

Even when a system is adopted, the productivity result does not decide how management uses any time or capacity released. Possible choices include serving more customers, assigning workers new responsibilities, improving quality or response times, reducing hours, or changing staffing. These are possible allocation decisions, not outcomes measured by the task and performance studies discussed earlier or by the OECD chapter’s barrier evidence. To determine which choice occurred, one needs workplace evidence about the workflow and staffing decisions; to infer a broader occupation-level change requires separate evidence at that level. A tool’s capability, its use by an individual, its integration into a firm and a change in labor demand are therefore different observations, each requiring its own support.

Practical use and firm-level adoption evidence still matter: they make it reasonable to ask whether local integration is under way instead of treating a task experiment as the whole story. But removing a barrier, such as improving data access or finding a compatible vendor, answers only whether adoption has become more feasible. It does not determine how widely the tool will be used, whether the workflow will be redesigned, or how any resulting capacity will be allocated. For an experienced worker, the relevant signal is a specific change in approved tools, recurring responsibilities, quality controls or staffing plans at their organization—not a general conclusion drawn from a tool’s performance on an isolated task.

Sources: New evidence for policy making in artificial intelligence

What is the next realistic move?

Choose one recurring, permitted task with limited consequences and an output you can check. Before trying AI, define the quality standard and record how long the whole task normally takes. Compare like with like: include preparation, review and corrections in the assisted time, and check whether the deliverable meets the same standard. If the result is useful, repeat the comparison; if review consumes the apparent saving, narrow the task or stop. One trial is local information, not a displacement estimate, and a task unlike your most consequential work should not drive a major career decision. Follow your employer’s rules and your own constraints before expanding the experiment.

After that answer, the free checker at /ai-job-risk-checker can help identify task-level change-pressure signals and first actions; it does not provide validated odds of job loss. If you need to compare a larger set of options against constraints such as training time, income needs or location, the paid career roadmap compares scenarios and does not guarantee employment or income. Let repeated, checkable results guide the next step: continue where quality and total effort support it, adapt where they do not, and avoid treating one noisy comparison as a forecast.

Questions readers ask