Why AI Productivity Studies Differ in Support and Software

Why do these studies seem to disagree?

At first glance, the findings seem to conflict: a customer-support study reports higher throughput with an assistant, company software trials report more completed tasks, while a study of experienced developers in familiar codebases reports slower completion. This does not show that one occupation benefits and the other does not. Each result answers a narrower question about a particular work unit, tool and workflow, evaluation boundary, and worker group. Check those four points before comparing percentages. Start with the unit of work and outcome. “Generative AI at Work” followed a staged introduction of a conversational assistant among 5,172 customer-support agents and measured issues resolved per hour. Its abstract reports an average increase of 15%; the detailed paper’s estimate is 13.8%, so summarize it as roughly 14–15%, not two findings. The 2026 “The Effects of Generative AI on High-Skilled Work” pooled randomized access trials across three companies and 4,867 software developers, reporting 26.08% more completed tasks, with a standard error of 10.3%. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” counted time to complete real tasks in mature repositories and found a 19% increase in completion time for its 16 experienced developers. Resolved cases per hour, counted tasks, and elapsed time have different denominators. Their percentages cannot be placed on one scale as if they measured the same thing. Next ask where the tool sat in the workflow and what counted as success. Conversational guidance during customer support and code completion in company trials meet different task conditions from AI tools used on issues in familiar codebases. A measure may capture speed or task counts while leaving quality checks, correction, integration, or rework outside its boundary. A percentage without its denominator and success condition is not a general productivity estimate. Finally, check who participated and how the tool was introduced. The support evidence comes from a staggered workplace rollout at one firm, rather than random assignment. Randomized software trials help attribute differences to access within each setting, but do not make workers, tasks, or outcomes interchangeable. The mature-repository trial’s small group of experienced contributors is not all company developers. The studies can each be accurate in context while giving different results. The practical verdict is conditional: AI may raise measured throughput when its assistance fits the task and workflow, and may add net time where understanding and checking outweigh drafting help. These studies do not establish which occupation is broadly more productive with AI or estimate job displacement. Before applying a headline to your work, match its task, tool integration, outcome, and worker context to yours.

Sources: Generative AI at Work; The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers; Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

What did the customer-support study actually count?

The customer-support result concerns a defined unit of work: customer issues successfully resolved per agent-hour. In “Generative AI at Work,” researchers examined the staggered introduction of a conversational assistant among 5,172 agents at one Fortune 500 business-process software firm. The assistant watched live chats and offered response suggestions and links to internal technical documentation. Agents remained in charge: they could accept, edit, or ignore suggestions. The study concerns a human using contextual guidance during a customer exchange, not a system independently closing cases or a general chatbot answering customers. The tool was part of the agent’s working screen and drew on support context, so the result belongs to that arrangement.

The headline outcome combines pace with whether an interaction reached a successful resolution. The detailed journal paper estimates about 13.8% more resolutions per hour, while its abstract summarizes the estimate as 15%; these are two presentations of the same study, so “roughly 14–15%” is a compact description, not two replications. Resolutions per hour is affected by three components the paper tracks: time to finish an individual chat, the number of chats handled per hour, and the share resolved successfully. Agents may handle multiple chats at once. The reported gain cannot be translated into an equal reduction in minutes for every case, or into 14–15% more customer value. It describes a change in this throughput measure.

The pattern across workers complicates a single average. The study reports larger improvements among less experienced and lower-skilled agents, including a 30% rise in issues resolved per hour for that group. In contrast, its abstract says the most experienced and highest-skilled agents gained a little speed while seeing small declines in quality. The paper describes suggestions that could convey response practices and relevant product documentation, which may be especially useful when an agent has less accumulated experience. This is a plausible interpretation of why the distribution shifted; it does not mean every newer agent benefits or every experienced agent loses quality. The average hides differences that matter when comparing a reader’s task mix with the tested workflow.

The researchers analyzed a staggered workplace rollout, rather than randomly assigning the full workforce to AI access. Their design compares changes over time, with controls for agent and tenure differences, and the authors discuss assumptions needed to interpret those estimates. That is meaningful evidence from support work, but it is not the same design as a randomized coding assignment or company trial. The setting has a clear case-resolution measure and an assistant built around ongoing conversations. Other service jobs may involve different channels, exceptions, escalation rules, or definitions of done. For a worker, the useful comparison is with recurring, knowledge-guided issues that end in a checkable resolution. The percentage alone does not establish broader customer value, staffing changes, layoffs, or a universal productivity effect for customer support.

Sources: Generative AI at Work; Generative AI at Work

Why can a timed coding assignment show a large gain?

The large number comes from a narrow question with a clear finish line: can a developer implement one specified HTTP server in JavaScript faster when a coding assistant is available? In “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot,” recruited software developers were asked to complete that assignment as quickly as possible. The study reports that the treatment group with Copilot finished 55.8% faster than the control group. That is evidence of a speed difference on this tested task. It is not a measurement of how much faster software developers work across a week, release cycle, or career.

A bounded assignment can make code-generation help legible. The task is specified in advance, completion time is the outcome, and participants are not being asked to maintain a service while coordinating with colleagues. In ordinary work, a programmer may need to discover what a request means, navigate an existing codebase, check dependencies, discuss tradeoffs, test edge cases, review changes, and support the result after release. Those activities do not disappear because a tool can draft code. The experiment’s stopwatch answers the time question it set; it does not capture every cost or benefit in a production workflow.

Random assignment matters because it gives the researchers a stronger basis for attributing the observed difference to access to Copilot within this experiment than a simple comparison of people who choose to use the tool and those who do not. That improves causal interpretation in the tested setting. It does not make the setting representative of every developer, task, tool configuration, or organization. Nor does randomization broaden the outcome: the endpoint remains elapsed time to finish one assignment, rather than verified long-run quality, maintenance burden, team throughput, or customer value.

That boundary is why the 55.8% figure should neither be dismissed nor carried over as a general productivity rate. It shows that assistant access can make a particular coding task faster under the experiment’s conditions. It cannot tell a team how much output it will deliver after review, integration, and rework, or whether a different task benefits. The useful comparison is with later field studies that count completed tasks or measure work in established repositories; their outcomes and settings must be read on their own terms. A worker deciding whether the finding applies should ask whether their task is similarly well-defined, whether the assistant fits the surrounding workflow, and whether “done” includes testing and review. Benchmark-like results are a reason to examine a bounded component of coding work, then seek field evidence before extrapolating to team throughput.

Sources: The Impact of AI on Developer Productivity: Evidence from GitHub Copilot

What does the larger company software study add?

The three-company study adds evidence from software work as companies organized it. Randomized trials at Microsoft, Accenture, and an anonymous Fortune 100 company gave a randomly selected group access to a coding assistant suggesting completions. Across 4,867 developers, the pooled estimate associated use with 26.08% more completed tasks; the standard error was 10.3 percentage points. The journal abstract says each experiment was noisy and results varied. This is a pooled average, not a claim that each developer completed a quarter more work. In “The Effects of Generative AI on High-Skilled Work,” the outcome is task throughput, not minutes saved on an assignment.

The design moves beyond asking whether a developer can finish an isolated exercise faster. It asks whether access changes measured output at work. Random assignment makes groups' access different by design, giving researchers a basis for estimating an effect within these trials. But access was not uniform use. The earlier working paper explains that take-up was imperfect and the preferred estimate used assignment as an instrument for actual use. The distinction between developers “using the AI tool” and all those assigned access matters: this is not an estimate of what happens if every worker uses an assistant on every task.

The setting was more realistic than a timed exercise, but bounded. Trials ran within company operations for two to eight months, with differing implementations. The February 2025 working paper describes low initial uptake at Microsoft, limited participation at Accenture, and a short period of differing access during the third company's staggered rollout. These details help explain the journal's warning that results varied and were noisy. A pooled sample does not make workplaces identical or remove uncertainty in each company's estimate.

The denominator changes what the percentage means. A count of completed tasks is not elapsed time on one task, nor successful customer cases resolved per hour. A team might complete more counted tasks while spending different amounts of time on each; support resolution can reflect handling speed, concurrent conversations, or successful resolution. These are different units of output. The 26.08% figure cannot be placed beside the 55.8% speed result from one coding assignment or the roughly 14–15% support estimate as though they were contestants in one test.

The authors report higher adoption and larger measured gains among less experienced developers. That may guide teams, but it does not establish that every junior developer benefits more or that senior developers do not benefit. A completed-task counter does not establish correctness, maintainability, or downstream value. The publisher abstract reports the headline and variation; it does not settle whether review and rework offset measured output gains.

The study strengthens the case that coding assistance can raise measured task throughput in some company settings. It does not settle what a particular team will gain. For a local decision, define “done” for a comparable workflow and track completed work alongside review findings, corrections, and rework. Those checks show whether its measured gain carries through to the quality and time costs that matter in another workplace.

Sources: The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers; The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers

Why did AI add time in familiar software repositories?

AI added time in this trial because it tested whether experienced contributors could finish real issues in repositories they already knew, using early-2025 tools. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” reports a randomized study of 16 developers and 246 tasks in mature projects where participants had an average of five years’ experience. Each issue was assigned to allow or disallow AI. In the AI condition, developers primarily used Cursor Pro with Claude 3.5 or 3.7 Sonnet. Tasks took 19% longer with AI allowed. That finding applies to this group, work, tools, and measurement window; it is not a general estimate for software work. The endpoint was total issue completion time, including implementation before and after pull-request review. Participants tackled real repository issues, submitted pull requests, and followed the projects’ review process. Code that appears quickly is useful only if it can be understood, integrated, tested, and revised without adding more time later. Prompting, waiting, checking, and correcting are plausible costs in this setting, but the result does not prove that any one cost caused every delay. Expectations also diverged from the measurement: developers forecast a 24% time reduction before starting and later estimated a 20% reduction, while the randomized comparison found longer completion. Perceived fluency and end-to-end duration were different measures. Familiarity can affect that balance. A contributor may already know where code lives and how a project handles changes; suggestions that miss local conventions can create review work. The paper examined factors such as project size, quality standards, and prior AI experience, so it would be too simple to attribute the slowdown to review alone. The authors say experimental artifacts cannot be fully ruled out, although analyses found the slowdown robust. The sample was small and selected, and the tools reflected the February-to-June 2025 frontier. Other tasks, tools, or workflows could yield a different result. The contrast with company trials is informative, not a contradiction. “The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers” reports randomized company experiments with a pooled increase in completed tasks. A task count over a work period is not the same outcome as minutes spent on real issues in familiar repositories. The studies differ in participants, tool integration, work unit, and observation. METR compared time on pre-defined issues; company counters recorded output in organizational settings. Neither cancels the other, and task counts alone do not settle quality, review burden, or maintenance. For developers maintaining systems, evaluate the whole issue rather than the speed of code generation. If policy permits, compare similar tasks through verified completion, recording prompts, waiting, rejected suggestions, testing, review changes, and rework. Keep the done condition and quality threshold stable. If drafting gets faster but verification takes longer, the tradeoff should appear in the result. That supports a local workflow test; it does not show that experienced developers should avoid AI, that software teams broadly lose productivity, or that any worker has a particular chance of displacement.

Sources: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity; Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

What does the 2026 follow-up change about the slowdown result?

METR’s 2026 follow-up changes how confidently we can interpret its later slowdown experiment; it does not cancel the earlier result. METR says the new data are an unreliable signal of current productivity effects because developers and tasks were selected in ways that favor AI use, while concurrent agents made task time harder to measure. METR believes developers were probably more sped up by AI in early 2026 than in early 2025, but says the follow-up is only very weak evidence about the increase’s size. The conclusion is uncertainty about today’s effect, not a clean reversal. The timeline matters. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” tested tools available from February to June 2025. It randomized 246 tasks from 16 experienced open-source developers to AI-allowed or AI-disallowed conditions, estimating that AI increased completion time by 19%. That is evidence about those developers, tasks, tools and conditions, not a timeless property of software work. METR began the later experiment in August 2025 with 57 developers overall: 10 from the earlier study and 47 newly recruited developers. Participants submitted tasks for random assignment and recorded completion time. But METR reports that developers increasingly declined to participate if they did not want to work without AI. METR says 30% to 50% of surveyed developers withheld some tasks they did not want assigned to the no-AI condition. In METR’s interpretation, the study systematically missed some developers and tasks expected to benefit most from AI. This suggests a downward bias in estimated speedup, without establishing its size. METR identifies additional concerns. Pay was $50 per hour, compared with $150 in the original, which METR believes may have contributed to selection. Participants said AI changed their task choices and that recording time was difficult when multiple agents ran alongside unrelated work. They show why randomization cannot repair earlier selection or unreliable measurement. The raw follow-up figures should not become a replacement headline. METR estimates an 18% speedup for returning developers, with a confidence interval from 38% faster to 9% slower, and a 4% speedup for newly recruited developers, with an interval from 15% faster to 9% slower. Both intervals include outcomes on either side of zero. METR warns that selection makes them weak evidence about effect size, so either point estimate alone implies unsupported certainty. A careful reader can hold both findings at once: the early-2025 trial found slower completion in its tested setting, while METR considers greater speedup likely by early 2026 and says its follow-up cannot measure that change reliably. Neither settles productivity across software development or between software and customer support. Compare studies by dating tools and workflows, checking who and what entered the sample, and examining time measurement. For a present-work decision, keep the effect open and test a comparable task through verified completion instead of borrowing an unreliable point estimate.

Sources: We are Changing our Developer Productivity Experiment Design; Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

Why do randomized studies still produce different results?

Randomization helps estimate what access to a tool changed inside a particular study. It does not make separate studies interchangeable. Separate studies still involve different work, tools, review demands, and measurement windows. A randomized result can be credible for its own setting while offering only limited guidance about another one. This is the difference between internal causal validity and transportability: who received access may be clear, while where the estimate travels remains uncertain. The three-company software experiments illustrate both the value and the boundary of random assignment. In ordinary business settings at Microsoft, Accenture, and an anonymous Fortune 100 company, a random subset of developers received access to a code-completion assistant. Across 4,867 developers, the pooled estimate was 26.08% more completed tasks (standard error 10.3%), though results were noisy and varied. The February 2025 working paper describes two-to-eight-month trials with different implementations and imperfect compliance. Random assignment helps separate access from pre-existing differences, but task counts do not show time per task or quality. These were distinct company trials, not one standardized test repeated unchanged. [The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers; working paper] Now compare that with the trial described in “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.” Sixteen experienced developers completed 246 tasks in mature projects they had worked on for an average of five years. Tasks were randomly assigned to allow or disallow AI; in this February-to-June 2025 study, allowing tools increased completion time by 19%. The unit was time to finish real issues in familiar repositories, not weekly task counts across company teams. Repository knowledge may make the work unlike a short, self-contained coding assignment, while the tools and period also differ from the company experiments. These contrasts make different results plausible but do not identify the cause. This small sample cannot settle software development as a whole. Customer support adds another design and another denominator. “Generative AI at Work” studies a staggered introduction of a conversational assistant among 5,172 agents and reports 15% more issues resolved per hour on average. This workplace evidence measures support throughput, but its staggered rollout was not randomized assignment. It cannot be compared directly with coding-task percentages or rank support above software generally. The practical reading is to keep each estimate attached to its work unit, tool, people, workflow, and measurement. Randomization strengthens claims about tested conditions, not transfer to another job. These studies support conditional local productivity claims; none measures whether jobs disappear. Before borrowing a headline, check whether your task and its completion test match, then account for review work the study may not count.

Sources: The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers; The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers; Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity; Generative AI at Work

What is outside these productivity measures?

A faster task, more counted tasks, or more customer cases resolved per hour does not by itself prove better durable quality, higher organization-wide output, greater labor demand, or fewer jobs. Each is a narrower result, tied to the outcome and boundary its study actually measured. Start with quality and rework. In “Generative AI at Work,” the customer-support study followed 5,172 agents during a staggered assistant rollout and measured successfully resolved issues per hour. The abstract reports a 15% average increase; the most experienced and highest-skilled agents had small speed gains and small quality declines. The paper describes a single firm and says its data cannot show the firm’s wage changes or overall labor demand. A resolution count is not a complete account of customer value, later corrections, or every exchange’s quality. The software studies leave a different gap. “The Effects of Generative AI on High-Skilled Work” reports randomized access trials across three companies and 4,867 developers, with a pooled 26.08% increase in completed tasks; the abstract says individual experiments were noisy and results varied. But the abstract does not establish task difficulty, downstream review or rework, or product quality. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” instead reports time to finish real tasks in familiar repositories for 16 experienced developers across 246 tasks, finding 19% longer completion time with AI allowed in that sample. Neither endpoint alone measures firm output, which also depends on priorities, usability, and how teams absorb completed work. Tool access is not the same as broad workplace adoption. Randomly giving some developers access can estimate an effect of access within the studied companies; it does not tell us how many employers use similar tools, how consistently workers use them elsewhere, or whether an organization has redesigned its process around them. The support paper studies an actual staged introduction, but its setting still cannot establish adoption across the customer-support occupation. Randomization strengthens conclusions about access in the tested setting; it does not erase that boundary. Finally, task performance is not displacement. None of these three studies measures market-wide job losses, hiring changes across an occupation, or an individual’s chance of losing work. The support paper explicitly says it was not designed to estimate aggregate employment or wage effects. These studies do not establish how employers or labor markets respond to more output per worker. Exposure to a task and evidence that a tool can assist with it are not evidence that a job will disappear. Use these results to ask which tasks merit a careful workflow test, while treating claims about organization-wide demand or job loss as unanswered here.

Sources: The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers; Generative AI at Work; Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

How can you tell whether a study transfers to your own task?

A productivity result transfers to your work only when the study and your task line up on more than the label “customer support” or “software.” Compare five things: the work unit, its context, the tool and how it enters the workflow, the workers, and the outcome counted as success. The closer those conditions are, the more useful the study is as a reason to test a workflow. A mismatch does not make the result false; it weakens what it tells you about your situation. First, match the work unit. “Generative AI at Work” counted successfully resolved customer chats per hour during a staged assistant rollout at one business-process software firm. It is relevant to recurring cases where agents use conversational guidance and resolution is recorded. It does not directly describe support work requiring a long investigation, handoff, or different definition of resolution. The GitHub Copilot experiment measured elapsed time on one JavaScript HTTP-server assignment. It can inform a bounded coding task, but is a poor stand-in for shipping and maintaining a feature involving dependencies and coordination. Second, compare context and worker experience. Is the task repeatable, or does it depend on local records, unwritten rules, or choices that need explanation? Do participants have experience like yours? “The Effects of Generative AI on High-Skilled Work” counted completed tasks in three company experiments with developers using coding assistants. METR’s early-2025 trial followed experienced open-source developers doing real issues in repositories they knew well. Neither result applies automatically to every software role. Third, inspect the workflow and finish line. Does the tool appear where work happens, or must the worker move information, prompt it, wait, and reconcile an answer? Who checks correctness, and does the outcome include review, corrections, or later rework? A fast draft is not equivalent to a verified result. A task count, elapsed time, and resolved-case rate answer different questions. If you care about reliable completion, a study recording only initial speed leaves a consequential gap. Use this as a transfer test, not a numerical score: write down the task, tool access, relevant experience, context dependencies, verification step, and outcome you need. If several differ from the study, treat its percentage as background, not a forecast. If the match is close, it may justify a small, permitted comparison of similar tasks, timed through checked completion and including prompting, review, and correction. That local test can guide one workflow decision; it cannot establish an occupation-wide productivity effect or a job-loss probability.

Sources: Generative AI at Work; The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers; Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

What is a proportionate next step?

Choose one permitted, recurring task with a clear definition of “done,” then compare similar attempts from start to verified completion. Before using a workplace tool, check rules for confidential data, approved software, and required review. Keep the task narrow enough to judge correctness, such as preparing a routine response from approved guidance or making a small, specified code change. Record total work time, including instructions, prompting, waiting, checking, corrections, and rework. Pair time with a quality signal that matters for the task: whether a response follows guidance or a code change passes team checks. Compare work of similar difficulty against the same completion standard. If AI saves drafting time but creates more review or errors, it has not improved this workflow. If results are unclear, narrow the task or repeat it before changing how you work. This local check is a decision aid, not a substitute for research design. “Generative AI at Work” measures resolved support issues per hour in one rollout. METR’s “We are Changing our Developer Productivity Experiment Design” explains why selection and time-measurement problems weaken its later estimate. Neither tells you what will happen in your task. The free task checker can organize change-pressure signals, but its result is not a validated probability of displacement.

Sources: Generative AI at Work; We are Changing our Developer Productivity Experiment Design

Questions readers ask