What Did METR’s Early-2025 AI Study Measure?

What counted as a task, and who did it?

METR’s Early-2025 experiment followed 16 experienced open-source developers completing 246 real issues in repositories they regularly contributed to. It ran from February through June 2025, so “early-2025” identifies a particular tool period, not a timeless test of AI coding. In “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” researchers report that participants typically had more than a decade of software-engineering experience and an average of five years working in the target repository, where they had made about 1,500 commits. These were contributors with deep knowledge of particular codebases, not a cross-section of developers simulating ordinary jobs.

The unit of work was an issue: a bug report, feature request, or other work item used to coordinate development. Developers proposed real issues from repositories where they were active; the paper says these were work they wanted to contribute. About half were not formally listed in the repository and were tracked separately for the experiment. Participants were asked to submit issues expected to take no more than two hours, splitting longer work into smaller issues. The 246 tasks took about two hours on average. After completing an issue to their satisfaction, developers submitted a pull request through the project’s code-management system, where it was typically reviewed; the methods describe requested changes and merging completed work. This was grounded in project work, but bounded to an issue rather than an entire release or job.

This setting helps answer how experienced contributors handled bounded work in familiar, mature repositories with the tools available at the time. The paper describes repositories averaging roughly 1.1 million lines of code and high contribution standards. Familiarity and review expectations matter: the result can illuminate work where local context and fitting a change into an existing project are part of the task. But recruitment through professional networks and outreach to active contributors, plus the small group and selected repositories, limits generalization. The study did not sample junior developers, people entering unfamiliar codebases, all software engineers, or other occupations. Its issues also cannot represent every activity in a developer’s role. Treat the participant and task description as the evidence boundary: this is a meaningful case of experienced open-source contribution, not a representative sample of software work as a whole.

Sources: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity; Early-2025 AI experienced open-source developer study

How did random assignment and AI access work?

METR first asked participating developers to nominate real issues from repositories they regularly contributed to. An issue could be a bug report, feature request, or other work item. Before treatment assignment, each developer estimated how long each issue would take with AI assistance and without it. The paper says those forecasts served as a measure of expected AI benefit and a proxy for issue difficulty. The sequence matters: developers identified the work and made forecasts before learning which condition an issue would receive, so assignment could not shape the original list or those estimates. This was not a researcher handing every participant an identical coding puzzle. It was work contributors wanted to do in projects they knew, making the setting relevant to repository maintenance while leaving participant and task selection as important limits.

METR then assigned each issue to one of two conditions using a simulated fair coin flip: AI allowed or AI disallowed. In the allowed condition, participants could use any AI tools or models they chose, including no AI if they judged it unhelpful. In the disallowed condition, they could not use generative AI tools. This distinction is central to interpreting the result. Assignment to “AI allowed” measured access under the study’s rules; it did not guarantee that every issue involved AI use, or test an autonomous agent doing the job alone. Nor was it a comparison between a human and a model working separately on identical jobs. Researchers compared issue completion when these developers had permission to use tools with completion when their use was prohibited.

Participants could complete assigned issues in their preferred order, work flexibly, and sometimes handle multiple issues at once. When they considered an issue complete, they submitted a pull request. Another developer typically reviewed it, participants made requested changes, and then reported time spent before and after review. That connected the task to a real contribution process, although the study relied on participants’ time reports and its own measurement protocol. METR supplied Cursor Pro subscriptions and basic training but did not require participants to use Cursor. The paper reports that AI users primarily chose Cursor Pro with Claude 3.5 or 3.7 Sonnet, tools described as state of the art for February through June 2025. The period and tool mix are part of the finding, not details that can be silently updated to today’s products.

Random assignment helps make the two sets of issues comparable on average within this experiment, Forecasts made before assignment also gave researchers information about expected difficulty. But randomization does not make participants representative of software workers. They volunteered issues they wanted to contribute to familiar open-source repositories; their expertise, standards, and context may differ from those of employees in unfamiliar codebases or product teams. It does not estimate employer adoption, labor-market usage, or whether a worker’s job is exposed or likely to disappear. Read it as a test of what happened when experienced contributors could choose whether to use the period’s tools on nominated work. Permission to use AI is not mandatory use, and neither measures workforce adoption.

Sources: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity; Early-2025 AI experienced open-source developer study

What did ‘completion time’ include—and what did it miss?

METR’s main outcome was total implementation time for an assigned issue, measured from developers’ reports of time spent before and after pull-request review. After completing work to their satisfaction, participants submitted a pull request to the repository’s usual review process, made changes requested by reviewers, and reported time spent on the issue both before and after review. The paper, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” defines its estimate using that total, including pre-review and post-review implementation time. This matters: the headline was not a stopwatch comparison of how quickly code appeared on screen. It was an estimate of time attached to completing issues through a contribution and review process.

The researchers estimated the difference between average issue time when AI was allowed and average issue time when it was disallowed, using a regression that incorporated developers’ pre-assignment estimates of issue difficulty. A few issues were not reviewed or merged by the study’s end; the researchers imputed their post-review implementation time using the mean post-review time for that treatment condition, then checked whether results were robust to the imputation method. This is more informative than raw typing speed, but it does not score correctness, maintainability, later bugs, or team output over a release cycle. Review-related time is represented in the reported measure; downstream effects after the measured contribution are a separate question.

METR also gathered evidence about process, but those records answer different questions. Researchers manually labeled a subset of screen recordings in roughly ten-second intervals, classifying visible activities such as writing code, testing or debugging, reading and searching, prompting a tool, waiting for output, and reviewing generated suggestions. Of 128 recordings initially labeled, filtering for recording problems, cheating, or mismatches with reported time left 74 issues and 84 hours for further analysis. That is a detailed view of observed activity in a subset, not an independent productivity score for all 246 issues. A screen can show that a developer spent time reviewing an output; it cannot establish whether the resulting code was correct or whether that review prevented a later problem. Nor can activity labels alone identify which step caused a difference in total time.

Other supplementary records have their own limits. Cursor analytics covered the 13 participants given Cursor Pro subscriptions and included their most-used model and counts of suggested and accepted AI-generated lines. Those counts describe tool interaction; accepted lines are not equivalent to correct, useful code or net output. Before assignment, developers forecast how long issues would take with and without AI, and after participation they estimated AI’s effect. Those forecasts and retrospective estimates reveal expectations and perceptions, not alternate measurements of completed work. Together, the measures let readers distinguish time, visible workflow, tool interaction, and belief. They do not establish quality-adjusted delivery, team throughput, learning, or a universal mechanism behind the result. For a local comparison, keep the same definition of done and include testing, review, corrections, and rework; a shorter coding phase alone is not evidence of faster completed work.

Sources: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

What does 19% longer mean beside the forecasts?

The headline is a comparison of average issue-completion times between two randomly assigned conditions. METR estimated that issues assigned to the AI-allowed condition took 19% longer than those assigned to the AI-disallowed condition. In the paper’s definition, the comparison is the expected completion time with AI allowed divided by expected completion time with AI disallowed, minus one. It does not mean that every participant was 19% slower, that a typical developer will lose 19% of productive capacity, or that the model itself needed 19% more time. The measured unit was the developers’ work on assigned issues in this experiment. That result differed from what participants expected. Before assignment, developers forecast that allowing AI would reduce completion time by 24%; after taking part, they estimated that AI had reduced it by 20%. Those are expectation and retrospective-estimate measures, respectively, not alternate versions of the randomized outcome. The distinction matters because a person can feel that a tool speeds up a visible step, such as drafting or finding a code pattern, while the study’s endpoint compares total reported implementation time across conditions. The contrast is a reason to check completed work rather than infer the whole workflow from its quickest moment. It does not establish that participants were careless or that developers generally misjudge their own work. The research team’s July 2025 summary reports a 2% to 39% confidence interval around the estimated increase in time. Read alongside the point estimate, this interval indicates statistical uncertainty in the study’s estimate: under the analysis, the data are consistent with an increase whose size could be smaller or larger than 19%. It is not a forecast range for one person, a range covering all software jobs, or a statement that any individual has a 2% to 39% chance of slowing down. The paper also reports checks using alternative analytical choices and examines possible features of the setting that could account for the result. The authors say the slowdown was robust across their analyses and unlikely to be primarily a function of the experimental design, while also stating that experimental artifacts cannot be entirely ruled out. That combination supports taking the result seriously as a bounded finding; it does not remove uncertainty or turn it into a universal law. Robustness does not answer whether other tools, tasks, teams, or periods would produce the same result. The practical reading is specific: the study challenges the assumption that access to AI automatically saves time on context-rich work by experienced contributors. Its result is not a claim that AI makes developers 19% slower in general. If you want to know whether a tool helps with your own recurring task, compare completion through checking and rework under your actual constraints. Keep the acceptance standard the same, and treat a small set of observations as local evidence to inform a decision, not as a controlled estimate of your personal productivity.

Sources: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity; Early-2025 AI experienced open-source developer study

Which workflow mechanisms are plausible, and which were proven?

The experiment supports a bounded conclusion about the AI-allowed condition; it does not identify one mechanism that caused the longer completion time. In “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” METR examined 20 possible properties of its setting using screen recordings, repository records, surveys, interviews, and analyses of subsets of tasks. The authors report evidence that five factors likely contributed to the slowdown, mixed, unclear, or no evidence for nine, and evidence against six. They explicitly call this mechanism work “speculative, suggestive evidence” and caution that their study was not powered for statistically significant multiple comparisons across subsets. The randomized comparison is stronger evidence for a difference between assigned conditions than the subsequent search for explanations. Several reported patterns point toward the particular match between experienced developers and familiar, mature repositories. Participants averaged five years and about 1,500 commits in the repositories they worked on. METR found that developers slowed down more on issues they said they knew better, and participants described their familiarity as making it harder for AI to help. The authors also identify repository size and maturity as factors that likely contributed. One interpretation is that, when a contributor already knows local conventions and where relevant code lives, assistance that still needs orientation or adaptation may add less value than it would on a less familiar task. This fits the study setting; it does not show that experience generally makes AI less useful. The paper records process clues. On 74 issues with valid recordings, researchers labeled activity in roughly ten-second intervals. In AI-allowed work, developers spent a smaller share of recorded time coding and searching, and more time prompting, waiting for, and reviewing AI outputs. In a separate analysis of 44 issues with valid recordings, the paper reports that developers spent about 9% of their time reviewing and cleaning generated outputs. Cursor analytics for participating users showed that fewer than 44% of generations were accepted. These observations make checking, adapting, and retrying plausible contributors. They do not establish that any one activity caused the treatment effect: the labeled sample is a subset, acceptance is not a measure of correctness, and activity shares alone cannot say what would have happened without the tool. Other possible explanations remain unsettled. Some participants had used large language models before but had limited experience with Cursor, so learning the interface or prompting could matter. Yet METR found no meaningful change in slowdown across developers’ first 50 hours of Cursor use and did not find evidence that below-average AI-use ability explained the result. Other explanations, including latency, external-resource use, task characteristics, and study procedures, remain uncertain. It would therefore overstate the findings to attribute the result simply to bad prompts, weak suggestions, or waiting. The authors say experimental artifacts cannot be entirely ruled out, even though the overall estimate appeared robust across several design checks. For your work, the useful question is where elapsed time goes: locating context, framing a request, generating, checking, testing, reviewing, and repairing. Compare similar instances under your organization’s rules and keep the acceptance standard constant. That can reveal a local friction point worth changing, but a handful of observations is not a controlled estimate and cannot isolate cause. METR’s evidence makes workflow accounting sensible; it does not diagnose your workflow from the 19% average.

Sources: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity; Early-2025 AI experienced open-source developer study

Why is a coding benchmark not the same measurement?

A coding benchmark and METR’s early-2025 study ask different questions. In the benchmark described in Introducing SWE-bench Verified, an agent receives a code repository and an issue description, then must generate a patch; evaluation runs tests against that patch. METR instead randomized real issues between conditions in which experienced developers could or could not use AI, and measured their completion time. Neither result cancels the other: an agent may solve some benchmark issues while AI access still adds time to developers’ work on other tasks, or under other conditions.

SWE-bench issues originate in resolved GitHub work, with the repository and issue statement supplied to the agent; the proposed patch counts as successful when the designated issue tests and regression tests pass. METR’s participants selected issues from repositories they knew and worked through them as contributors, with time to complete as the principal outcome. A benchmark score therefore summarizes task success under benchmark rules, not the time a developer and tool need to understand a codebase, decide what to change, test it, discuss a pull request, and finish the work. Conversely, METR’s 19% longer estimate does not tell us how often an autonomous agent could pass a benchmark’s tests. The measures are related because both concern software tasks, but they are not interchangeable measures of productivity or capability.

Benchmarks also depend on their test design. Introducing SWE-bench Verified describes a curation effort: professional developers reviewed 1,699 samples, and the resulting Verified subset contained 500 tasks selected to address underspecified prompts and tests that could reject valid solutions. That process improves the intended evaluation, but it does not turn a test pass into a measure of workplace delivery. In Why SWE-bench Verified no longer measures frontier coding capabilities, OpenAI later reported auditing 138 difficult Verified tasks, with at least 59.4% containing material issues in their tests or problem descriptions. These are findings about that benchmark audit, from its creator; they do not reanalyze METR’s trial or invalidate its measured time result. They show why a benchmark score needs a date, task set, scoring rule, and validity caveat.

For a developer reading a claim about AI coding, ask four practical questions: Who or what performed the task? What tools and permissions were allowed? Did the outcome mean a tested patch, elapsed time, accepted work, or something else? What work happened outside the scoring boundary, especially review and rework? These questions prevent a capability result from being quietly converted into a promise of faster team delivery. A strong benchmark score cannot erase METR’s finding in its tested workflow; METR’s time estimate cannot show that coding agents lack capability. To judge your own work, compare like tasks through the same completion and quality checks, and count the time spent verifying results. That local observation remains distinct from either benchmark success or a general claim about software jobs.

Sources: Introducing SWE-bench Verified; Why SWE-bench Verified no longer measures frontier coding capabilities; Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

What does METR’s later experiment change?

METR’s later experiment does not replace the early-2025 result with a measure of current productivity. In “We are Changing our Developer Productivity Experiment Design,” METR says its follow-up began in August 2025, but selection and time-accounting problems make its data an unreliable signal of AI’s current productivity effect. The update reports some evidence of speedup, while calling evidence for its size very weak. Conditions and participant behavior changed, so the later estimate cannot simply reverse the earlier one.

The follow-up involved 57 developers, 143 repositories, and more than 800 tasks, compared with 16 developers and 246 tasks in the February–June 2025 study. METR says 10 follow-up participants came from the original study and 47 were newly recruited from more diverse open-source projects. Pay fell from $150 to $50 per hour. A larger sample does not ensure a comparable estimate when willingness to participate or task submission may depend on AI access; the pay change may also affect who joins.

METR reports more developers were unwilling to do half their study work without AI, and some declined to participate for that reason. In surveys, 30% to 50% said they had chosen not to submit some tasks because they did not want to do them without AI. If contributors or tasks expected to benefit most are missing, the comparison may understate those gains. METR says that is likely, while noting lower pay could also have contributed. This is the team’s interpretation of survey and participation evidence, not a measurement of productivity for absent developers.

Task comparability also weakened. Some participants told METR that AI changed which tasks they chose; others reported differences in output, including subjective code quality or the amount of testing and documentation. Some were less likely to complete AI-disallowed tasks; METR says one developer completed none. These reports mean time may compare different work or definitions of completion across conditions. They do not show that everyone changed tasks or that AI necessarily lowered quality. They limit what elapsed time alone can say about value delivered.

METR says concurrent agent use made time reporting difficult: while an agent worked, a participant might work on another task. Wall-clock time, active attention, and parallel work are different measures. If they cannot be separated consistently, task time is a weak proxy for productivity. METR describes its later central estimate as a bad proxy for real productivity impact among these developers. That limits interpretation; it does not show that AI had no effect.

The update estimates an 18% speedup for 10 returning developers, with a confidence interval from 38% speedup to 9% slowdown. For newly recruited developers, it estimates 4% speedup, with an interval from 15% speedup to 9% slowdown. Both intervals include no effect, and METR says selection effects obscure the result. These figures do not establish an early-2026 average. The returning group is not a clean correction to the original trial because participation, task selection, tools, and measurement changed. METR says participant conversations suggest greater speedup in early 2026, but calls its data weak evidence for the size of the increase.

The original study remains a historical result: in February–June 2025, AI allowance increased completion time by 19%, with a 2%–39% confidence interval. The follow-up neither erases that estimate nor shows current tools slow developers. A stronger comparison needs stable recruitment and compliance, comparable tasks and tool conditions, explicit treatment of unfinished work and parallel agents, and transparent measures of completion time and output quality. Until then, keep the early result tied to its period and treat later numbers as uncertain signals, not a verdict on current software work.

Sources: We are Changing our Developer Productivity Experiment Design; Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

What can a worker reasonably do with this study?

Do not make a career decision from METR’s 19% result alone. It is evidence about one early-2025 workflow, not a forecast of your speed, role exposure, or job prospects. In “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” the primary outcome was developers’ self-reported implementation time, including time before and after pull-request review. For a practical check, choose one recurring, low-stakes task and use an approved tool on comparable instances, following your usual process on others. Keep acceptance criteria the same throughout. Track elapsed time through verification; note errors, review comments, rework, and where judgment or accountability stayed with you. This is a bounded observation, not a controlled experiment: a few cases cannot establish a general effect, and task differences can shape the comparison. This can reveal whether assistance shifts effort from drafting to checking. Keep the signals separate. A tool’s capability, your task’s exposure, employer adoption, labor demand, and displacement are different questions. If uncertainty spans more of your role, the free task checker organizes task-level change-pressure signals; it does not report a validated probability of job loss. Start with the work you actually do, then decide whether a workflow change merits a broader career conversation.

Sources: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity