When Does AI Assistance Improve Knowledge Work?

When does AI assistance improve knowledge work quality?

AI assistance is most likely to improve knowledge work when the task has clear requirements, the tool can handle the specific subtask, and a person can check consequential parts of the result. Quality can fall when a worker delegates context-sensitive judgment beyond that demonstrated fit or accepts a plausible answer without a dependable check. For a recurring task, compare completed work against the same criteria; fluency and speed alone do not show that the result is better. The evidence is conditional. In the professional-writing experiment summarized as Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence, college-educated participants given ChatGPT access completed the tested occupation-specific writing tasks faster and received higher quality ratings. The BCG consultant experiment, Navigating the Jagged Technological Frontier, found gains on tasks judged within GPT-4’s capability frontier, but lower accuracy on one difficult recommendation task. These studies concern particular tasks, participants, tools, and measures, not every workplace workflow. So assess quality, correctness, total effort after review, and what the worker learns as separate outcomes. This is a guide to choosing how much help to use on a task, not a forecast of whether an occupation or person will lose work.

Sources: Experimental evidence on the productivity effects of generative artificial intelligence; Navigating the Jagged Technological Frontier

What counts as better work, beyond a faster first draft?

“Better work” should mean that the completed deliverable meets the task’s purpose and acceptance criteria, including correctness when facts, calculations, or recommendations matter. Speed, volume, polish, confidence, customer response, and later learning may matter too, but none substitutes for that standard. A draft can arrive sooner and read smoothly while missing a decisive constraint. The question is not simply whether assistance made work faster, but whether the finished result met the same standard after checking and repair. The studies in this evidence base measure different kinds of quality. In “Navigating the Jagged Technological Frontier,” the BCG consultant experiment assessed whether recommendations on a deliberately challenging business task were correct, alongside performance on other tasks judged within the model’s capability frontier. Correctness matters because an attractive recommendation that reaches the wrong conclusion is not an improvement. In the customer-support rollout, “Generative AI at Work” tracked issues resolved per hour, resolution rates, and customer satisfaction. Those outcomes matter in that workflow, but throughput, resolution, and satisfaction are distinct; an increase in one does not establish that every response was more accurate or complete. Other studies use other yardsticks. The METR trial of experienced developers in familiar open-source repositories measured time to complete software issues and reported similar quality across treatment conditions. Its slowdown result concerns measured completion time in that setting, so it cannot establish a general quality decline. The professional-writing experiment assessed bounded writing outputs as well as time. These differences make cross-study averages easy to misread: task success, reviewer ratings, customer outcomes, and elapsed time answer related but separate questions. For a practical comparison, define the task class and acceptance criteria before using a tool. Inspect the completed deliverable, not only the first draft. Record whether it meets the purpose, which errors or omissions appeared, what revisions were needed, and how much time verification and repair added. For a recommendation, check whether its conclusion follows from evidence and respects constraints. For a customer response, use the service criteria that matter for that exchange. This is not a universal scoring formula; it keeps measurement tied to the work being judged. Write criteria in observable terms before comparing results, such as required facts, exclusions, or a decision that must be justified. Fluency can create a misleading signal. A response may be clear and well organized while relying on a wrong assumption; a polished recommendation can fail a correctness test. Likewise, more resolved cases per hour may be operationally valuable without proving improvement on every dimension of service quality. Ask what the work was meant to accomplish, what would count as a material failure, and whether the final version meets that bar after review. For a recurring task, compare like with like: use the same task type and written rubric for assisted and unaided deliverables, then consider quality separately from speed, volume, and confidence. An aggregate gain is evidence about the tested setting, not permission to skip the criteria governing your own work. Infer improvement only when the completed result meets its purpose and acceptance standard.

Sources: Navigating the Jagged Technological Frontier; Generative AI at Work; Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

Where does assistance improve the finished work?

A bounded first pass is the strongest positive case in the available evidence. In “Experimental evidence on the productivity effects of generative artificial intelligence,” a preregistered online experiment assigned occupation-specific, incentivized midlevel writing tasks to 453 college-educated professionals and randomly gave half access to ChatGPT. The researchers report that average completion time fell by 40% and assessed output quality rose by 18%. Because access was randomized, this comparison supports a causal effect for those participants and tasks, rather than merely showing that people who chose to use a tool differed from those who did not. It does not establish that any writing task, any worker, or any workplace will see the same change. The useful lesson is about the shape of the task. The experiment concerned bounded professional writing, where a person could receive a defined assignment and produce a deliverable that evaluators could assess. That setting gives the tool a tractable role: it can help turn an explicit request into a draft, while the worker retains responsibility for whether the final text is fit for its purpose. A faster draft is not automatically better work; here, however, the researchers also assessed quality and found an improvement on their measure. For a practical illustration, consider a worker who already has meeting notes and a specific audience in mind, and needs to turn those notes into a short internal explanation. They might ask for a structure or a first draft, then check that each point is supported by the notes, restore necessary qualifications, and edit for the audience. This is an application of the experimental result, not a task reported in the study. If the notes omit a crucial decision or contain disagreement, fluent prose cannot resolve that gap by itself. The same distinction appears in “Navigating the Jagged Technological Frontier,” a preregistered randomized experiment involving 758 knowledge workers at Boston Consulting Group. Participants were assigned to no-AI, GPT-4, or GPT-4 plus a prompt-engineering overview conditions. Across 18 realistic consulting tasks judged to be within the tested system’s capability frontier, the AI groups completed 12.2% more tasks and worked 25.1% faster on average, while delivering significantly higher-quality solutions. What can a worker reasonably take from these studies? Look for work where the goal and acceptable form can be stated before prompting: organizing supplied material, producing alternative phrasings, outlining a routine explanation, or generating options for a person to select and revise. These are illustrations of task characteristics, not claims that either experiment tested each example. The key is not that writing is easy or low-value. It is that a defined contribution can be inspected against an explicit purpose. A draft can reduce the effort of starting or offer a structure to react to, while domain knowledge remains necessary to catch omissions, preserve important distinctions, and decide what should actually be sent. This positive case has boundaries. The writing study tested occupation-specific tasks online, not a long-running organizational workflow, and its measured quality result belongs to the assignments and assessment used there. Neither shows that AI use has been adopted across an industry, raises demand for a role, or changes the probability that a particular worker will keep a job. The studies therefore support a modest starting rule: use assistance first on a subtask with available inputs, a visible target, and an independent way to judge the finished contribution. Preserve the human check, and evaluate the completed work rather than the smoothness of the draft. If the output cannot be judged without evidence the worker lacks, or if the task’s real goal is still unsettled, the positive writing results do not justify handing over the central decision. That distinction keeps the evidence useful: it identifies a reasonable place to try assistance without turning a result from particular experiments into a promise about knowledge work as a whole.

Sources: Experimental evidence on the productivity effects of generative artificial intelligence; Navigating the Jagged Technological Frontier

Why can similar tasks move in opposite directions?

A workday can contain tasks that look equally demanding to a person but place very different demands on an AI system. The same consultant might ask for a set of plausible product ideas, a first draft of a client explanation, and a recommendation that depends on figures in a spreadsheet plus details from interviews. Fluency on the first two does not establish reliability on the third. The practical question is therefore not whether the overall assignment is “easy” or “hard,” but whether the specific contribution requested fits the tool’s demonstrated capability and can be checked against the evidence that matters. That distinction is visible in “Navigating the Jagged Technological Frontier,” a preregistered randomized experiment with 758 Boston Consulting Group consultants. Participants worked on realistic management-consulting tasks using GPT-4, GPT-4 with a prompting overview, or no AI. On 18 tasks classified by the researchers as within the model’s capability frontier, AI users completed 12.2% more tasks, finished 25.1% faster on average, and delivered solutions rated as significantly higher in quality. On a separate complex managerial task selected as outside that frontier, participants with AI access were 19% less likely to produce a correct solution than those without it. These are results from a particular experiment and model available in June 2023, not estimates for all consultants, current systems, or all strategy work. The contrast helps explain how an apparently helpful response can make a result worse. In the outside-frontier task, participants had to recommend whether a company should acquire or invest in a company making electric moped batteries. Reaching the correct answer required combining spreadsheet analysis with relevant information from interviews. Analysis limited to the spreadsheet could point toward the wrong choice. The study reports that the AI-supported participants were more likely to give an incorrect recommendation, consistent with a failure to integrate the decisive context. A clean explanation could make that gap harder to notice: the recommendation may read smoothly while resting on incomplete evidence. The experiment does not show that every model or person will miss interview details; it shows that, in this designed task, assistance reduced correctness for a group of expert consultants. This is why human-perceived difficulty is a weak screening test. A task that feels intellectually sophisticated may have a clear output, usable examples, and criteria a worker can apply, making a bounded contribution easier to evaluate. A task that seems routine may still depend on a small contextual detail, a constraint buried in a conversation, or a relationship between data sources. The paper’s term “jagged frontier” describes that uneven fit: tasks that appear similarly difficult can fall on different sides of the point where assistance helps or harms performance. A boundary observed for a given model and task can shift when the model, supplied context, instructions, or evaluation changes. For a mixed assignment, separate production from synthesis. It may be reasonable to ask for alternatives, a first draft from supplied facts, or a critique of an existing outline, then check that contribution against the brief and source material. Keep the final recommendation with the person responsible when the decision turns on evidence spread across files, interviews, and local knowledge that has not been made explicit. This is a practical inference from the experiment, not a validated rule that assigns every task to a person or tool. If the deciding information can be assembled and independently checked, a bounded AI contribution may still help. If a polished answer can conceal that it missed the crucial fact, retain human control over the synthesis and state what evidence would change the decision. The useful lesson is not that AI is good at simple work and bad at difficult work. It is that capability must be tested at the level of the actual subtask, while accountability for the whole recommendation remains with the worker.

Sources: Navigating the Jagged Technological Frontier; Navigating the Jagged Technological Frontier

What does workplace use add to controlled task experiments?

A workplace rollout answers a different question from a short, controlled task experiment. In “Generative AI at Work,” researchers studied the staggered introduction of a conversational AI assistant among 5,172 customer-support agents. Access increased issues resolved per hour by 15% on average. Because the assistant entered an existing service operation through a staggered rollout, the evidence concerns access to assistance in practice, rather than a claim that every agent used every suggestion. The measured average combines outcomes across workers and cases, so its size does not describe what each person experienced. The researchers report that less-experienced and lower-skilled agents improved both speed and quality, while the most experienced and highest-skilled agents gained a little speed and saw small declines in quality. The study also reports evidence of worker learning and improved English fluency, particularly among international agents, as well as customers being more polite and less likely to ask for a manager. Issues resolved per hour measures throughput; the paper also examines resolution rates and customer satisfaction, while customer behavior is another outcome again. These are useful operational indicators, not a universal rubric for whether every piece of knowledge work is correct, complete, or well judged. Its quality findings are especially instructive because they differ across worker groups: the average can conceal a subgroup for whom speed and quality move in different directions. That is why the positive overall result should inform a question about local measurement, rather than settle it in advance. The pattern is consistent with a plausible mechanism, though it does not prove that mechanism for every workplace. In a repeatable support workflow, suggested replies can make useful response patterns available to agents who have had less time to encounter unusual cases. The paper reports the largest gains for relatively rare problems, where human agents had less baseline training and experience. Assistance may therefore have more room to help someone who has not yet built a strong store of examples. For a highly experienced agent, by contrast, a suggestion may add little to an established approach, or require attention to assess and adapt. That distinction is one reason field evidence complements controlled writing experiments. A bounded writing study can compare outputs on a defined assignment under controlled conditions, using a scored deliverable as its quality measure. A field rollout can show how access relates to operational measures across a real workflow and workforce, including differences by experience. The settings answer related but distinct questions: one isolates performance on a specified task, while the other captures a tool embedded in day-to-day service. This support study concerns one firm's customer-service operation and its conversational assistant, not all knowledge work, all support systems, or the labor market's adoption of AI. For a worker or manager, the practical implication is to check results by task and experience level instead of applying the average uplift to every person. If AI is introduced for drafting routine replies, track both completed cases and a suitable quality measure, and look for differences between newer and experienced staff. Keep a route for agents to reject, edit, or escalate suggestions, because access to a recommendation does not remove the need to judge whether it fits the case. Field use can reveal real operational trade-offs; it cannot make those trade-offs universal. Local monitoring should therefore include both aggregate and subgroup results.

Sources: Generative AI at Work

Why can a useful-looking workflow still cost time or accuracy?

A workflow can look efficient because the first answer arrives quickly while the work required to make it usable remains hidden. Measure the time to complete the task to its actual standard, including checking, integration, testing, and repair. Time, delivered quality, and the worker’s ability to make the same judgment unaided are different outcomes; a study measuring one has not necessarily measured the others. The METR report, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” illustrates why total workflow cost matters. In a randomized trial, 16 experienced developers worked on 246 real issues, such as bug fixes, features, and refactors, in large open-source repositories they had contributed to for years. Each issue was assigned to an AI-allowed or no-AI condition. The issues averaged about two hours; in the AI condition, participants could choose tools, primarily Cursor Pro with Claude 3.5 or 3.7 Sonnet, which were frontier models at that time. Developers took 19% longer when allowed to use AI. They had expected a 24% speedup, and after the trial still estimated that AI had sped them up by 20%. METR’s report also says submitted pull requests were of similar quality with and without AI, so its slowdown is not evidence that AI made code worse in this trial. The result is about completion time in this particular setting, not a general quality decline. The setting limits generalization. These experienced contributors worked in mature repositories, where local conventions, tests, documentation, and implicit requirements mattered, using tools available from February through June 2025. METR calls the result a snapshot and says it does not show that AI fails to speed up most developers, other occupations, or users of later systems. The narrower inference is that integration and review can offset a fast suggestion when work has substantial context and a high acceptance bar. A three-month patent-drafting experiment complicates a different assumption: that receiving assistance necessarily weakens independent expertise. The Google Research page for “Does AI Assistance Enhance or Erode Expertise? Evidence from a Three-Month Field Experiment in Patent Drafting” identifies it as an NBER 2026 working paper. Its preregistered randomized trial gave 133 practicing patent lawyers at 11 U.S. firms access to a custom AI drafting assistant. Blinded expert patent attorneys scored work. Assisted benchmark drafting quality was higher at both 10 days and 90 days, with reported gains of 0.34 and 0.38 standard deviations, respectively. After three months, lawyers also completed a redlining task without AI, requiring professional judgment. Results differed by experience. The overall advantage on unaided redlining was concentrated among senior lawyers. Junior lawyers showed no average gain; scores split, with fewer middling results but more poor and more good ones. This does not prove AI builds expertise for experienced workers or harms newer ones. It is a patent-practice result using a custom tool, from a modest sample, and remains a working paper. It shows why assisted deliverable quality and unaided judgment deserve separate measurement: a strong assisted draft does not establish whether the worker can later evaluate similar material independently. METR measured time on familiar software issues and found a slowdown alongside similar pull-request quality. The patent trial measured expert-rated drafting quality during assistance and later unaided judgment, finding gains in delivered drafts and a more mixed learning pattern. Their populations, tools, workflows, and outcomes differ, so they answer separate questions. For an individual task, record whether the finished work met its criteria and what errors needed repair, total time including review and integration, and, where relevant, whether you can explain and defend the result without relying on the generated answer. These are practical measures, not a validated assessment of long-term expertise. They can expose workflows that feel faster but consume more time, or improve deliverables while leaving independent judgment untested. Use a bounded task trial to learn whether the tool helps under your constraints, and decide separately whether the task’s core judgment needs practice. If review costs erase the time saved, narrow the tool’s role or stop using it for that workflow. If the deliverable improves but unaided judgment matters too, retain opportunities to perform and explain that judgment yourself. That distinction keeps an attractive first pass from being mistaken for a completed, efficient, or expertise-building result.

Sources: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity; Does AI Assistance Enhance or Erode Expertise? Evidence from a Three-Month Field Experiment in Patent Drafting

What makes verification part of quality rather than an afterthought?

Review protects quality only when the reviewer can recognize consequential errors, consult evidence independent of the generated response, and correct what is wrong before the work is used. Reading a fluent answer twice is not an independent check. The reviewer needs source material, a test, a relevant record, or enough task knowledge to compare the output with the work's requirements. They also need time and authority to revise it. Without those conditions, review can create confidence without reducing the chance that a material error survives.

The study “The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects from a Survey of Knowledge Workers” offers evidence about how workers describe this shift. Its survey included 319 knowledge workers who shared 936 examples of generative AI use. Participants described critical work moving toward checking information, integrating responses, and stewardship. Higher confidence in the tool was associated with less reported critical-thinking effort. These self-rereports do not establish that AI caused lasting skill loss, reduced completed-work quality, or changed every worker's effort. They do show why evaluation belongs in the workflow: assistance can move effort from composing an answer to checking and incorporating it. That later work is part of the job, not optional polish.

A useful way to choose how much to delegate is to ask three questions. This editorial aid is not a validated score or safety test. First, can success be specified before the tool is used? “Summarize these approved policy changes in a short client note, preserve every date and exception, and use plain language” gives a reviewer observable requirements. “Tell us what the market should do” leaves the goal, evidence standard, and acceptable trade-offs unsettled. When success is vague, an agreeable answer can seem suitable because no one has defined what would make it wrong.

Second, can important parts be checked against something independent? For a document grounded in traceable facts, a reviewer can compare dates, amounts, quotations, and exceptions with approved records, then check whether the summary preserves their meaning. A draft may help because inputs and constraints are available and errors can be inspected. Now consider a strategic recommendation based on incomplete internal notes, informal conversations, and competing objectives. A polished rationale cannot verify missing context. If decisive evidence is scattered or known only through accountable colleagues, the reviewer must gather and weigh it directly; the model's confidence does not fill the gap.

The BCG consultant experiment described in “Navigating the Jagged Technological Frontier” shows why this distinction matters. In a randomized experiment with 758 consultants, GPT-4 users did better on 18 tasks judged within the model's capability frontier. On a separate, deliberately difficult business recommendation task outside it, control participants were correct about 84.5% of the time, versus 60% and 70.6% in the two AI conditions, an average decline of 19 percentage points. The task required combining quantitative information with important interview details; spreadsheet analysis alone could lead to the wrong recommendation. This was one tool, population, and designed task, not a general failure rate. It shows that checking a visible calculation differs from checking whether the analysis noticed the evidence controlling a decision.

Third, can detected errors be repaired within available time and authority? Identifying an unsupported claim helps only if someone can return to the source, revise the draft, and obtain sign-off before delivery. A worker may own the final recommendation yet lack access to relevant data or permission to change assumptions. Nominal human review cannot compensate reliably for weak inputs. A routine document with source files, clear ownership, and revision time may support a checked first draft. A decision with unresolved evidence or consequences the reviewer cannot authorize should keep its core judgment with responsible people; assistance can organize questions or alternatives.

The three questions suggest three levels of use. When purpose is clear, consequential claims are independently checkable, and there is room to fix errors, delegate a bounded first pass and review it against stated criteria. When only some conditions hold, limit assistance to options, an outline, or critique of a human-led draft; treat suggestions as leads to inspect, not conclusions. When decisive context cannot be checked or errors cannot be repaired in time, keep analysis and decision human-led. Assistance may still structure material, but it should not supply authority the workflow cannot verify.

Use this gate to plan a small task trial, not assign a permanent label. A successful check on one document does not establish that another task, tool, or workflow can be delegated equally. Nor does the survey show that verification automatically preserves quality or expertise. Ask whether review detects meaningful errors and whether the corrected work meets its requirements. If checks repeatedly uncover omissions the worker cannot resolve, narrow the use or return the central analysis to a human-led process. If evidence is accessible and repairs manageable, a bounded first pass may be reasonable. The framework guides a review plan; it is not a safety guarantee, validated score, or measure of an individual's job risk.

Sources: The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects from a Survey of Knowledge Workers; Navigating the Jagged Technological Frontier; Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

What is a proportionate next step?

Choose one recurring task you can reverse easily, such as turning supplied meeting notes into a client update. Before using AI, write the purpose and acceptance criteria: required facts, audience, decisions, and constraints. Keep a comparable unaided example and judge both outputs against the same standard. Stanford’s SCALE Initiative summary of a professional-writing experiment reports faster completion and higher evaluated quality for its tested occupation-specific assignments with ChatGPT access. Check important claims against source material. Record final quality, errors, corrections, and total time including review and repair. Ask whether you can explain and defend the result. If the assisted version meets the criteria and saves effort after checking, keep AI to that bounded first pass. If it produces options but the final choice depends on context you cannot verify, use it for alternatives or critique. If you cannot check or repair a consequential error in time, keep the core analysis human-led. “Navigating the Jagged Technological Frontier” found different effects across task types, so its result cannot decide this task for you. Repeat the comparison before extending assistance elsewhere; the trial answers only for this task, tool, and workflow. Use assistance where checked results meet your standard, and narrow it when they do not. For a task-level view of change pressure, try the free [AI task checker](/ai-job-risk-checker). Its signals are not a probability of job loss.

Sources: Experimental evidence on the productivity effects of generative artificial intelligence; Navigating the Jagged Technological Frontier

Questions readers ask