What the Generative AI Education-Gap Study Shows About Unaided Performance

What does the study show about unaided job performance?

The experiment found that access to generative AI raised average scores on a short business task, while a modest signal of improvement remained on an immediate, related follow-up completed without the assistant. That follow-up is the study’s measure of “unaided” performance: participants answered new prompts about the same scenario after tool access was removed. It does not measure sustained performance across a job, later retention, or a worker’s likely career outcome. The paper, [“Does generative AI narrow education-based productivity gaps? Evidence from a randomized experiment”](https://arxiv.org/abs/2608.04198), is by Cruces, Fernández Meijide, Galiani, Gálvez, and Lombardi. Stanford SCALE catalogs it; that catalog listing does not mean Stanford conducted the experiment. The result is evidence about one controlled task and its near-term follow-up, not a forecast of who will perform well at work.

Sources: Does generative AI narrow education-based productivity gaps? Evidence from a randomized experiment | arXiv

Who ran the experiment, and what did participants do?

The findings come from an online randomized experiment conducted in Argentina from September through November 2025, rather than a study of workers observed inside their employers. The World Bank author presentation reports that 1,795 eligible adults were assigned to the experiment and 1,174 completed it. Completers were ages 25–45 and recruited through survey-panel companies. The researchers divided participants using education thresholds for this study: the lower-education group had a high-school education or less than half of postsecondary study completed; the higher-education group had completed more than half of postsecondary study. These labels describe the study’s grouping rule, not a universal measure of occupational skill or individual ability.

Each participant received a self-contained business scenario with a boss’s email and a report containing text, a figure, and a table. The task was to diagnose the situation and email a proposed solution. The scenario varied across three business contexts, was designed to be answerable without specialist industry knowledge, and included performance incentives. Participants were randomly assigned either access to a GPT-4.1 assistant embedded in the exercise—with the report already supplied to it—or to a control condition without that assistant. After the main task, participants answered related questions with no assistant available, including an open-ended explanation of the root cause, graph interpretation, and factual recall, as described in the [World Bank author presentation](https://thedocs.worldbank.org/en/doc/5fa3b40f263a4a22e1572954980189c9-0370012026/original/5-Does-Generative-AI-narrow-skill-productivity-gaps.pdf).

Random assignment makes access to the assistant the designed difference between the treatment and control groups, so their average contrast can support a causal estimate of access for this exercise and these participants. Education was not randomly assigned: it was an observed characteristic used to compare groups, so those comparisons cannot establish that education itself caused any performance gap. The setup has a useful strength: participants had to work through a concrete, incentivized problem rather than answer an abstract question about AI. Its boundary is equally concrete. A brief online scenario with all relevant material supplied does not reproduce a continuing role, employer workflow, or the coordination and responsibility that can shape work beyond an individual written response. Nor does panel recruitment make the completed sample a representative cross-section of Argentina’s workforce. The difference between those assigned and those who completed the exercise matters when generalizing: the estimates describe the observed experimental participants, and similar attrition across study groups cannot by itself show that non-completion introduced no selection. The design therefore gives a focused comparison of assistant access under common task conditions, while its population and setting leave open how the same tool would affect different occupations or workers facing other constraints. The [Stanford SCALE catalog entry](https://scale.stanford.edu/ai/repository/does-generative-ai-narrow-education-based-productivity-gaps-evidence-randomized) helps identify the paper, while its record and the paper’s author information show why the Stanford label should be read as a catalog location rather than the experiment’s institutional attribution.

Sources: Does generative AI narrow skill productivity gaps? Evidence from a randomized experiment | World Bank author presentation; Does Generative AI Narrow Education-Based Productivity Gaps? Evidence From A Randomized Experiment | Stanford SCALE Initiative

What did the study count as better performance?

In the World Bank author presentation for “Does generative AI narrow skill productivity gaps? Evidence from a randomized experiment,” the main task outcome is a preregistered score from 0 to 10. Two-thirds of the composite reflects content and one-third writing. The presentation says scores were standardized relative to the lower-education control group, so the reported standardized results express differences on a common scale built from that group’s score variation. This is a measure of how participants’ diagnosis-and-solution email performed under the study’s rubric; it is not a direct count of tasks completed, minutes saved, or business value created.

The reported scoring process offers several checks on whether results depended on a single grading pass. The presentation says GPT-5 graded responses over ten iterations, with the resulting scores averaged. The researchers also manually graded a random 10% sample and checked robustness across iterations, models, and alternative grading procedures; the slides report a correlation above 0.9 with manual scores. Together, these procedures support consistency between the reported scoring approach and the comparison checks. They make it less plausible that the group pattern arose solely from one arbitrary model output or one unstable run. They do not turn the rubric into an independent measure of what a manager, customer, or organization would regard as a successful decision.

That distinction matters because a response can be clear, well organized, and persuasive while still resting on a mistaken diagnosis or an untested assumption. The presentation describes a supplied scenario and scores its written solution; the score does not establish that participants’ proposed actions were implemented, that the underlying business problem was correctly resolved in a live operation, or that the decision produced a beneficial downstream result. Those outcomes require other observations, such as checking factual and analytical correctness against a defensible answer standard or observing consequences after a decision is carried out. The study’s score therefore captures performance on the defined deliverable, rather than every condition that determines whether work succeeds in practice.

The weighting also affects what a change in the total can mean. Because content and writing contribute in fixed proportions, the composite can register a stronger total even if improvement is concentrated in one component, or if both move by different amounts. By itself, it cannot tell a reader whether access mainly helped participants identify the business issue, select a sound recommendation, explain it more effectively, or combine several of those improvements. Component-level results would be needed to attribute the composite change to a particular part of the work. So “better performance” here means a higher average score on a combined content-and-writing rubric—not proof of greater correctness, better judgment, or higher productivity across a role.

The manual comparison, alternate grading procedures, and repeated scoring strengthen confidence that the measured pattern was not merely an artifact of one scoring run. Their reach is narrower than criterion validation: consistency in applying this rubric does not establish that it predicts business outcomes or measures performance across an occupation.

Sources: Experimental evidence on the productivity effects of generative artificial intelligence | Science

How much did AI access change the assisted task result?

The World Bank author presentation reports two different kinds of comparison for the assisted task. Within the lower-education group, assignment to assistant access raised the average score by 1.242 standard deviations relative to the no-assistant control; within the higher-education group, the corresponding access effect was 0.834 standard deviations. These are treatment contrasts: each compares people assigned to access with people in the control condition inside the same education category. They are not the education gap, and they do not mean that every participant gained by the group average.

The education-group gap asks a separate question: how far apart were the two groups’ average scores under a given condition? In the control condition, the lower-education group’s average was 0.548 standard deviations below the higher-education group’s average. Among participants with assistant access, the estimated gap was 0.139 standard deviations. The presentation describes this as a reduction of about 74% in the gap. That percentage compares the size of these two estimated between-group gaps; it is not a 74% increase in anyone’s performance, nor does it mean the original difference disappeared. The point estimate with access still places the higher-education group above the lower-education group on the study score.

A standard deviation is a unit for expressing a difference relative to the spread of scores in a reference distribution. Here it helps put score contrasts on a scale that can be compared across the study’s education groups. It is not a percentage: +1.242 standard deviations does not mean a 124.2% improvement. Nor does a standardized score tell us how many hours a worker saved, how much revenue changed, or how the result would translate into a particular organization’s performance measures. To make such claims, the experiment would need to measure those outcomes directly. Its reported estimates concern average rubric scores on the bounded exercise described earlier.

The larger estimated access effect for the lower-education group is consistent with stronger average catch-up on this task: the control-condition gap was substantially smaller among those assigned access. But the estimate describes group averages under this experiment’s assignment and scoring design. It does not show that each lower-education participant benefited more than each higher-education participant, or that education itself caused the different treatment estimates. People within either category can vary in their starting knowledge, task approach, use of the assistant, and score response; an average contrast cannot resolve that individual variation.

For the remaining assisted gap, the presentation reports p=0.110. At conventional significance thresholds, that result means the analysis did not reject a zero gap. It is not evidence that the groups were equivalent, and it does not erase the positive 0.139-standard-deviation point estimate. Equivalence requires a defined range of differences considered practically negligible and an analysis designed to test whether the estimate is sufficiently precise to fall inside that range. A failure to reject zero can instead reflect uncertainty wide enough to include both little difference and a more meaningful one. The proper reading is therefore substantial estimated convergence on this score, alongside an unresolved question about the size of any residual gap.

Taken together, the figures support a bounded claim: access improved mean task scores within both education groups, and the estimated gap between their means narrowed considerably under access. The within-group gains and between-group gaps answer different questions, so they should not be collapsed into a single claim that AI eliminated an education-based productivity difference. The evidence establishes neither a universal individual benefit nor equivalence between groups; it quantifies average outcomes on this particular assisted deliverable.

Sources: Generative AI and jobs: A 2025 update | International Labour Organization

What remained when the assistant was removed?

The World Bank author presentation reports a small positive immediate follow-up estimate for the lower-education group, a less precise positive estimate for the higher-education group, and a statistically detectable gap between treated groups. The questions came after the business task, with the assistant unavailable: participants gave an open explanation of the root cause, interpreted a graph, and recalled facts from the supplied scenario. This is the experiment’s direct evidence about what participants could answer unaided immediately after assisted work. It is a narrow test of near-term performance on related prompts, rather than a measure of the original email alone.

For participants in the lower-education group, assignment to assistant access was associated with a 0.171-standard-deviation higher follow-up score than the no-assistant control, with p=0.049, according to the World Bank author presentation. The estimate is modest in size. Its p-value sits just below the conventional 0.05 threshold, so the result is threshold-sensitive: a small change in specification, sample, or uncertainty could change whether it crosses that cutoff. The finding is evidence of a limited positive average difference on these immediate questions under the reported analysis. It should not be inflated into a broad claim that access taught a large new skill, nor dismissed as if the estimate were zero simply because it is small.

For the higher-education group, the corresponding estimate was +0.071 standard deviations, with p=0.375. The positive point estimate alone does not make a reliable case for improvement: the reported result is uncertain and does not reject a zero effect at conventional thresholds. That uncertainty also does not establish that assistant access had no effect for this group. A p-value is not the probability that the effect exists, and the estimate should be read with its precision rather than turned into a yes-or-no verdict about learning.

Among participants assigned to the assistant condition, the presentation reports that the higher-education group still scored 0.200 standard deviations above the lower-education group on follow-up, with p=0.018. Thus, the measured group separation persisted on these prompts even after the substantial convergence described for the assisted task. This is a between-group contrast among treated participants; it is not another estimate of how much access changed either group’s score. The two treatment estimates and this remaining gap answer different questions. Taken together, they show a modest near-term lower-group signal alongside continuing measured separation, not the disappearance of the education-group difference.

It would be tempting to compare p=0.049 with p=0.375 and conclude that access helped the lower-education group more. That comparison is not a test of the difference between the groups’ treatment effects. One estimate crossing a significance threshold while the other does not cannot establish that the estimates differ; the reported point estimates alone do not settle that question either. A direct interaction or contrast with its uncertainty would be needed to support such a claim. Likewise, the remaining treated gap does not show that the lower group failed to benefit: group gaps and within-group access effects are distinct quantities.

The timing and content set the boundary on what this result can establish. Because follow-up came immediately and concerned the same scenario, participants may have carried forward facts, reasoning, or structure encountered while preparing their response. The observed answers are consistent with some immediate carryover, but the test cannot separate retained understanding from short-lived familiarity with the materials or task. Nor does it show whether participants would remember the explanation days later, apply it to a changed problem, repeat the reasoning reliably, or perform independent work over time. Those are different outcomes that this immediate module did not measure. The careful conclusion is therefore neither that assistance necessarily leaves people worse nor that it produces durable, transferable competence: for these related, no-tool prompts, the study reports a modest signal for one education group and a persistent group gap, with longer-run retention and transfer unresolved.

Sources: Does generative AI narrow skill productivity gaps? Evidence from a randomized experiment | World Bank author presentation

What might the follow-up say about engagement, and what does it not prove?

The arXiv abstract reports a pattern linking intensive assistant use and sustained effort to the study’s two outcomes: intensive use raised assisted performance regardless of effort, while improved follow-up performance appeared only when intensive use coincided with sustained effort. This is a clue about variation in what happened around tool use, not a standalone demonstration of how learning occurred. In particular, the abstract reports the pattern at a summary level; it does not provide enough detail to reconstruct the exact measure of “intensive” use or “sustained” effort, or the analysis that produced the interaction. The result should be read at that level of specificity. It does not show that every intensive user engaged deeply, or that every sustained effort produced an advantage.

One reasonable interpretation is that the immediate no-tool responses may have depended partly on continued engagement with the task while using the assistant. A participant who remains involved in diagnosing the scenario, checking the material, and following how a proposed answer is formed could plausibly carry more of that reasoning into the next prompt than someone who treats the assistant’s output as a finished deliverable. This is a possible explanation of the reported pattern, not an observed account of each participant’s process. The abstract does not establish that those particular actions defined effort, that intensive use was experimentally assigned, or that engagement caused the follow-up difference.

The distinction between assisted and follow-up performance is useful here. Intensive use was associated with higher performance on the assisted task across effort levels, according to the abstract; the reported improvement on follow-up was more conditional. That makes it unsafe to infer that a strong assisted result automatically reflects what the participant can later explain unaided. It also does not license the reverse conclusion that heavy tool use prevents learning. The follow-up result appeared in one reported usage-and-effort pattern, and the available abstract does not tell us enough about the comparisons, uncertainty, or coding to quantify how much that pattern should change expectations.

Effort, in this account, is not a guaranteed recipe. The reported association cannot show that telling workers to try harder would cause them to retain more, that effort alone would be sufficient, or that the same approach would transfer to another occupation or task. People may differ in prior knowledge, available time, task familiarity, and reasons for using the tool; without the detailed methods and a design that isolates these factors, the effort label cannot carry a causal prescription. The authors’ abstract supports reporting the interaction it describes, while a stronger claim about mechanism would require evidence beyond that summary.

To change the interpretation from a short-run clue to a claim about learning, further observations would need to test different horizons and outcomes. Delayed tests could show whether any advantage remains after the immediate encounter. New tasks could test transfer beyond the supplied scenario; repeated practice could show whether performance changes across attempts; and workplace outcomes could reveal whether unaided explanation corresponds to recurring work quality or judgment in context. These are examples of evidence that would answer unanswered questions, not studies this paper is reported to have conducted. Until those measures are available, the effort-related result can guide what researchers should distinguish and measure, but it cannot establish durable learning or prescribe a learning method to workers.

Sources: Does generative AI narrow education-based productivity gaps? Evidence from a randomized experiment | arXiv

Why do assisted productivity results not settle unaided capability?

A separate writing experiment shows why an assisted result should be read on its own terms. In “Experimental evidence on the productivity effects of generative artificial intelligence,” published in Science, 453 college-educated professionals were assigned occupation-specific writing tasks with or without ChatGPT. The researchers reported 40% lower average completion time and 18% higher rated output quality for participants with access. Those outcomes describe the speed and assessed quality of work produced with assistance. They make a meaningful point in the other direction from a narrow substitution story: in this experimental setting, assistance improved the completed professional-writing artifacts rather than merely removing work from the process.

That finding does not answer what participants could later do with the tool closed. The Science study’s reported outcomes were time to complete and rated quality of the assisted writing task; it did not test a subsequent unaided response of the kind included in “Does generative AI narrow education-based productivity gaps? Evidence from a randomized experiment.” The latter’s immediate follow-up asks what participants could answer without the assistant after working through a related business scenario. Time, quality ratings, and responses to later no-tool prompts are different outcomes. A faster, better-rated document can coexist with uncertainty about retention or independent explanation; conversely, a short follow-up score does not tell us whether the assisted document was useful at work.

The studies also cannot be ranked by comparing their headline numbers. The Science experiment involved college-educated professionals doing occupation-specific writing, whereas the Argentina study used a different participant group and a bounded business diagnosis task. Their tools, task conditions, outcome rubrics, and units differ: percentages for average time and rated quality do not share a scale with standardized scores on an experimental rubric. The comparisons answer separate questions about access under their own designs. Neither is a direct measure of sustained performance across a job. This is why the positive writing results should inform the interpretation without being treated as a replication, contradiction, or numerical benchmark for the other study’s follow-up.

Workplace conclusions sit another step away. The International Labour Organization’s “Generative AI and jobs: A 2025 update” uses a task-level occupational exposure framework; that is a separate construct from experimental performance under assigned tool access. Exposure describes which kinds of tasks may be affected by generative AI, not whether an employer has adopted a tool, how a particular job will be redesigned, how labor demand will change, or whether an individual will be displaced. An experiment can show a capability or access effect in a defined task. It cannot, by itself, establish those labor-market transitions. Adoption depends on organizational choices and workflow conditions, while labor demand reflects forces beyond a task’s technical exposure; those questions require workplace or labor-market evidence. The ILO framework can help name the broader task-level question, but it does not convert either experiment into a job-loss probability or a verdict that an occupation is safe or unsafe. Each claim needs evidence suited to its own level.

Sources: Experimental evidence on the productivity effects of generative artificial intelligence | Science; Generative AI and jobs: A 2025 update | International Labour Organization

What can a worker reasonably take into a career decision?

A worker can use the difference between assisted output and unaided explanation to frame one small, practical inquiry, not to diagnose their overall capability. Choose a recurring task that is safe to inspect and follow workplace rules for data and AI use. If using an assistant is permitted, complete the task and then, with the tool closed, explain the evidence, assumptions, and checks behind the result. Note which part was delegated, which part the tool helped develop, and where judgment or verification remained necessary. This exercise is a reasoning aid suggested by the contrast between study outcomes; it is not a tested intervention, a skills assessment, or a forecast about the occupation. One attempt may identify a question to investigate, such as what you need to learn or verify more reliably, but it cannot establish employer adoption or the future of your role.

For a broader task inventory, the free checker at /ai-job-risk-checker offers task-level change-pressure signals, not a validated probability of displacement. If comparing possible next steps against constraints such as experience, income needs, location, or available learning time, the paid career roadmap compares scenarios but does not guarantee employment or income. Start with the recurring task you can actually examine, and treat what you learn as one input to a larger decision.

Questions readers ask