Why AI Chat Logs and Worker Surveys Classify Tasks Differently

Why the same work can appear differently in chat logs and worker surveys

AI chat-log studies and worker surveys can classify the same work differently because they observe different things and ask different questions. A log captures conversations on a service, then classifies their text into task labels. A worker survey can identify respondents’ occupations, ask which listed tasks they perform, and whether AI helps with them. The 2026 CEPR comparison links a worker survey to O*NET activities and compares it with task shares from Anthropic, Microsoft, and OpenAI chat data. Its authors report disagreement, in part because a classifier may see an action such as analyzing data without knowing the worker’s occupation or its purpose. The label can describe a generic activity rather than its occupational context. This mismatch is not evidence that one group of workers is more likely to lose jobs. Platform activity, reported workplace use, perceived potential, and modeled exposure answer different questions; none alone establishes employer adoption or displacement. When task maps conflict, check what each source observed, whom it includes, and how it assigned labels. Then compare those signals with recurring tasks and review demands in your own work, rather than treating divergent rankings as a personal risk score.

Sources: Measuring what work generative AI does: Survey evidence versus chat logs; What Work Does Generative AI Do?

What is actually being counted in a chat log and a worker survey?

A chat log and a worker survey begin from opposite ends of the question. A log samples conversations on a service and classifies their contents into task categories. A worker-linked survey begins with people: it can identify their occupation, ask which tasks they perform, and ask whether AI regularly helps with those tasks. The first describes activity in a product’s traffic; the second estimates reported use among workers doing a task. Neither measures an individual’s job-loss likelihood. In “Measuring what work generative AI does: Survey evidence versus chat logs,” the authors describe chat-based measures as the share of all sampled chats classified to a task. A task’s chat share is not the percentage of its workers who use AI: it reflects task prevalence and service use together. Logs omit people who never use that platform, so cannot separate those parts. The authors’ Real-Time Population Survey, by contrast, asks workers about task performance and regular AI help, making it possible to calculate adoption among task performers as well as a survey-based share of AI-assisted worker-task pairs. That survey has a defined frame rather than a complete record of every job activity. The CEPR account says the Real-Time Population Survey is a nationally representative online survey of U.S. adults that mirrors the Current Population Survey. Since August 2025, it has elicited each worker’s detailed occupation and shown the ten most important detailed work activities listed for that occupation in O*NET. Workers report which activities they perform and, if they use AI at work, which ones AI regularly helps with. The comparison pools four quarterly waves through May 2026, nearly 14,000 workers aged 18–64. For a chat corpus, the sampling frame is a service’s conversations; its unit is a classified chat; its denominator is chats in that corpus; and task attribution comes from interpreting conversation text. Non-users of the service are absent. It describes traffic composition, not worker adoption or each user’s occupation. For the worker survey, the frame is sampled workers; the unit can be a worker-task report; and the denominator can be workers performing that task. Non-adopters can be included, while occupation and task attribution are reported by respondents against the task list. Its valid inference is reported task use for the surveyed population and period, subject to the survey’s questions and coverage. “Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversations” maps more than four million Claude.ai conversations to U.S. O*NET tasks and occupations, showing detailed activity on that service. It remains a single-platform view, not workers identifying their own occupations and tasks. This survey design also presents only ten leading activities per occupation and records whether AI helps, not task time share. Before comparing a chart with a survey, ask whether its bars represent conversations or task-performing workers, and whether the source can see non-users.

Sources: Measuring what work generative AI does: Survey evidence versus chat logs; What Work Does Generative AI Do?; Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversations

Why can a popular chat task be rare in worker-linked task data?

A task's share of chats combines two things: how common that task is among workers and how often the people who perform it use AI in that service. A chat corpus contains conversations from that service; it does not include the workers who never use it. It cannot tell whether a task dominates because many people do it, its practitioners use AI especially often, or both. In “Measuring what work generative AI does: Survey evidence versus chat logs,” the authors use worker-linked survey data to construct a comparable measure: each task's share of all AI-assisted worker-task pairs. This aligns denominators more closely, but does not make the sources identical. Imagine, as an illustration rather than observed data, that preparing routine summaries is a task performed by many workers, while reconciling a specialized set of records is performed by fewer. If only a modest portion of summary writers use a chat service, their conversations could still make summaries a large category because the task is common. If most of the smaller group uses that service for reconciliation, that task could remain a smaller slice of all chats because relatively few people do it. This is why chat shares alone cannot distinguish prevalence from adoption among task performers: they describe a service's conversations, not workers across the economy. The CEPR authors' comparison shows that this denominator issue matters in the data they examined. After creating the survey analogue, they report that the largest task accounts for 15% to 23% of use in three chat-based series, while the largest task accounts for 4% in the survey-based series. The ten largest tasks account for 46% to 61% of chat use, compared with 22% in the survey. These are shares of AI use across classified activities, not percentages of workers, work hours, or tasks automated. The ranges reflect different chat sources, not uncertainty bounds or forecasts. The survey analogue is useful because the Real-Time Population Survey observes both whether a worker performs a listed activity and whether AI regularly helps with it. The authors can therefore combine task prevalence with reported use among its performers, then express the result as a share of AI-assisted worker-task pairs. This differs from asking which labels occupy the most conversations and supplies an adoption denominator absent from logs alone. But coverage is limited: respondents see ten important O*NET activities for their occupation, and report whether AI helps, not the task's share of work time. This is not a complete duty inventory or a measure of hours saved. So a large bar for a popular chat task should prompt a denominator question: among workers who perform this activity, what share report using AI for it? A high chat share alone does not show that most practitioners use AI, that AI performs most of the task, or that jobs are displaced. The survey narrows one mismatch by measuring prevalence and use together, but differences remain in platform users, sampled conversations, classification, and survey questions. Read the chart as activity within its dataset; seek worker-linked adoption evidence before claiming use is widespread among practitioners.

Sources: Measuring what work generative AI does: Survey evidence versus chat logs

How does missing occupational context turn one chat into the wrong job label?

When a classifier sees a request but not the worker’s occupation or purpose, it can attach a generic action to the wrong occupational task. The difference is between what someone asks a tool to do and what that action means inside their job. In “Measuring what work generative AI does: Survey evidence versus chat logs,” CEPR authors illustrate this with a professor asking for help analyzing trends in data. The conversation may be labeled “analyse data to identify trends,” an O*NET task associated with data scientists and analysts. Yet for the professor, the action may be part of “research topics in area of expertise.” Missing context can therefore mislabel both task and, when occupation is inferred from task, occupation. This example identifies a mechanism, not its frequency across platforms. It does not establish that every classifier makes the same error or how the output was used at work. A task label should be read as a classification, not a verified account of a worker’s job. Why can a task name mislead? O*NET describes work at different levels of abstraction. Some statements name a purpose tied to a field or responsibility; others capture a basic action found across jobs. The CEPR authors note that editing may be embedded in higher-purpose tasks such as preparing reports or drafting legal documents. A request to edit text identifies an operation, but not whether it serves a lawyer’s document preparation, a researcher’s report, or another responsibility. The wording can reveal the immediate action while leaving its work purpose unstated. This matters when researchers map task labels to occupations. A generic activity may be listed for occupations whose workers did not generate the sampled conversations. In the CEPR illustration, the professor’s request can be assigned to analyst work because that occupation lists data analysis, although the professor’s relevant task is framed around research in their field. This illustrates a possible mapping error, not a universal error rate. The 2025 paper “Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversations” shows both the value and boundary of granular mapping. It analyzes more than four million Claude.ai conversations through U.S. O*NET tasks and occupations, providing a detailed view of activity on that service. Its abstract describes a single-platform picture. The Anthropic paper also notes limitations in its data and methods. A conversation can show a request and response without showing how the output enters actual work. A request classified as drafting or analysis does not tell us whether a worker accepted, revised, or rejected it. Granular labels alone do not establish who performed a task or how it functioned in a job. When reading a task-to-occupation chart, ask how the study knows the occupation: was it reported by a worker, linked through workplace data, or inferred from text and task labels? Then ask whether its taxonomy names a generic action or the purpose that makes it part of a role. If the method does not explain those links, treat the result as model-based attribution, not a direct count of workers in that occupation using AI. For a worker, a task label is not a mirror of the whole role. A task classification can locate a possible connection between a tool and an activity, but occupational meaning depends on information outside the prompt: the worker, purpose, constraints, and responsibility for checking the result.

Sources: Measuring what work generative AI does: Survey evidence versus chat logs; Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversations

Why do the maps diverge even after the researchers make them more comparable?

The maps still disagree after comparison because harmonizing labels does not make the underlying observations equivalent. In “Measuring what work generative AI does: Survey evidence versus chat logs,” CEPR authors compare worker-linked reports with task shares from Anthropic, Microsoft, and OpenAI conversation data. Across 332 intermediate work activities, correlations between survey-based shares and the three chat series are 0.34, 0.10, and 0.11. These are not accuracy scores or job-loss estimates. The authors do more than place unlike charts side by side. The Real-Time Population Survey asks workers which listed activities they perform and whether AI regularly helps with them. The authors combine task prevalence and reported use to construct a survey analogue of chat share: each activity’s share of AI-assisted worker-task pairs. They compare that with each platform’s conversation distribution assigned to the same O*NET intermediate activity categories. The CEPR column and “What Work Does Generative AI Do?” discussion-paper record describe nearly 14,000 U.S. workers aged 18–64 across four quarterly waves through May 2026. Even after adjustment, the units differ. A survey response is a worker’s report about a task in their job; a chat record is an interaction from a particular service classified by content. It includes non-users and links activities to stated occupations. A chat classifier may not know who made a request or what responsibility it serves. The authors also report that chat sources correlate only 0.08 to 0.38 with one another across activities. Aggregation offers a partial explanation, not a cure. When the authors group the 332 activities into nine broad groups, correlations with the survey rise to about 0.6 for Anthropic and OpenAI. Broad buckets can absorb disagreements about neighboring or generic labels. Compression can hide different purposes within grouped activities. A broad category can make two sources look aligned while leaving unresolved whether they describe the same work in practice. The authors interpret some detailed mismatch as classification noise, while concluding much disagreement remains. The change in correlation does not show how much of each gap has a particular cause. Other differences persist: platform selection, users choosing each product, conversation sampling, classification rules, observation period, and whether a task is self-reported or inferred from text. Shared O*NET categories provide a common vocabulary; they do not standardize those inputs or make occupation context visible in a conversation. It shows that rankings vary materially under the study’s definitions and data; the CEPR discussion paper describes chat measures as conceptually different from worker-linked adoption measures. For a reader, the conclusion is not to average correlations into a blended occupation ranking. They describe disagreement among aggregate measures, not an individual worker’s risk. Inspect each source’s population, denominator, classification, and time window. A chat map indicates what appears in sampled product traffic; the survey estimates reported use among workers performing listed activities. Neither establishes that an employer adopted a workflow, a task will be removed, or a person’s job will disappear. The remaining divergence is a reason to keep source boundaries visible when interpreting a task map, especially when a chart labels occupations from task text rather than observed worker roles.

Sources: Measuring what work generative AI does: Survey evidence versus chat logs; What Work Does Generative AI Do?

Why do reported use and perceived potential produce different answers?

A worker can report little or no current AI use while believing some work could be done with AI because these are different questions. One concerns reported use; the other concerns perceived applicability. Pew Research Center’s October 2024 U.S. worker survey illustrates the gap. Its findings describe that survey and period, not a current platform-use estimate or direct comparison with chat logs. Perceived possibility does not establish that a tool is available, approved, reliable, or used at work.

Pew’s report, “Workers’ exposure to AI,” found that 63% of employed respondents said they used AI little or not at all in their job, while 16% said at least some of their work was done with AI. They indicate reported experience, not a measured share of all tasks performed with AI.

Among non-AI users, defined as workers who said they used AI little or not at all at work or had not heard about AI use there, 31% said at least some of their work could be done with AI. That denominator is this non-user group, not all workers or everyone represented in a chat corpus. Another 45% of these non-users said not much or none of their job could be done with AI; 21% had not heard of workplace AI use. These varied perceptions do not identify which tasks a system can perform accurately or what an employer has deployed.

The Pew Research Center’s “Methodology” says Wave 157 of its American Trends Panel was fielded October 7–13, 2024, among U.S. adults working full or part time for pay. Of 6,490 sampled panelists, 5,395 responded. Interviews were online or by telephone, in English or Spanish. These self-reports support estimates of workers’ answers, not observation of task performance. Pew notes that wording and practical difficulties can introduce survey error or bias.

A worker may see a plausible use for summarizing correspondence yet not use AI because no tool is approved, material is sensitive, or checking takes too long. This illustrates why possibility and use can differ; it is not a claim about frequency. Another worker may use an approved tool for one task without thinking much of the job could be done with AI. A response about possibility is not a capability test, deployment record, or proof of occupational adoption.

This distinction helps explain apparent disagreement with chat-log classifications. A log records activity represented in a selected service; a survey about perceived potential can include non-users and may not name the same task categories. Even survey answers about current use differ from logs in coverage and reporting. One describes what workers say they do or think possible; the other classifies conversations in its observed corpus. Pew’s dated figures cannot validate every platform’s task labels.

When studies seem to conflict, inspect the question and denominator. Did people report current use, workplace awareness, perceived possibility, or use for a specified task? Were results calculated among all employed respondents, non-users, or performers of a named activity? Perceived possibility is a signal to investigate, not proof of capability or an employer decision. Compare it with a recurring task in your setting, including whether a tool is permitted and how its output is reviewed.

Sources: Workers’ exposure to AI; AI and workers: Methodology

What can a worker-linked survey still miss?

A worker-linked survey improves task attribution without becoming a complete record of work. In “Measuring what work generative AI does: Survey evidence versus chat logs,” the Real-Time Population Survey analysis links respondents to occupations and shows them the ten most important O*NET activities for each. Workers report which listed activities they perform and, if they use AI at work, which ones AI regularly helps with. This lets researchers estimate reported use among task performers, including people who do not use AI. It does not measure every duty, time spent on each duty, or every instance of assistance. This is a U.S. survey, not a census of all workplace activity. The task list sets a boundary. Occasional, emerging, or lower-priority activities may not appear among an occupation’s ten listed activities. A worker may also do something differently from the standard description attached to their occupation. When asked about a named activity, respondents must decide whether their own work fits that wording. Answers can therefore depend on memory and interpretation as well as use. These are reasons to read the measure carefully, not proof that people report inaccurately; the CEPR comparison does not quantify such effects. Occupation detail brings another trade-off: the analysis cannot resolve small occupations, so broad estimates may be more stable than fine-grained claims about small groups. And asking whether AI “helps” with a task does not show how often it helps, how much work changes, whether outputs are accepted, or whether an employer has formally adopted a tool. The authors distinguish adoption from time allocation: the survey asks whether workers perform an activity and whether AI regularly helps, not what share of work time it consumes. Chat logs have a different strength and blind spot. “Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversations” describes records that show interactions at scale within Claude.ai and can be classified into task categories. For activity appearing in the sampled service data, these are records of interactions rather than workers’ recollections. But a log does not automatically reveal a user’s occupation, the workplace purpose of a request, activity on other services, or people who never use that product. It cannot provide the non-adopter denominator needed to estimate use among all workers performing a task. The sources therefore answer complementary questions. Worker-linked surveys are better suited to estimating how many people performing a listed task report using AI for it, within the survey’s population and design. Platform logs help examine what users bring to a particular service and how those interactions are classified. Neither is ground truth against which the other can simply be scored. As “Measuring what work generative AI does: Survey evidence versus chat logs” notes, survey task coverage and small-occupation limits coexist with chat logs’ scale and missing worker context. Read together, they illuminate different parts of work; neither alone establishes how a job will be redesigned or whether a worker will be displaced.

Sources: Measuring what work generative AI does: Survey evidence versus chat logs; What Work Does Generative AI Do?; Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversations; AI and workers: Methodology

Why are exposure indexes a third kind of evidence rather than a tie-breaker?

Exposure indexes answer a different question from chat logs and worker surveys: given a task description and an account of AI capability, how much might the technology assist or complete? They organize evidence about technical applicability, but cannot break a tie between measures of use. A high exposure category is not a probability that a worker will lose a job. The International Labour Organization’s “Generative AI and Jobs: A Refined Global Index of Occupational Exposure,” published in 2025, combines task data, worker input, expert discussions, and AI predictions. A survey of 1,640 employed people assessed automation potential for 2,861 tasks; researchers compared responses with experts and used AI to predict scores for tasks in the international occupational classification. This is modeled capability evidence, not a record of deployments. Global estimates do not establish employer adoption, role redesign, or displacement. The U.S. Bureau of Labor Statistics (BLS) keeps theoretical fit and observed use distinct. Its “Artificial Intelligence (AI) exposure categories” combines five sources: three theoretical measures and two based on observed AI use mapped to occupational tasks. BLS converts scores to within-source percentile ranks, imputes some missing values, then clusters occupations using separate theoretical and current-evidence dimensions. The categories compare occupations; they do not show directly whether workers used AI on the job. BLS says they are not employment or wage forecasts, adoption probabilities, productivity estimates, or replacement estimates. Results depend on source selection, crosswalks, imputation, normalization, and clustering. Exposure indexes can identify tasks worth examining. But an index is not a referee between a platform’s conversation share and a survey’s reported adoption. Those instruments observe different events and populations; capability evidence cannot supply the missing workplace-use denominator. The ILO’s 2026 brief, “New ILO brief explains what AI exposure indicators reveal about jobs,” says exposure measures rely on static task descriptions, omit economic feasibility and adoption constraints, and reflect measurement assumptions. They are signals of where work may change, not predictions. Technical capability means a system can perform or assist a task under defined conditions; exposure estimates how applicable that capability may be; observed use records activity in a setting; adoption concerns whether an organization puts a tool into work. Demand is employers’ need for labor, while displacement is an employment outcome. An index alone does not bridge these steps. Use indexes to identify recurring tasks worth checking, then ask whether approved tools are used at work, who reviews outputs, and what errors would cost. If an index and survey disagree, keep each attached to its own question. Neither is a personal risk score or proof that a job is safe or doomed.

Sources: Generative AI and Jobs: A Refined Global Index of Occupational Exposure; Artificial Intelligence (AI) exposure categories; New ILO brief explains what AI exposure indicators reveal about jobs

How should you use conflicting task signals in a real career decision?

When task maps disagree, identify what each measures, then check your recurring work. A chat share, worker-reported use, and modeled exposure prompt different checks; they are not personal risk scores. The CEPR analysis, “Measuring what work generative AI does: Survey evidence versus chat logs,” shows why source and denominator matter; the Bureau of Labor Statistics’ “Artificial Intelligence (AI) exposure categories” warns that its categories are not replacement estimates. Anthropic’s Claude conversation study also cautions that platform activity alone cannot show how outputs enter work practice. First, note population and date: survey workers, users of a service, or occupations represented in a model. Next, identify the unit and denominator. Is the figure a share of conversations, workers reporting use, task performers reporting help, or an exposure rating? Then ask how occupation was established: linked to worker data, self-reported, or inferred from conversation text. Check which task labels were used and whether results are detailed or grouped. Finally, distinguish current use from perceived potential and modeled capability, and ask whether non-users or people using other tools are visible. These checks bound the number; none establishes employer adoption or job removal. Consider an example: a recurring client update involves gathering approved material, drafting a summary, checking figures against records, and answering follow-up questions. A chat classification might capture drafting or summarizing; a worker survey may ask whether AI helps with a listed task in the occupation. An exposure index may assess whether parts of those tasks appear technically applicable. These descriptions can differ without documenting the whole workflow. Record how often each part occurs, which tools the workplace permits, whether sensitive information is involved, how much review is needed, and what an error would affect. The classification starts an inventory; it does not deliver a job verdict. If policy permits, choose one low-stakes step, such as testing an approved tool on non-sensitive material or learning to verify its output. Compare it with your usual process: did it save effort after checking, and was quality adequate? If review erases the time saved, or the task carries consequences you cannot safely verify, that is useful evidence against using the tool there. This check does not prove broader adoption or justify buying a general credential or abandoning experience. Aggregate disagreement should prompt a closer look at tasks, permission, review burden, and error stakes. A larger career decision needs evidence about your own constraints and options.

Sources: Measuring what work generative AI does: Survey evidence versus chat logs; Artificial Intelligence (AI) exposure categories; Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversations

What is the next useful check?

Choose one recurring task and verify what happens in your workplace before concluding. The CEPR comparison, “Measuring what work generative AI does: Survey evidence versus chat logs,” shows whether signals come from platform conversations or worker-linked reports. The Bureau of Labor Statistics’ “Artificial Intelligence (AI) exposure categories” adds a different lens: modeled task applicability and observed-use inputs, combined into relative occupation categories. Neither estimates your personal displacement risk. Write down the task, how often it occurs, whether your employer permits an AI tool, what review the result requires, and what an error would cost. If use is allowed, a low-stakes trial can test whether it saves time after checking and correction. If use is not allowed, ask what approved process is appropriate; do not put sensitive work into an unapproved service. Keep the decision reversible until you have evidence about quality and review burden. The free task checker can organize change-pressure signals around your work, but it is not a validated displacement probability. If you are weighing a stay-and-upgrade path against an adjacent move or larger change, the paid career roadmap can compare those scenarios with your constraints; it does not guarantee employment or income. Start with the task you can observe, then decide whether a small workflow change is justified.

Sources: Measuring what work generative AI does: Survey evidence versus chat logs; Artificial Intelligence (AI) exposure categories