What to Record When Testing AI-Assisted Work Quality

What should you record to judge AI-assisted work fairly?

To judge AI-assisted work fairly, record comparable episodes from one recurring task and give both workflows the same acceptance bar. Note the task’s purpose and stakes, process used, and relevant tool or workflow version. For each episode, record whether the first result met the agreed checks, verification and correction needed, final acceptance, and any repair after handoff. Keep checking time, correction time, total active effort, and downstream rework distinct; note who reviewed the work and who carried out repairs. The result is a signal about this task under these conditions, not a verdict about AI generally or your job security.

A draft is not the same as accepted work. Someone may need to check sources or figures, correct omissions, rerun a step, or respond to a later problem. If the log records only first-output time, those efforts disappear and a polished result can look cheaper than it was. If the acceptance standard changes between ordinary and assisted episodes, the comparison is hard to interpret. Agreeing on the standard first lets you see whether quality held, effort moved to another stage, or someone else absorbed repair.

NIST AI 600-1, the Generative AI Profile, advises evaluating performance under deployment-like conditions and documenting them; it cautions against broad conclusions from narrow or anecdotal assessments. The NIST AI RMF Playbook prompts teams to document why they chose measures, how tests can be repeated, and who oversees errors. This is organizational guidance, not a validated personal diary or a prescribed sample size. A concise, policy-permitted record can support a practical choice: keep the workflow, narrow its use, change review, or pause it, then bring the evidence to the person who sets the quality standard.

Sources: Generative Artificial Intelligence Profile; NIST AI RMF Playbook: Measure

What does ‘good work’ mean for this task?

Before comparing ordinary and AI-assisted work, write down what an acceptable deliverable must contain and what would make it unusable. Make the boundary specific enough that reviewers can identify the same requirements, even if they phrase feedback differently. Name who accepts the work and the consequence of a miss: a routine edit, delayed handoff, returned deliverable, or escalation. These are different outcomes, not one general label such as ‘needs improvement.’ Start with the deliverable’s purpose. For a summary supporting a client decision, checks might include whether claims can be traced to supplied sources, the requested decision is stated, and material limits remain visible. For a spreadsheet update, they might be whether required fields are complete, values match the authoritative input, and the result works in its destination. These are illustrations, not universal rubrics. Define checks for the actual task and the person accountable for using it. The NIST AI Risk Management Framework Playbook advises documenting selected performance measures and why they fit; the Generative AI Profile says evaluation criteria should reflect and be documented for conditions similar to deployment. This supports a practical rule: define the measure before seeing which workflow appears better. NIST does not provide a ready-made checklist for every occupation or establish that one rubric makes a workflow effective. The worker and reviewer still need to identify what matters in their task. Separate must-pass requirements from preferences. A factual error, missing required field, or absent source reference may make an item unacceptable even if it is polished. A preference about tone or layout may be negotiable if the content remains usable. Agree in advance whether a defect is disqualifying, repairable before handoff, or acceptable with a note. This keeps an improvement on an easy-to-count feature from hiding a more consequential failure. It also distinguishes a first-pass miss caught in review from an error passed to the next person. Do not treat a convenient proxy as the whole definition of quality. Word count can show length, elapsed time can show duration, and reviewer satisfaction captures a response, but each can miss accuracy, completeness, fitness for destination, or dependability. The NIST AI Measurement and Evaluation resource emphasizes evaluating systems in context. For a worker’s log, choose observable checks tied to purpose instead of assuming one score represents every quality dimension. Keep the checks few enough to apply consistently, while retaining consequential requirements as separate gates. Stakes determine how to use the criteria. If a miss could cause a materially wrong decision or require escalation, treat the check as a threshold that speed cannot offset. If the issue is a revisable style preference, it may count as a correction without failing the entire episode. The boundary depends on the task and accountable reviewer; it cannot be set for an unknown workplace. Write criteria before logging episodes, use them for both workflows, and preserve the rubric version. If a reviewer changes the standard, or the task’s destination or stakes change, note the date and start a distinct comparison period. Otherwise an apparent quality difference may reflect a moving acceptance bar. A stable rubric makes first-pass and final acceptance interpretable, though it does not turn a local record into causal proof. Ask the person who accepts the deliverable which checks are mandatory, which are preferences, and what miss requires correction before handoff.

Sources: NIST AI RMF Playbook: Measure; Generative Artificial Intelligence Profile; AI Measurement and Evaluation

What belongs in each task episode record?

Use one row for each completed episode in the task family you are comparing. Give it a date and a short task code, then record whether the ordinary or AI-assisted workflow was used, the stakes or complexity category, and the tool or workflow version. Add the acceptance checks that applied, the first-pass result, any defect category, verification and correction time, total active time, elapsed time if delay matters, final status, and any later rework. Note the reviewer or handoff and unusual conditions.

Keep the unit of observation consistent. Record outcomes against that unit: for example, whether the episode passed its required checks on first review, whether it was accepted after correction, and whether later repair was needed. If you calculate a rate, write down its denominator. Mark work that was abandoned, escalated, or never reviewed instead of counting it as an ordinary completed case.

Separate time into categories that answer different questions. Active drafting or execution, verification, correction or reruns, and total active effort should not be collapsed into one number. Waiting for review is elapsed time rather than labor, but it may matter if the task has a service deadline or a dependent handoff. If another person repairs the output later, record that rework when the information is available and identify who performed it by role or team where permitted. Otherwise, a first-pass time saving can look larger simply because some of the work moved beyond the original worker’s clock.

Make the row reconstructable, not invasive. Use a coded identifier rather than copying client records, unpublished business material, personal data, or a full prompt into a private spreadsheet. Keep only details allowed by workplace policy, and use an approved work log if one exists. The NIST AI 600-1 Generative AI Profile calls for documenting evaluation conditions and cautions against extrapolating from narrow, anecdotal assessments; for this comparison, that supports recording context.

A practical row can therefore be compact: date; task-family code; condition; stakes or complexity; workflow version; required checks; first-pass result; defect and detection stage; verification minutes; correction or rerun minutes; total active minutes; elapsed time if relevant; final acceptance; later rework; reviewer or handoff; and a short context note. NIST’s AI Risk Management Framework Measure Playbook asks organizations to document why metrics were selected, how tests can be repeated, what errors or complaints are tracked, and who oversees them. They suggest asking what the measure means, whether conditions can be recognized later, and who owns the final decision.

Do not add a weighted score unless the people accountable for the work have agreed what the weights mean. A count of defects, a pass rate, and minutes spent are interpretable measures; combining them into a single “quality-efficiency” score can conceal a missed mandatory check or imply precision the record does not provide. Keep material defects distinct from preference edits, and record whether a defect was caught before delivery or surfaced downstream. The goal is not a sophisticated dashboard. It is a small, stable record that lets a worker and reviewer discuss whether this task met its stated bar, what repair it required, and which conditions should be checked before the workflow is widened.

Sources: Generative Artificial Intelligence Profile; NIST AI RMF Playbook: Measure

How do you count review and repair instead of stopping at the draft?

Start both clocks at the same point, such as when an assigned item is ready, and stop them only when it meets the acceptance state. A first draft is an intermediate event, not the finished unit of work. Record drafting or execution, verification, correction, reruns, and later repair separately. Also note elapsed time from start to acceptance: waiting for a reviewer is not labor, but it can matter when a deadline depends on delivery. Use equivalent boundaries in both conditions, even if the ordinary workflow has no distinct draft stage. Keep first-pass and final results separate. For each episode, record whether the initial output passed the agreed checks, what defects were found, whether the final deliverable passed, and whether a later correction or escalation followed. A final accepted item alone can make two processes look equally sound even if one needed substantial checking and repair. A defect fixed before handoff also differs from an omission discovered downstream. Use a short defect category and detection stage, not copied confidential text. Note who verified or repaired the work, including when it moved to another person. This can reveal a transfer of effort that a tally of the original worker's minutes would miss. Use the same definitions in each workflow. Count active minutes checking sources or required fields as verification; time changing an unacceptable output as correction; a new run after failure as rerun; and a later fix separately from work needed for initial acceptance. Keep definitions stable. If an episode needs several repairs, record the components consistently rather than compressing them into one “extra time” number. Keep each denominator visible: episodes attempted, first passes accepted, final deliverables accepted, or defects found. These are not interchangeable rates. An episode with multiple defects can produce a different error rate from the share of episodes with at least one defect. The Quarterly Journal of Economics field study “Generative AI at Work” followed 5,172 customer-support agents after staggered introduction of a GPT-3-based assistant. It reported an average 15% increase in successfully resolved chats per hour, with different results by agent experience and skill. That result concerns resolved chats in one firm's support setting, not verification minutes, full work quality, or downstream repair elsewhere. It is a reason to preserve the outcome actually counted: throughput cannot stand in for every quality or effort dimension. The controlled laboratory study “Managing Production Yields and Rework through Feedback on Speed, Quality, and Quantity” examined repetitive medical-record digitization with feedback emphasizing speed, accuracy, both, or yield. Its abstract reports worse production yields under speed-focused feedback and improved yields when feedback included quality; quality-focused conditions also changed how errors were fixed or prevented. This was not a generative-AI study, and its bounded task does not forecast knowledge-work results. It does illustrate why speed targets and rework belong beside quality measures: incentives can affect which errors are produced or corrected. NIST's AI RMF Playbook recommends tracking errors, incidents, response times and time to repair, and making oversight roles visible. This is organizational guidance, not a validated personal time-study method, but it supports recording repair stage and owner. If first-pass quality improves while checking or later fixes rise, investigate who absorbed the effort before calling the workflow efficient. Treat consequential late defects separately from minor edits; do not average a correctness miss away with a time saving.

Sources: Generative AI at Work; Managing Production Yields and Rework through Feedback on Speed, Quality, and Quantity; NIST AI RMF Playbook: Measure

Why should the comparison stay inside one task family?

Keep the comparison inside one task family because the question is whether assistance changes this particular work under its actual acceptance rules. “Knowledge work” is too broad: preparing a source-backed brief, cleaning a recurring spreadsheet, and summarizing a meeting have different inputs, failure costs, and review standards. If ordinary episodes contain one mix of work and assisted episodes another, an overall change may reflect the mix rather than the workflow. Compare recurring deliverables with similar stakes, input quality, deadlines, and reviewer expectations. The goal is not identical episodes; it is to avoid treating unlike work as interchangeable. Keep the ordinary-process baseline visible, and record when a supposedly comparable episode falls outside the agreed task boundary. Research supports this caution, but offers no forecast for an individual worker. “Navigating the Jagged Technological Frontier” reports a preregistered randomized experiment with 758 knowledge workers. On 18 selected consulting tasks within the tested AI capability frontier, AI users completed 12.2% more work and were 25.1% faster on average, with significantly higher solution quality. On a separate managerial task selected as outside the frontier, they were 19% less likely to give a correct solution. Researcher-designed tasks and the study setting limit transfer. The relevant lesson is that outcomes differed across tasks even within a broad knowledge-work setting. A weekly “AI productivity” figure cannot replace a task-level record. “Generative AI at Work” analyzed staggered assistant deployment among 5,172 customer-support agents at one firm. It reported an average 15% increase in successfully resolved chats per hour, with results differing by experience and skill. That output measure does not establish first-pass correctness, verification time, later rework, or results in another role. A randomized lab-in-the-field preprint, “Faster, Higher, Stronger? The Impact of GenAI on Knowledge Work Productivity,” studied 128 workers at one multinational industrial organization across knowledge acquisition, packaging, and creation tasks. Its abstract reports efficiency gains across all three, but quality gains for packaging and creation and a decline for acquisition. This bounded, non-peer-reviewed finding reinforces task-level measurement; it is not a universal rule. Define the task family before logging: specify the deliverable, normal input range, stakes, and who accepts it. Pair similar ordinary and assisted episodes when practical, or alternate conditions across comparable items if the workflow permits. Apply the same acceptance checklist and clock rules. Note an unusually difficult input, reviewer change, deadline shift, or workflow-version change. If the task changes substantially, place it in a separate category. A small sample can identify questions, but cannot establish typical performance precisely; show the episode count and exceptions with any summary. The practical rule is to compare like with like and separate tasks when pooling could hide a defect or benefit. If work mix, review standards, or workflow changes, mark the change and interpret periods separately. An overall average helps only if its component tasks and weights still fit the decision. This observational log cannot prove AI caused a difference, and it cannot measure occupational exposure, employer adoption, or displacement. It can show whether one recurring workflow deserves review while preserving enough context to decide what to test next.

Sources: Navigating the Jagged Technological Frontier; Generative AI at Work; Faster, Higher, Stronger? The Impact of GenAI on Knowledge Work Productivity

How should you interpret a mixed result?

Interpret a mixed result with a two-part gate. First ask whether every required quality and safety condition was met; only then weigh verification, correction, total effort, and delay. A faster first pass that misses a mandatory check fails the task’s agreed bar, even if it saves minutes upstream. If the required conditions hold but review or repair effort rises, narrow the AI-assisted step or redesign its handoff and check it again. If required quality holds and the full repair burden repeatedly falls, retain the workflow cautiously for that task family. The accountable reviewer should define mandatory conditions before comparison. Keep “mixed” specific. A task can have a clean final deliverable after substantial correction, or a quick deliverable that still contains an unacceptable omission. They are not equivalent outcomes. Separate a defect that affects factual correctness, completeness, safety, or a required field from a preference edit like tone or formatting. Count the stage where each problem was caught and the work needed to fix it. If a substantive defect escapes the first review and causes later rework, do not let that burden disappear inside a single “accepted” label. A longer review is not automatically poorer work when stakes require it; still report its cost and assess whether it is proportionate. When the standard itself changes during the comparison, mark the change and assess the new period separately rather than crediting one workflow with a moving target. The evidence explains why speed and quality need separate columns, but does not forecast an individual workplace. In a 2026 randomized lab-in-the-field experiment, 128 knowledge workers at a multinational industrial organization performed three defined tasks with or without GenAI. The study’s abstract reports greater efficiency across all three, while quality improved on packaging and creation tasks and declined on knowledge acquisition. This bounded experiment supports measuring by task type; it does not predict a worker’s result or quantify later rework. A separate 2020 controlled laboratory study examined repetitive medical-record digitization with feedback emphasizing speed, accuracy, both, or yield. It reported worse production yields under speed-only feedback, while quality-containing feedback improved yields and the share of errors correctly reworked. Predating generative AI and using a repetitive task, it informs how feedback emphasis may matter, not AI performance. Taken together, the studies make a speed-only verdict hard to defend: a process can move faster while quality or correction behaves differently. For a consequential task, ask the person who accepts the deliverable to classify the observed defects and agree what must pass and what repair is acceptable before widening use. If the evidence is mixed, the next move can be limited: keep assistance on the portion that met the bar, restore human work at the failure point, or change the review step and continue recording. Do not average away a serious miss with time saved elsewhere. A local log helps decide how this workflow is working under recorded conditions; it cannot establish a causal effect by itself, and the studies above cannot decide the acceptance bar for your team.

Sources: Faster, Higher, Stronger? The Impact of GenAI on Knowledge Work Productivity; Managing Production Yields and Rework through Feedback on Speed, Quality, and Quantity; Navigating the Jagged Technological Frontier

How much can a personal record let you conclude?

A personal record can support a cautious decision about one recurring workflow if it describes what was observed and what remains uncertain. It can show whether comparable episodes meet the agreed quality bar, how much checking and repair they require, and whether the pattern supports keeping, narrowing, redesigning, or pausing assistance. It cannot prove AI caused a difference, show that results transfer to another task or team, or estimate whether a role will be displaced. Treat it as a local diagnostic, not a forecast. Conditions rarely stand still. Assisted and ordinary episodes may differ in difficulty, urgency, or reviewer. The worker may become more practiced; standards, workload, tools, or process may change. Any can move quality or time alongside AI use. Pair similar episodes and record dates, versions, task mix, and exceptions, while recognizing this does not remove competing explanations. If the acceptance checklist changes, mark it and treat later observations as a new period. Random assignment can help researchers separate an intervention from other differences, but a routine worker log usually does not assign work at random. Studies provide context, not results to copy into a personal spreadsheet. “Shifting Work Patterns with Generative AI” reports a field experiment across 66 firms involving 7,137 knowledge workers over six months. In the second half, 80% of treated workers who used the tool spent two fewer hours on email each week and reduced work outside regular hours; researchers did not detect changes in task quantity or composition from individual-level access. These outcomes concern time use and task composition. They do not report whether deliverables passed first review or downstream rework rose. Task setting matters. The preregistered randomized field experiment described in “Navigating the Jagged Technological Frontier” assigned 758 consultants to no-AI, GPT-4, or GPT-4 with a prompting overview. Its abstract reports faster completion and more tasks completed on average across 18 selected tasks inside the capability frontier, but lower correctness on one selected managerial task outside it. This supports measuring the task at hand and separating speed from correctness. It does not tell a worker how often errors escape review, and researcher-designed tasks do not represent every workplace. NIST’s “Generative AI Profile” advises evaluating performance under conditions similar to deployment and cautions against extrapolating from narrow, anecdotal assessments. Describe the observation window, task family, episode count, missing records, workflow version, reviewer changes, and unusual cases. Keep denominators visible: state how many episodes passed first review out of how many recorded, rather than giving a percentage without its base. Include failed or incomplete episodes. A small log can reveal a verification bottleneck or show that apparent time savings disappear when correction effort is included. If results are mixed, stakes are high, or a decision would extend assistance to more consequential work, collect another bounded set of comparable episodes and discuss the acceptance standard with the accountable reviewer. No universal sample count turns a personal log into a controlled trial. Report limits plainly, then choose a reversible next step. Evidence can guide a local workflow decision; it cannot stand in for causal proof or a labor-market forecast.

Sources: Shifting Work Patterns with Generative AI; Generative Artificial Intelligence Profile; Navigating the Jagged Technological Frontier

What should you ask your team to agree on?

Bring a small, anonymized set of ordinary and AI-assisted examples to the person who accepts the work. Set the standard before comparing results; otherwise the team may move the bar after seeing which workflow looks faster. The NIST AI RMF Playbook’s Measure function says metric choice depends on purpose, audience, and evaluation needs, and recommends documenting the criteria used to select metrics. This organizational guidance supports making the local purpose explicit. First, agree on the task boundary. Define one episode, where it starts, and what counts as completion. “Write a report” may bundle research, drafting, checking figures, and adapting the result for a reader. If those steps have different acceptance conditions, narrow the comparison or record stages separately. Then identify must-pass checks: perhaps required fields are present, figures trace to an approved source, or a recommendation follows from supplied evidence. The accountable reviewer should distinguish mandatory requirements from preferences. The Generative AI Profile of the NIST AI Risk Management Framework advises measuring performance or assurance criteria under conditions similar to deployment and documenting the measures. For a worker, record conditions and checks clearly enough to judge the next episode on the same basis. Second, name who owns final verification and what follows a failed check. Ask who reviews before delivery and who handles ambiguity, correction, or escalation. If another colleague must repair the assisted output, include that work even if it is absent from the original worker’s time sheet. Agree on defect categories that separate a miss caught before handoff from one found later, and note which person or stage absorbed the correction. The NIST AI RMF Playbook’s Measure function calls for tracking errors and impacts, documenting oversight roles, and assessing whether metrics and controls remain appropriate. These prompts make responsibility visible without prescribing one review structure. Third, set a boundary for correction and downstream rework before widening use. Ask which defects require correction, which require escalation or stopping delivery, and what kind of rework signals that the process needs redesign. Set any threshold with those accountable for the work and in light of its consequences; the cited guidance supplies no universal acceptable rework allowance. A coded task identifier, counts, time by stage, reviewer role, and workflow version may suffice; confirm what workplace policy permits, and do not copy client, colleague, or employer material into an unapproved log. Keep the record limited to what the decision needs, with access and retention handled under existing rules. NIST guidance is directed at organizations managing AI risks, so applying it to an individual work log is an adaptation, not a requirement imposed by the framework. Agree when to revisit the result, such as after a change in task, tool, inputs, criteria, or review ownership. List the task and cases, must-pass checks, verification owner, escalation route, fields, rework boundary, and review point. Then the log can inform a bounded choice to keep, narrow, redesign, or pause this use. It does not show that all teams have adopted AI, predict the occupation’s future, or establish whether a worker’s role will disappear. The useful conversation is specific: “Before we expand this step, can we agree what counts as acceptable work, who signs off, and how we will count repairs?”

Sources: NIST AI RMF Playbook: Measure; Generative Artificial Intelligence Profile

What is the next bounded step?

Start with one recurring task whose AI use is allowed by workplace policy. Before comparing workflows, ask the person who accepts the deliverable to agree on its task boundary, required checks, and acceptable correction burden. Then record comparable ordinary and AI-assisted episodes under that same standard. Note the workflow version and task conditions, and keep first-pass quality, final acceptance, verification, correction, total effort, and later rework distinct. A bounded record can help decide whether to keep the workflow as used, narrow where assistance enters, redesign review, or pause it. Revisit that choice when the task, tool, review standard, or stakes change. NIST's Generative AI Profile supports evaluating performance under conditions similar to deployment and documenting conditions; it does not prescribe a personal sample size or make a worker's log a controlled trial. Treat the result as a local signal, not proof that AI caused a difference or that it will transfer to another task or team. The International Labour Organization's refined exposure index addresses occupational exposure, a separate question from whether this workflow meets its quality and rework expectations. A task log cannot estimate a person's chance of job loss. For the next conversation, bring the record and ask: “Which checks must every deliverable pass, who owns final verification, and how much rework is acceptable before we widen this workflow?”

Sources: Generative Artificial Intelligence Profile; Generative AI and Jobs: A Refined Global Index of Occupational Exposure