How to Test Human Review of AI-Drafted Work

Why does document-level confidence hide claim-level errors?

A document-level judgment can hide a mixed result: several statements may be well supported while one consequential sentence is not. Test material claims against evidence and check whether the wording preserves what it says. It makes the path visible, but cannot establish completeness or usefulness. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation offers one useful way to think about that first step. It breaks generated biographies into atomic facts, then measures the share supported by a reliable source. The study includes human evaluation of biographies generated by several language models. Its unit of analysis is useful beyond that task: a paragraph can contain claims with different levels of support. It does not validate workplace review, measure completeness or usefulness, or set an office-document pass threshold. Applying its claim-level idea to work products is an editorial inference, not a reported workplace result. Consider an illustrative internal report summary that says, “Support requests fell 18 percent after the new process.” A linked table might contain a figure near 18 percent, yet that alone does not establish the sentence. The reviewer needs to identify the table’s period, denominator, and comparison, and compare them with the sentence. If the table reports a particular request type over one month, the draft cannot silently turn it into all requests over a quarter. A citation is a locator, not proof of a match; relevant qualifiers must remain. That check has several distinct failure points. First, claim extraction may omit a material assertion or split one sentence in a way that loses its condition. Second, retrieval may find no source or the wrong passage, even when a valid source exists elsewhere. Third, interpretation may confuse what a source measures with what the draft claims. Finally, wording may overstate a correctly retrieved result by dropping a date, population, or limitation. A single score can blur these causes; naming the stage helps diagnose the miss: did the process fail to identify the claim, locate the evidence, understand it, or carry its boundaries into the final sentence? Factcheck-Bench: A Benchmark for Claim-Level, Sentence-Level, and Document-Level Fact-Checking describes a multi-stage annotation framework that evaluates claims, sentences, and documents, including intermediate steps as well as final predictions. That structure reinforces a practical distinction: a document verdict is the endpoint of a chain, not proof that each link worked. The benchmark is research on open-domain automatic fact-checking, not validation of employee processes or workplace error rates. Its methodological value is that intermediate evidence can show where review broke down. Even a successful claim-to-source match leaves another question: is the evidence set adequate for the document’s purpose? One table may accurately support a percentage without establishing why it changed or whether another measure moved in the opposite direction. NIST’s Building Evaluation Probes into Agentic AI describes faithfulness, completeness, and sufficiency as separate citation-quality dimensions. NIST presents ongoing project work against a curated corpus, not a finished workplace review tool. The distinction is still useful: a supported sentence is not automatically a complete or sufficiently evidenced conclusion. In practice, begin with claims whose error could change a decision. For each, record the assertion, source passage, and qualifiers that must remain intact; then ask separately whether the sources cover the question. Keep style and readability review as a different task. Make consequential evidence links inspectable and direct attention to where human judgment matters.

Sources: FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation; Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers; Building Evaluation Probes into Agentic AI

What can a source-based check actually test?

A source-based check can test three different things: whether a source supports a statement, whether the statement preserves the source’s material meaning, and whether the evidence is strong enough for the conclusion. A clickable link alone answers none of them. NIST’s ongoing project, “Building Evaluation Probes into Agentic AI,” offers a useful vocabulary. Its experimental probes compare claims with a human-curated reference corpus and distinguish faithfulness, completeness, and sufficiency. Faithfulness asks whether the source supports the claim. Completeness asks whether the wording captures the source’s full message rather than selecting a convenient fragment. Sufficiency asks whether the evidence carries the burden of the claim. NIST describes ongoing research, not a validated workplace product. Consider an illustrative policy summary: “Employees may retain local copies of project files.” The cited clause contains those words, so the sentence may pass a narrow faithfulness check. But if the next sentence says copies must be deleted after a set period, or that the rule applies only to approved devices, omitting it could make the summary incomplete. The reviewer must ask whether the missing condition would change how a reader acts; citation presence cannot establish practical meaning. Sufficiency is different. Several passages may accurately describe a policy and still fail to support a broad conclusion about what an organization permits in every location or situation. Scope matters: a local procedure may not establish a company-wide rule, and an example may not establish a general requirement. A source that describes an earlier rule may not support a statement about the current one. Date, version, jurisdiction, population, and document scope define what evidence can carry. A reviewer must still decide whether references are authoritative and current. Supplied-source checking and open-web retrieval answer different questions. Given a bounded collection of approved documents, a check can ask whether claims match that collection and show the passages used. This does not prove the collection is complete, current, or authoritative. Open-web search may find relevant material, but relevance does not establish authority or currency. Source selection remains distinct from checking a selected source. “Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers” presents an open-domain research benchmark at claim, sentence, and document levels, with labels for intermediate stages as well as final predictions. It evaluates research systems; it does not validate a particular office workflow. This illustrates why stages matter: a wrong conclusion may begin with a missed claim, poor match, or interpretation error. A final pass or fail hides where the chain broke. For a recurring work product, name the property before choosing a check: “Does this sentence match the cited source?”, “Did the summary omit a qualification that changes the action?”, or “Do these sources justify this conclusion at this scope?” The first may be tested against an identified passage. The second requires a coverage question and surrounding context. The third requires expertise about the claim’s burden and the source set. They are not interchangeable. NIST’s “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile” discusses the risk of people over-relying on fluent AI output. A visible citation can invite deference if the reviewer does not know what was tested. Label the check narrowly and preserve its evidence: match, coverage, or sufficiency. When authority, currency, or scope is not guaranteed by the workflow, assign those judgments to a knowledgeable person and record the rationale. A pass means only that a specified property held against a specified reference in the cases reviewed. It does not mean the whole work product is complete or ready to send. This directs attention to decisions a source check cannot establish.

Sources: Building Evaluation Probes into Agentic AI; Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers; Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile

What can a check establish, and which part of the chain can fail?

A passing check establishes only that its stated rule passed on the inputs examined. It does not, by itself, establish that the draft is accurate overall, complete, useful, or ready to send. The result depends on a chain: a reviewer or system must identify the relevant claim, find the right evidence, compare the two appropriately, apply a decision rule, and interpret the result in context. A weakness at any link can make a clean-looking score misleading. The research literature offers a practical reason to make that chain visible. “Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers” describes a multi-stage annotation framework for evaluating factuality, with labels not only for a final prediction but also for intermediate steps. Its open-domain benchmark works at claim, sentence, and document levels. That structure can help locate where an evaluation went wrong: perhaps a claim was split incorrectly, evidence was missed, a source was interpreted poorly, or a final label failed to reflect the evidence. The paper presents a framework and benchmark for automatic fact-checking; it does not show that a workplace review process has been validated. More detail, however, does not guarantee a dependable measure. “Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics” reevaluated five factuality metrics across 11 datasets covering summarization, retrieval-augmented generation, and question answering. The authors report that metrics disagreed with one another, often misestimated factual accuracy, and showed bias against highly paraphrased outputs and evidence located far from the relevant text. They recommend manually validating metric reliability in the domain where it will be used. These are results from language-generation evaluation datasets, not a study of office documents or employee review accuracy. This distinction helps separate checks that look similar but test different things. A deterministic check can verify that a required heading exists, a specified date matches a record, or a field is present. Its rule is explicit, and a reviewer can usually inspect the comparison directly. A semantic check asks whether a paraphrased claim is supported, whether a qualification was lost, or whether evidence is adequate. It must interpret meaning and context, so a reference match may be uncertain even when the words differ, and similar wording may still distort the source. “AI measurement and evaluation” from NIST emphasizes that characteristics such as accuracy, reliability, robustness, and bias require their own measures, and that context changes how a component should be evaluated. The practical rule follows: as interpretation between draft and reference increases, the check should show more of its evidence and leave more room for a knowledgeable person to decide. A source-support result should identify the claim, the passage or record used, and the reason the comparison passed or failed. Record both missed defects and unnecessary flags: the first can let a real problem through, while the second can consume attention and weaken trust in the process. Keeping examples of each makes the result actionable: teams can revise the rule, improve the reference, or narrow the check when its errors reveal a mismatch. A check is therefore useful when its criterion is narrow, its reference is visible, and its limits are understood. It cannot quietly expand that finding into proof that the whole work product is sound. A score without those details is not enough to justify changing responsibility or release decisions.

Sources: Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers; Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics; AI measurement and evaluation

How should a team test whether its check is itself reliable?

A team can test whether a review check deserves a narrow role by comparing it with decisions made by a knowledgeable person on real examples. The useful question is not whether the check sounds persuasive or returns a high score. It is whether it catches the particular defect it was designed to catch, on the kinds of work where the team intends to use it, without creating so many unnecessary alerts that people stop examining them. Treat the exercise as a practical local evaluation, not a universal protocol or proof that the workflow is safe. Research on factuality metrics supports the need for this caution: Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics reevaluated five metrics across 11 datasets covering summarization, retrieval-augmented generation, and question answering. The authors report disagreement between metrics, misestimated factual accuracy, and sensitivity to paraphrased output or evidence far from a source passage. Those results concern research datasets and metrics, not office review error rates, but they show why a check should be challenged against examples from its intended domain.

Start by defining what counts as a case. Keep the work product, the exact criterion, the source material or requirement the reviewer was allowed to use, the expected finding, and the reason for that finding together. If the answer depends on interpretation, record the ambiguity instead of forcing a clean label. A second knowledgeable reviewer can adjudicate disagreements where practical; the check being evaluated should not generate its own answer key. Otherwise, a shared mistake can look like agreement. Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers illustrates why it helps to keep intermediate steps visible: its framework labels factuality at claim, sentence, and document levels and includes stages in the checking process, rather than recording only a final verdict. That benchmark is about automatic fact-checking, so the transfer here is methodological. For a workplace check, the same idea means noting whether a problem began with a missed claim, unavailable evidence, a mistaken match, or an incorrect conclusion. A single pass/fail label would hide which part needs repair.

Include ordinary examples and boundary cases that could expose a predictable blind spot. For a source-support check, that might mean a faithful paraphrase, a qualifier several pages away, a missing citation, a source whose version is stale, or a plausible excerpt that does not support the breadth of the sentence. These are test cases to consider, not a claim that each will cause a particular system to fail. Mark each result in plain terms: a missed defect is a case the check passed even though the criterion failed; an unnecessary flag is a case sent for correction even though the reference review found no defect. Looking at both matters. A check that misses important defects may create unwarranted confidence; one that flags too much can consume review time and teach staff to dismiss alerts. NIST’s AI Measurement and Evaluation guidance says measurement depends on operational context and that metrics have strengths and limitations. That is a reason to test against the actual document, source access, and review conditions, rather than borrow a result from a different task. No fixed sample count or numerical threshold is justified by these sources. Repeat the comparison when the workflow changes.

Sources: Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics; Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers; AI measurement and evaluation

Why should review be matched to the actual work product?

Because AI’s contribution and errors can vary by task, evaluate the particular work product and review criterion you use. Evidence that assistance helps on selected tasks does not establish that a check proven on one document type transfers to another. The practical unit is a recurring task with a defined purpose, reference material, and consequence, not a model brand or broad label such as “writing.” The accessible abstract for “Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality” describes a preregistered experiment with 758 knowledge workers randomly assigned to work without AI, with GPT-4, or with GPT-4 plus a prompt-engineering overview. On 18 realistic consulting tasks within the researchers’ capability frontier, AI users completed more tasks, worked faster, and produced higher-quality solutions. On one selected complex managerial task outside it, they were less likely to be correct. This shows task variation under those conditions, not workplace review accuracy. Consider an illustrative workflow where a team checks AI-assisted meeting summaries against a template: each action has an owner and date, and quoted decisions link to the meeting record. These are bounded criteria because expected fields and sources are clear. Reusing the template on a recommendation memo would answer different questions poorly. An owner field cannot show whether evidence supports the recommendation or whether alternatives were considered. An employee assessment raises other questions about context, fairness, and consequences. These examples apply, but were not tested by, that study. Before checking, specify the document’s purpose, authoritative evidence or requirement, error the check should catch, and consequence if it passes. A summary may have a narrow record for comparison. A recommendation may depend on assumptions, uncertain evidence, and judgment about trade-offs. An external commitment may require an authorized person to confirm wording and authority. These differences change what counts as a pass, whether references must be current, and where someone must interpret rather than compare. NIST’s “AI measurement and evaluation” supports this context-specific approach as general guidance: distinct characteristics need their own measures, and context can change how a component is evaluated. It does not provide a validated workplace-document rubric. The inference is that a result belongs to the criterion, reference, and workflow tested. A meeting-summary test may be a bounded aid; it is not evidence that a recommendation is complete or an external statement ready to send. Using the same model and checklist everywhere is simpler, but hides assumptions about transfer. A workflow-specific check takes setup and maintenance as sources or purpose change. It is worthwhile when it targets a material error. Name its scope rather than presenting it as a general quality score. For a local comparison, collect permitted examples of one recurring document type. Have a knowledgeable reviewer record findings before seeing the check, including on difficult cases such as paraphrased evidence or an ambiguous recommendation. Compare whether it catches the issue, flags acceptable work, and how long resolution takes. This is not universal; a small sample cannot establish general reliability, but can reveal where the check helps and where human review remains necessary. The experiment should narrow the question: where does assistance help on this task, and what criterion can be verified against an appropriate reference? Test the recurring work product, not “AI” in the abstract. A pass supports only the defined check on examined cases; it does not certify meaning, completeness, fairness, or readiness for release. Start with one document type and consequential criterion, then identify who can judge hard cases and what evidence that person needs.

Sources: Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality; AI measurement and evaluation

How should a team compare speed with review quality and workload?

A fair comparison starts with the same drafts, the same source versions, and the same target defect. If one method sees different material, or reviewers are asked different questions, the result mixes the method with the conditions around it. First define the criterion in plain terms: for example, whether each factual claim in a summary is supported by the supplied report. Then have a knowledgeable reviewer adjudicate the examples against that criterion, preserving the relevant evidence and any genuine ambiguity. Compare informal proofreading, a reference-based check, and a second automated reviewer against those decisions. Informal proofreading is flexible. A reviewer can notice awkward wording, missing context, or a problem outside a checklist. But if the criterion stays implicit, two people may inspect different things, and a later team may not be able to tell what “reviewed” meant. A reference-based check makes one question visible: does this statement match the specified source or requirement? It can leave a trace that another person can inspect. Its boundary is equally important: the check depends on the selected reference being relevant and current, and it may not notice that the source set omits a necessary perspective. A second automated reviewer may help surface candidate issues for attention, but a matching verdict is not independent confirmation. Two systems can share weaknesses, and agreement does not show that either one used the right evidence. Keep the comparison narrow enough to diagnose. For each method, record whether it caught the target defect, whether it flagged a sound passage, and what evidence led to the flag. A missed defect means a problem in scope was left unflagged; a false alarm means sound material was sent for correction. These are different costs. A method that raises many alerts may still be useful if a person can resolve them quickly, while a quiet method may conceal misses. Do not collapse both outcomes into one score without showing what the score hides. When a method disagrees with the adjudicated decision, inspect the case: perhaps it missed a paraphrase, relied on a distant passage, or applied the criterion too broadly. This diagnosis matters more than a polished aggregate result. Measure effort across the whole path from draft to release, not just the time spent producing the first draft. Track drafting time, review time, correction and rework time, and time spent resolving flagged items. Also record whether corrections were substantive or merely stylistic, so a faster first pass does not mask work shifted to the reviewer. NIST’s “AI Measurement and Evaluation” guidance emphasizes that measurement depends on operational context and that metrics have strengths and limitations. That supports comparing the same task under the conditions where it will be used; it does not prescribe a universal workplace rubric or prove that one approach saves time. A worked example from the vendor-authored “AI-Assisted Drafting of a Regulatory Submission Section: A Worked Example” proposes tracking cycle time, reviewer time, citation resolution, and factual corrections. Treat that as a proposed measurement set, not an independent result: the page does not establish measured outcomes or general applicability. The useful comparison is therefore practical and bounded. Keep a check if it makes the target criterion and evidence easier to inspect, catches relevant defects, and leaves total effort acceptable for the work. If it saves drafting time but increases unresolved flags or correction burden, the workflow has moved effort rather than demonstrated a gain.

Sources: AI measurement and evaluation; AI-Assisted Drafting of a Regulatory Submission Section: A Worked Example; Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics

Which judgments remain human even after measurable checks pass?

A check can establish that a defined condition held; it cannot decide by itself whether the document deserves to be sent. Someone with relevant knowledge still has to ask whether the draft answers its intended question, whether its evidence is current and adequate, and whether important context or affected people are missing. Record the question, source rationale, uncertainty, disagreement, and who accepted the version to make these judgments more inspectable. But a complete record is evidence that a process occurred, not a score proving the judgment was sound. Human presence does not automatically correct the weaknesses of a fluent draft. NIST’s Generative Artificial Intelligence Profile identifies automation bias and over-reliance as risks: people may defer excessively to automated systems or unjustifiably see their output as higher quality than other sources. It also describes high-integrity information as distinguishing fact from opinion and inference, acknowledging uncertainty, linking claims to evidence, and being transparent about vetting. For a work product, the practical question is whether the reviewer examined the underlying evidence and the draft’s purpose, or mainly confirmed that the text looked complete and carried citations. A signature or approval click cannot answer that. It is guidance, not evidence of workplace prevalence. The human role is strongest where a criterion depends on context or consequence. A source check may show that a policy clause is quoted accurately, while a qualified reviewer must decide whether it applies here, whether an exception matters, or whether the summary leaves a misleading impression. A draft can contain required fields and still recommend the wrong action because the underlying question was framed poorly. A record preserves rationale for challenge; it cannot make interpretation objectively correct. How much scrutiny is proportionate depends on what could follow from an error. A formatting defect in an internal note may be caught with a simple rule. A document that affects a person, commits an organization externally, or is difficult to reverse calls for stronger source verification, attention to omitted context, and sign-off by someone qualified to make the relevant judgment. This is risk-based editorial guidance, not a universal policy or legal requirement. NIST’s AI Measurement and Evaluation guidance says measurement depends on operating context and different characteristics require their own measures. A result about citation support therefore cannot stand in for a result about fairness, completeness, or suitability for release. A useful division of labor follows: automate checks whose criterion and reference are explicit, then reserve human attention for whether those checks cover the real question and whether remaining uncertainty is acceptable. NIST’s ongoing Building Evaluation Probes into Agentic AI project separates faithfulness, completeness, and sufficiency when examining source-grounded claims. That distinction clarifies the boundary, but the project describes research work, not a ready-made approval system for ordinary documents. If support, coverage, and evidentiary adequacy are distinct even in a bounded citation check, passing one cannot settle the larger judgment of usefulness or release. For a recurring report, name a reviewer who understands its purpose, can inspect the sources, and has time to disagree with the draft or its checks. Record the review question, material sources, unresolved ambiguity, decision, and escalation where needed. These notes make accountability and later learning more concrete; they are not proof that no error remains. The meaningful signal is review against purpose and evidence, not simply the presence of a human at the workflow’s end.

Sources: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile; AI measurement and evaluation; Building Evaluation Probes into Agentic AI

What is one proportionate next step for your workflow?

Pick one recurring AI-assisted document and name an error. In a weekly summary, a figure might not match the approved report; in a customer response, a promise might conflict with policy. Identify the record or requirement that can settle the criterion. Checking whether a value matches a table is narrower than asking whether a draft is good. NIST’s AI Measurement and Evaluation guidance says evaluation depends on operational context and metrics have limits, so define the task first. The 2025 ACL Findings study reevaluated five factuality metrics across 11 datasets, reporting inconsistency and sensitivity to paraphrase and evidence distance. These datasets do not establish workplace error rates; they are a reason to test locally rather than assume a check transfers.

On permitted material, compare the check with a knowledgeable reviewer’s decisions. Include routine and difficult examples, such as a paraphrased claim, a qualifier, or a missing reference. Record defects missed, sound work flagged, and resolution time. Treat this as a bounded pilot, not proof that every document type is covered. Keep a person responsible for whether the document answers the real question and is fit to release. Ask a manager or teammate: “For this recurring draft, what is the one error we most need to catch, what source or requirement proves it, and who decides whether the result is fit to send?” If you are also mapping AI’s effect on your broader task mix, the free task checker offers change-pressure signals; it cannot determine whether a draft is safe or predict job loss.

Sources: AI measurement and evaluation; Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics

Questions readers ask