What Should Urban Planners Verify in AI Summaries of Public Comments?

Start with what the summary will be used to do

Before accepting an AI summary, write down what it will do . An internal aid can point an officer to submissions. A public account tells residents and decision-makers what the authority says it heard. Formal casework evidence or an input to a recommendation can carry greater consequences. The same wording error may be tolerable as a temporary label yet misleading when it describes a community’s position or an issue’s significance. Review effort should follow intended use, not fluency or time saved. The Digital community engagement toolkit for planning consultations describes AI as a way to help tag comments, summarize feedback and identify patterns, while stating that officers should review results before drawing conclusions. This supports using a summary to organize work; it does not make the summary the consultation record or settle what a comment means. The Planning Inspectorate’s Use of artificial intelligence in casework evidence guidance sets a specific boundary for the English casework it covers: local planning authorities using AI summaries must provide original case documents, because material considerations can be omitted. Requirements depend on context, but the practical lesson is to retain access to underlying submissions. Set three questions before anyone relies on the output. Is each material claim faithful to the comments behind it? Does the summary cover the record, including exceptions? Does its grouping fit the local planning question? These tests separate paraphrase, coverage and interpretation. A summary might pass one and fail another. Naming the use first lets a planner decide which claims need source-level checking, what limits to state, and who remains accountable for review. It keeps a provisional aid provisional until those checks are complete.

Sources: Digital community engagement toolkit for planning consultations; Use of artificial intelligence in casework evidence

Can you trace each material claim to the comments behind it?

Before using a summary to describe a planning issue, trace each material statement back to the comment or submission it represents. Check whether the source supports the same point, with the same qualification, location and requested action. Flag invented policy references, positions assigned to the wrong person or place, and wording that makes a tentative concern sound categorical. Fluency is not evidence that this chain is intact. The chain begins with the submitted document or recorded comment, then passes through any extracted passage, location or topic tag, grouping of related comments, and final sentence in the summary. A planner should be able to move backwards from a claim through these steps and explain its connection to the source. A citation is useful only when it leads to the relevant original comment and that comment supports the summary’s wording. A link to a broad consultation page, or a citation that opens a comment about the same general subject, does not establish a specific claim. For a claim that could shape a recommendation or public account, checking a few representative examples cannot establish that every material assertion is traceable; review the assertions that carry the conclusion. This is a practical response to a known generative AI risk, not a claim about a particular planning tool’s error rate. The U.S. National Institute of Standards and Technology’s *Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile* defines “confabulation” as confidently stated erroneous or false content, including output that diverges from its prompt or other input. It notes that such content can mislead users, and that generated citations may themselves be false. NIST’s framework is cross-sector guidance; it does not measure how often a local consultation summarizer makes these errors. Here, its relevance is that a confident sentence or plausible citation still needs to be checked against its source. The NYC Office of Technology and Innovation’s AI register describes its Department of City Planning workflow, Voice to Vision, as tagging, geolocating and summarizing public comments, with summaries citing original statements and staff reviewing outputs before publication. The register also identifies possible misrepresentation and incorrect location attribution as risks, with source links and staff review as safeguards. The register is a system description, not an independent test of accuracy. This workflow makes provenance visible, while leaving planners to check whether each source supports its claim. Use also determines the consequence of a mismatch. A provisional label that helps an officer find comments can be corrected during review. A sentence in a public report or recommendation can shape how an issue is understood, so its source connection needs to survive scrutiny. When a consequential statement cannot be traced, or its wording goes beyond what the comment says, correct it, qualify it or remove it. Keep the original record available; do not let polished phrasing stand in for evidence.

Sources: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile; AI-Assisted Summarization of Community Planning Input

What changes between a faithful summary and a complete record?

A summary can accurately represent the comments it includes and still be incomplete as an account of the consultation. Fidelity asks whether a paraphrase preserves its source’s meaning. Coverage asks which submissions and which parts of them entered the summary. Before relying on a polished overview, establish what material was processed and look for omissions that could change what a planner or reader concludes. Start with the corpus. Confirm which consultation, channels and dates the input covers, and whether the supplied files represent the available record. Check how the process handled unreadable attachments, scanned documents, tables, different file formats and language variation where relevant. If some material could not be processed, disclose that limit and decide whether it needs separate review. Then inspect coverage within documents. A long representation may raise several distinct points, qualify an objection or describe a condition that matters at one location. Ask whether the summary preserves those strands or compresses them into a dominant theme. Repetition can affect how often a topic appears, while deduplication can change what readers see; neither should happen invisibly. Imagine a summary that accurately reports many requests for more housing but leaves out one submission identifying an access constraint at a particular site. The dominant theme may be faithfully paraphrased, yet the overview could be insufficient for work requiring attention to site-specific considerations. It shows why frequency and completeness differ: a less common concern may warrant checking because of its content and relevance, even if it does not define the overall pattern. For formal casework in England, the Planning Inspectorate’s “Use of artificial intelligence in casework evidence” guidance sets a boundary for the Local Planning Authority process it describes. An authority using AI summarisation must provide original case documents; a summary alone may not meet relevant requirements and may deprive an Inspector of material considerations omitted from submissions. This is a procedural instruction for that context, not a universal rule for every planning task or jurisdiction. The summary must not displace the source record. The wider reason to inspect omissions is not unique to planning. NIST’s “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile” defines confabulation as confidently stated erroneous or false content and discusses output that diverges from input. The cross-sector framework establishes a risk category, not an error rate for planning tools or proof that a particular summary omitted material. It supports checking output against input instead of treating fluent prose as evidence of coverage. Keep original comments available, record the corpus and processing limits, and inspect source material when a missing point could affect the summary’s intended use. If the input set is uncertain or a material concern cannot be traced, qualify the overview, check the underlying documents or omit the unsupported claim. Completeness depends on knowing which record it represents and keeping important exceptions findable.

Sources: Use of artificial intelligence in casework evidence; Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile

Do the counts describe submissions or the wider public?

Counts describe the consultation material and the rules used to count it; they do not automatically describe what residents generally think. Before reporting that a theme was common, establish the unit: individual comments, submissions, participants, or mentions within submissions. These are not interchangeable. One person may submit several comments, a group may send one response, and a single long representation may raise multiple issues. A frequency can therefore be accurate for its chosen unit while giving a misleading impression of how many people held a view. State the denominator and the counting rule beside any number, and explain how repeated or group responses were treated.

Aggregation can also change which concerns appear important. Repeated or near-identical comments may dominate a count; a broad category can combine different reasons for support or objection; and a low-frequency concern can disappear beneath a dominant theme. Meanwhile, people who did not take part are absent from the response set. Representativeness depends on who was reached, which channels were available, who chose to respond, and what the consultation asked. Unless the engagement design supports inference to a wider population, describe the findings as what respondents or submissions said, not what the public believes. Keep material exceptions visible even when they are numerically uncommon, especially when tied to a site, access issue, or policy consequence.

The official case study, “Plymouth, South Hams and West Devon improve analysis with AI,” illustrates why reach, processing speed, and representativeness must be reported separately. In a 2023–24 pilot involving three local authorities, more than 1,400 residents took part, participation increased in three of four electoral wards compared with the previous consultation, and 69% of participants had not joined a joint local plan consultation before. The case study also reports a 66% improvement in NLP processing time, saving up to 10 minutes per response. Yet residents under 35 remained under-represented compared with population estimates, and participation in areas with higher deprivation remained lower. These linked digital engagement and analysis results do not show that AI caused participation changes, respondents represented the whole area, or faster processing corrected participation gaps. The account itself says workshops and in-person events still matter for reaching a broader demographic.

The “Digital community engagement toolkit for planning consultations” similarly advises authorities to define who they need to hear from, monitor participation, and combine digital methods with approaches such as drop-ins and community meetings. For a summary, report the response set and its limits plainly: which channels produced the material, what groups or places appear under-represented, and whether a finding concerns participants, submissions, or the broader population. Pair theme counts with important low-volume exceptions and participation gaps. If the consultation cannot support a population-wide claim, narrow the wording rather than allowing a precise count to imply broader evidence than the process collected.

Sources: Plymouth, South Hams and West Devon improve analysis with AI; Digital community engagement toolkit for planning consultations

Do themes and locations fit the local planning question?

Check whether the categories map to the actual plan, policy names, consultation question and geography before reading them as positions or place-based patterns. A label can be applied consistently and still answer the wrong question. Review overlapping categories, comments about multiple policies, ambiguous descriptions and location tags against the consultation’s own terms. Keep the original comment visible beside its classification so the reviewer can see what the label captures and what it leaves out.

Classification interprets language in context. A person may describe a lived condition without naming a policy, use an informal place name, or raise connected concerns in one submission. Forcing one category can erase those connections; accepting a location guess without checking can assign a concern to the wrong area. Ask whether another planner could explain why the category and location fit, using the comment and consultation question.

The NYC Office of Technology and Innovation’s AI register entry, “AI-Assisted Summarization of Community Planning Input,” describes New York City’s Voice to Vision workflow: it tags themes, identifies mentioned locations and drafts summaries linked to source comments. City staff review outputs before publication. The register also names risks of cultural or linguistic misrepresentation and incorrect geolocation, with source links and staff review as safeguards. This is a disclosed workflow, not an independent evaluation showing that its classifications are accurate. Its useful lesson is to make tags inspectable and checkable against the comments, rather than treating a citation link as proof that the interpretation is right.

Consider an illustrative comment: “The bus stop by the clinic needs better lighting, and the crossing feels unsafe after dark.” Depending on the consultation, it could concern pedestrian safety, access to health services, public-realm design or a nearby site policy. A single “transport” label may hide safety; “safety” may lose access context. Preserve linked concerns or use multiple categories, then check whether the stop’s location is specific enough to map. If not, mark it uncertain rather than guessing.

The peer-reviewed study “Uses, opportunities and risks of artificial intelligence in participatory urban planning” reports a Delphi process with 34 planners from Finnish municipalities above 20,000 inhabitants, followed by focus-group and survey work with 13 participants. Respondents discussed possible uses such as analyzing citizen feedback and risks including misinformation. These are professional views, not a benchmark of tagging performance or proof that a classification scheme works locally. They support reviewing categories in the setting where they will be used.

Validate the scheme on current consultation comments that span policies, use informal place names or resist existing categories. Check that labels reflect the question and map boundaries; retain multi-category comments and flag uncertain locations for human review. Describe only the themes the record supports, without making categories appear more definite than the comments.

Sources: AI-Assisted Summarization of Community Planning Input; Uses, opportunities and risks of artificial intelligence in participatory urban planning; Using AI to support faster local plan consultation analysis

What does current evidence say about the work AI can assist?

Available planning examples support using AI to organize comments, apply tags and draft summaries for officers to review. They do not establish that a generated account is accurate, complete, representative or suitable to decide a planning question on its own. The evidence answers different questions: official guidance describes safeguards, local pilots show particular workflows, and a planner survey records professional views. The Ministry of Housing, Communities and Local Government’s Digital community engagement toolkit for planning consultations says AI may tag comments by theme, summarize feedback and identify emerging patterns. It sets a boundary: officers should review results before drawing conclusions, and professional judgment remains necessary. This is practice guidance, not a benchmark proving a system performs well. It supports trying bounded tasks such as grouping similar comments or drafting an initial account, while keeping interpretation and sign-off with planners who understand the consultation. Uses, opportunities and risks of artificial intelligence in participatory urban planning reports a three-phase Delphi survey of 34 urban planners across Finnish municipalities with more than 20,000 inhabitants, followed by focus-group and survey work involving 13 participants. Respondents saw potential for analytical AI to examine citizen feedback and generative AI to draft planning content or summarize expert reports. They also identified risks including data-protection problems, outsourcing decisions to tools, misinformation and incorrect content. These are professional perceptions and anticipated uses, not an accuracy test of a public-comment summarizer or evidence that authorities generally use one. In practice, a system might group comments about a crossing, tag a paragraph to a policy heading, or draft a short summary of a long submission. An officer must still decide whether the grouping joins similar concerns, whether the tag fits the current plan, what qualifications the draft omits, and whether the issue matters to the decision. Producing an organized first pass does not perform the professional interpretation that follows it. The National Institute of Standards and Technology’s Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile describes “confabulation” as confidently presented false or erroneous content, including output that diverges from its input. It advises against extrapolating capability from narrow or anecdotal assessments and recommends verifying sources in outputs. This is general risk guidance, not a measured error rate for planning tools. A result from one council or consultation format is a reason to test locally, not a guarantee for another setting. The practical comparison is an AI-assisted first pass checked against local comments and followed by human interpretation versus a manual first pass. Pilot the exact task on local material. Judge it by whether officers can verify claims and categories for the intended use, not by general capability claims or a neighboring authority’s experience.

Sources: Digital community engagement toolkit for planning consultations; Uses, opportunities and risks of artificial intelligence in participatory urban planning; Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile

What does the time-saving evidence establish—and what does it leave open?

Local pilots show that some consultation-processing steps can be much faster with AI assistance. They do not show that a summary is accurate, complete, representative, or ready to support a planning conclusion. Interpret a time figure by asking what task and material it covered, which human checks remained, and whether quality was assessed. Faster processing is evidence about one workflow step, not a substitute for reviewing comments or exercising professional judgment.

The Local Government Association’s “Greater Cambridge Shared Planning: AI-Driven Summarisation for Local Plan Consultations” reports a pilot across three supplementary planning document consultations in winter 2024–25. A summarizing task reported as requiring 18.5 officer hours was completed by the model in 16 minutes. The account says officers still read every representation. This measures summary production within a process retaining officer reading, not the whole consultation. It describes one local task, not equivalent quality elsewhere.

“Plymouth, South Hams and West Devon improve analysis with AI,” describes linked digital-engagement and analysis pilots conducted in 2023–24. It reports a 66% improvement in processing time during natural-language-processing testing, saving up to 10 minutes per response. Officers used the tools for early analysis, and oversight remained essential. Because digital engagement, mapping and analysis tools were tested together, participation results do not isolate AI’s effect. The processing figure concerns testing, not a universal estimate for other consultation records or evidence that summaries need no review.

The later MHCLG Digital update, “Using AI to support faster local plan consultation analysis,” reports that five authorities extended the pilot work and participating authorities saw efficiency gains of around 90%. It also describes work outside the headline comparison: redaction, data formatting and iterations for usable outputs took two to three weeks for some authorities, and officers still needed to check results. In Greenwich, some policy tagging was inaccurate and overlapping policies posed a challenge; early thematic reports also needed adjustment when they failed to distinguish response volumes between issues. The update says further evidence is needed on where AI adds value and what safeguards or guidance may be required. The gain is meaningful but conditional: setup and review remain work, and the evidence base is developing.

A fair comparison is manual summarization versus AI-assisted first-pass processing plus the human verification required for the intended use. Measure preparation, correction, review and reporting separately from model run time; the reports do not settle total staff time or equivalent accuracy. Faster processing may create room for analysis or engagement only if a team reallocates the time. Before using a pilot result to change practice, check the task and corpus behind it, the review required, and whether quality was independently assessed. If those details are missing, treat speed as a reason to test locally, not proof the summary is decision-ready.

Sources: Greater Cambridge Shared Planning: AI-Driven Summarisation for Local Plan Consultations; Plymouth, South Hams and West Devon improve analysis with AI; Using AI to support faster local plan consultation analysis

What review is proportionate to the summary's intended use?

The review needed depends on what the summary will do next. For internal triage, a provisional label can help if a planner can correct it before it shapes a conclusion. A public account of consultation themes needs traceable claims, clear counting rules, and visible exceptions. Formal evidence or a recommendation calls for source-level checking, the complete record, and an accountable decision-maker. This is a consequence-based rule, not a fixed audit or universal number of checks. For a low-consequence check, choose comments likely to reveal classification problems: common and uncommon themes, different policy areas or locations, short and long submissions, ambiguous wording, and unusual concerns. Compare each item with its generated label or summary; note omissions, corrections, and cases that could fit multiple categories. The sources reviewed establish no universal sample size or threshold for reliability. If categories collapse issues or drop qualifications, revise them or expand review before relying on the output. For public reporting, identify source comments for consequential themes, state the period and material included, and say whether a count refers to comments, submissions, or another unit. Respondents are the people represented in the record; theme frequency does not establish wider public opinion unless the engagement design supports that inference. Review low-frequency concerns that could matter to a site or policy instead of letting the most repeated theme stand for the whole account. The Ministry of Housing, Communities and Local Government’s Digital Community Engagement Toolkit says officers should review AI results before drawing conclusions and report how engagement informed decisions. This is practice guidance, not a technical accuracy threshold. Formal casework adds a procedural boundary. Planning Inspectorate guidance for the described English planning casework says local planning authorities using AI summaries must provide original case documents; summary-only submissions may fail relevant requirements or deprive an Inspector of material considerations. In that setting, a summary cannot replace the representations. The guidance is specific to its casework context, so authorities must check the procedure that applies. Where analysis supports a recommendation, a planner should trace the reasoning to the retained record, identify what the summary did and did not establish, and own the final interpretation. Review begins before comments enter a tool. Check whether the authority’s privacy, records, procurement, disclosure, and consultation commitments permit the proposed data handling; this depends on jurisdiction, system, and information. Retain the source record and document the tool’s role, corrections, and unresolved limits. NIST’s Generative AI Risk Management Framework recommends evaluating validity, reliability, and limits under intended conditions; it sets no planning-specific threshold. The NYC Department of City Planning’s Voice to Vision register describes staff review of tags, locations, and summaries linked to original statements, and identifies risks including cultural or linguistic misrepresentation. These are control examples, not independent proof of accuracy. Stop when: if reviewers cannot trace a material claim, establish what record it covers, or explain why its interpretation fits the intended use, narrow the claim, add human review, or return to manual analysis. A provisional wayfinding aid may need less scrutiny if staff can correct it before it matters. Public conclusions and formal submissions need review proportionate to their consequences, with original comments available wherever procedure or sound analysis requires them.

Sources: Use of artificial intelligence in casework evidence; Digital community engagement toolkit for planning consultations; Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile; AI-Assisted Summarization of Community Planning Input

What should you do before the next batch is summarized?

Before a small batch is processed, write down its intended use: internal navigation, a public account of themes, or input to a recommendation or formal process. Record which comments are in scope, the categories to apply, who will review the output, and what would make you stop or narrow the task. Keep original comments available. The Digital community engagement toolkit for planning consultations advises reviewing AI results before drawing conclusions; NIST’s Generative AI Risk Management Framework cautions against extending narrow assessments beyond the conditions tested. Include common themes and awkward cases in the batch: comments spanning policies, unclear wording, different locations, and concerns appearing once. Compare claims and classifications with originals. Log corrections, omissions, and categories that misrepresent what people said. This is a practical local check, not a statistically validated sample size. If a material claim cannot be traced or a concern disappears, revise the categories or instructions, narrow the summary, add review, or handle that part manually. Public reporting and formal use require stronger source and procedural checks; a pilot does not replace them. Finally, decide whether the output fits its stated use and name the accountable reviewer. If you are assessing how summarizing and verification affect your own work, the free task checker provides task-level change-pressure signals, not job-loss probabilities.

Sources: Digital community engagement toolkit for planning consultations; Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile

Questions readers ask