How Urban Planners Can Test AI on Work Documents

How can an urban planner test AI on planning work documents before using it in a public process?

An urban planner should start with one repeatable, low-consequence document task, not a broad test of whether AI can do planning. First ask which service and material the organization approves. If unclear, do not put real planning or resident data into a tool. For an authorized internal test, compare AI-assisted work with the ordinary staff process using the same representative, cleared documents. Check material outputs against the source, note omissions and unsupported additions, and count preparation, review, correction, and sign-off time alongside generation time. Keep results internal. A pass supports only the claim that this task worked under these conditions; it does not authorize analysis of a live consultation, an official record, a public summary, or decision support. Those uses change data, audience, or consequences, so ask responsible local governance owners to review purpose, permissions, accountability, and safeguards separately. If an approved tool and documented procedure already cover the exact proposed use, follow that route rather than duplicate a pilot. Define the task observably: extract policy references from a published document, or draft an internal summary whose statements trace to passages. Record document versions, material-error criteria, reviewer, and stop conditions. The NIST AI Risk Management Framework says accuracy measures should use realistic test sets representative of expected conditions, with the method documented. This supports a local comparison, not a universal pass score. Keep the ordinary staff method as a baseline and count work around generated text: input preparation, source checking, correction, and sign-off. A quick first output may still take more staff time if review is extensive. NIST defines reliability under given conditions and calls for ongoing testing or monitoring, so a short exercise offers bounded evidence, not proof across future documents. The geography matters. The Digital community engagement toolkit for planning consultations, published by England’s Ministry of Housing, Communities and Local Government, says AI can tag comments, summarize feedback, or identify patterns, while officers should review results before drawing conclusions. It distinguishes formal consultation responses, part of the official record, from operational feedback. The toolkit does not replace statutory requirements and is England-specific. An internal test measures task performance; permission to use data or outputs in a public process requires local approval.

Sources: Digital community engagement toolkit for planning consultations; AI Risk Management Framework resource: AI risks and trustworthiness

Which planning document task makes a useful first test?

Choose a repeatable task with a clear source of truth, low sensitivity, and a reviewer who can verify the result without relying on the tool. A useful exercise might extract policy references from a published local-plan document or draft an internal summary with each statement tied to a passage. The test concerns that defined task under stated conditions; it is not a test of whether AI can “do planning.” Separate the task from the document category. A consultation file might be represented by published, cleared material or synthetic text, while actual resident responses may contain personal or confidential information and form part of an official record. The Digital community engagement toolkit for planning consultations describes formal responses as part of the official consultation record and says AI can help organize, tag, summarize, or identify patterns, with officers reviewing results before drawing conclusions. That supports defining the boundary before choosing inputs; it does not authorize uploading a live file to any service. Ask the responsible local information-governance or data-protection contact which tool and data route are allowed. If unclear, pause real-data testing and ask whether published material can serve instead. Write a short task specification first: who will use the output, which input version is in scope, what instruction the tool receives, the required output form, and what it must not be used for. “Summarize this policy” is loose if the actual need is to list references, quote relevant passages, and flag uncertainty. A narrower instruction separates clerical drafting from professional interpretation. Keep the source available so a reviewer can trace each result back to it. Choose material reflecting ordinary difficulties. For a policy document, that might include passages with tables, cross-references, repeated wording, or terms whose meaning depends on a nearby definition. These cases reveal missed relationships hidden by polished output. NIST’s AI Risk Management Framework resource on AI risks and trustworthiness calls for testing intended systems in conditions resembling expected use and with representative test sets. Applied here, that supports including relevant document variation; a small sample still cannot prove performance across all plans, consultations, or authorities. Note how staff complete the task now and where effort lies. Record steps such as copying references, drafting from several sections, or having a second planner check citations; do not assume generation time is the whole job. For an internal summary, reviewers can check whether each statement is supported, qualifications remain, and the source trail is usable. The NIST Generative AI Profile describes “confabulation,” including confidently presented false content and fabricated citations. This cross-sector guidance does not establish an error rate for a planning tool, but it explains why traceability belongs in the test design. Keep the task narrow enough to explain why the sample is permitted and who can verify it. If it requires live responses or conclusions about residents collectively, resolve its data route and purpose with the local owner first; it exceeds a simple document trial.

Sources: Digital community engagement toolkit for planning consultations; Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile; AI Risk Management Framework resource: AI risks and trustworthiness

How should the planner compare AI-assisted work with staff work?

Use the same approved document sample and task definition for the ordinary staff workflow and the AI-assisted workflow, then compare each result with the underlying source. Before starting, write down what the task asks for and what counts as a correct, usable answer. For example, if the task is to extract policy references from a planning document, the output should identify each reference and provide enough source context for a planner to check it. Keep the task fixed: the comparison concerns this bounded piece of work, not planning in general.

Record the document versions, tool and version if available, settings, instructions or prompts, and any preparation applied to the files. Define likely error categories before reviewing outputs: missing source material, unsupported addition, incorrect extraction, changed qualification, or an answer that cannot be traced back to its source. Keep the source passage beside each extracted claim so a reviewer can verify it. Record who reviewed each output, what they corrected, and how disagreements were resolved. This records the method as well as the result, so changes in instructions or documents do not obscure what was tested.

Measure the full staff effort for both routes. Include selecting and clearing documents, redaction or formatting, preparing instructions, prompt iteration, checking, correction, and sign-off. A short generation step can still leave more work overall if staff must search for evidence or repair subtle errors. The government account “Using AI to support faster local plan consultation analysis” reports a striking processing-time comparison, while also describing setup, policy-tagging problems, iteration, and officer quality checks. That is a reason to count the whole workflow locally, not to assume another task will produce the same saving. Count generation time, but also the effort needed to make the output dependable for internal use.

Where practical, have a reviewer compare the outputs with the source documents without being told which version came from which workflow. The reviewer still needs planning knowledge, but this can reduce the pull of fluent wording. NIST’s “AI Risks and Trustworthiness” says validation should be tied to a specific intended use and realistic test sets representative of expected conditions; it also says accuracy measures should be documented and may need to account for different data segments. NIST’s “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile” describes how generated text can be misleading, including when it appears plausible. Together, these sources support checking traceability and errors against actual documents rather than scoring style alone.

A one-off demonstration can reveal whether the task is feasible and expose obvious failure modes. It cannot establish repeatability across document types, consultation stages, policy vocabularies, or future tool versions. If the task matters enough to continue, repeat it on additional permitted documents that reflect the formats and awkward cases staff expect, with the same criteria, recording any changes. A small pilot can inform a narrow internal decision; it is not a universal benchmark or proof of public-process readiness. The useful result is a comparison that shows total effort, error patterns, reviewer work, and conditions that need another test, so the planner can decide whether to stop, refine, or request approval for a separately scoped next step.

Sources: AI Risk Management Framework resource: AI risks and trustworthiness; Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile; Using AI to support faster local plan consultation analysis

Which document errors should stop a test or trigger redesign?

Do not judge a planning output by fluency alone. Check each material claim, reference, qualification, category, and omission against the source. Smooth wording can still misstate a policy, merge distinct views, or drop a qualification that changes the meaning. Treat a claim that cannot be traced to its source, a fabricated reference, lost disagreement, or misleading emphasis as a substantive error. Correcting one may be possible, but polished language by itself is not evidence that the test succeeded.

The NIST Generative AI Profile describes confabulation: generative systems can produce false or internally inconsistent content, including plausible-looking citations. For every material summary statement or policy reference, record the source passage that supports it. If a reviewer cannot find that support, mark the claim unresolved instead of accepting authoritative-sounding wording. Check that a cited reference exists and actually supports the attached claim.

Consultation analysis needs a representation check. Compare the output with responses for disagreement, qualifications, conditional support, and concerns raised by fewer participants. A summary may retain the most common theme and still distort the record if it erases an opposing view or makes conditional support sound unconditional. Keep a respondent’s stated position distinct from an officer’s interpretation of its planning significance. The Digital Community Engagement Toolkit for Planning Consultations says officers should review AI-assisted analysis before drawing conclusions.

Categories and counts can mislead even when individual summaries look reasonable. Check for overlapping policy themes, duplicate wording counted as separate evidence, and labels that do not fit the local policy question. Preserve enough traceability to see which responses contribute to each category and how ambiguous cases were handled. The England toolkit describes AI support for tagging, summarizing, and identifying patterns; it does not set a universal validation threshold.

Set stop conditions before reviewing results, with responsible colleagues considering the consequences of error. A typo that leaves meaning intact may be correctable in an internal drafting task. A false policy citation, changed qualification, or summary that alters the apparent balance of views calls for correction and may require narrowing or redesign. The PlanAI government account reports inaccurate policy tagging alongside setup and iteration work, a reason to log failure types rather than report speed alone.

For each issue, record what was wrong, its consequence, how it was detected, the correction, who checked it, and remaining uncertainty. A proposed local rule is to continue internally only when material errors can be found and corrected through an accepted review process. If reviewers cannot reliably identify unsupported claims, lost views, or category distortions, stop or redesign before relying on the output. This is a proposed decision rule, not an experiment conducted for this article; no single numeric accuracy threshold is safe for every planning task. The planner should distinguish manageable editing from a failure that undermines the source record or requires expert review before a next stage.

Sources: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile; Digital community engagement toolkit for planning consultations; Using AI to support faster local plan consultation analysis

What do the planning pilots establish—and what do they leave open?

The published England PlanAI account makes faster consultation processing plausible in the participating settings, but it does not show that another planner, authority, document set, or tool will achieve the same speed or quality. In “Using AI to support faster local plan consultation analysis,” the Ministry of Housing, Communities and Local Government’s Digital Planning team reports that the initial Greater Cambridge Shared Planning and University of Liverpool trial that analysis time fell from around 18.5 hours to approximately 16 minutes. Five further planning authorities joined later testing, with reported efficiency gains of around 90%. These are government-reported pilot figures, evidence that consultation summarisation merits a bounded test. They concern a workflow task and say nothing about staffing reductions or job-loss probability. The post lacks controlled methodological detail, denominators, and independently reported accuracy results, so the figures are not forecasts for another workplace. It does not establish that the comparison included every workflow step.

The same “Using AI to support faster local plan consultation analysis” account says Greenwich officers found that some comments were tagged to the wrong policies, and overlapping policies made classification harder. The team says clear policy names and well-structured documents matter to the tool’s behavior. In some participating authorities, redacting responses, formatting data, and iterating toward usable outputs took two to three weeks. Planning officers still needed to check the outputs; some thematic reports initially failed to distinguish the volume of responses on different issues and needed parameter changes. It says that it does not offer advice, recommendations, or weighting of views. A time figure cannot establish interpretive quality, public-explanation value, or suitability for another task.

The result may change with document structure, sample volume, overlapping categories, redaction and formatting needs, local policy context, configuration, and the stages counted as “analysis” or “efficiency.” These are reasons to measure the planner’s own proposed workflow, not claims that any one factor explains the published results. In a local test, record preparation, review, correction, and sign-off alongside generation time, and inspect errors against the source. The pilot figures give a reason to test where manual grouping is burdensome; they do not remove the need to account for checking work or substantive mistakes.

The Greater London Authority’s “ADD2869 Use of Plan AI to support London Plan consultation analysis” decision record offers a useful example of a context boundary. In September 2026, the authority approved expenditure to procure PlanAI to support initial analysis of draft London Plan consultation representations. Its record says the tool had been developed primarily for Local Plan consultations and had not yet been tested on a Strategic Development Strategy such as the London Plan, although it described prior testing in other contexts and retained officers’ responsibility to review every representation and exercise professional judgment. This was a decision record about planned use, not a completed evaluation of London Plan results. It shows that experience in one planning context did not answer every question about a larger-scale strategy. A successful pilot elsewhere can support a carefully scoped local inquiry; it cannot substitute for checking the exact task, evidence, and authorization in front of the planner.

Sources: Using AI to support faster local plan consultation analysis; ADD2869 Use of Plan AI to support London Plan consultation analysis

When does faster processing count as a useful result?

Speed counts only when it improves the whole planning task on terms the authority can defend. Compare the workflows by quality, full effort, usability, and risk. The OECD paper *Generative AI Experimentation in Government* offers a lens for this judgment. It reviews official guidance across 14 countries, academic literature, and case studies, and identifies weaknesses in systematic monitoring and evaluation. Its proposed framework considers performance, public value, feasibility, usability, and risk management. This is a synthesis framework for government experimentation, not a planning-specific benchmark or local rule. It helps keep speed from standing in for the whole result. For performance, ask whether the output preserves source meaning and whether important statements can be traced back to it. Feasibility includes preparing documents, licensing or configuring a service, supporting staff, and fitting the task into existing review and records processes. Usability asks whether planners can understand the output, recognize its limits, and challenge it. Risk management asks whether the data route and safeguards match the proposed use. Public value asks whether the task helps the authority respond to residents and planning evidence, or merely shifts labor from sorting to verification. A faster draft may be valuable if staff use the time saved to examine difficult representations or explain what was heard, but that is a benefit to test, not assume. Planning guidance makes these measures concrete. *Using community engagement platforms in planning consultations* asks councils to consider accessibility, security, data protection, support, storage, reporting, and the way digital methods work alongside offline engagement. It also identifies licensing, configuration, training, maintenance, and supplier support as possible costs. The guidance concerns engagement platforms broadly, not proof that a particular AI task improves participation. The surrounding workflow matters: processing gains have limited value if a route excludes residents or obscures how feedback will be used. There is a reasonable case for supervised use even when output is imperfect. *Planning and preparing for artificial intelligence implementation* says a model that does not exceed human accuracy may still be useful if it works at greater scale and speed. Assistance could let staff cover more material while retaining responsibility. The same guidance separates prototype evaluation from integration with live data and decision workflows, calling for performance testing, continuous evaluation, and users able to interpret and challenge outputs. This supports only a bounded continuation when local evidence shows that the assisted task is worthwhile and review safeguards fit the use. A useful result is therefore a documented reason to keep, narrow, repeat, or stop one supervised task, not a declaration that AI is ready for planning. If correction consumes the time saved, source meaning is lost, staff cannot challenge output, or accessible participation is weakened, speed does not justify progress. If the task remains worthwhile after effort and risks are counted, the pilot may support a limited next step under local controls. It does not establish suitability for another document set, audience, or public purpose.

Sources: Generative AI Experimentation in Government; Using community engagement platforms in planning consultations; Planning and preparing for artificial intelligence implementation

What is changing when an internal test becomes public planning work?

Moving from a staff-only exercise to live consultation responses, an official record, a public summary, or decision support changes more than the audience. It changes which information is handled, who may rely on the result, what the authority must explain, and what an error could affect. Treat that move as a new intended use requiring local review, even if the same tool and prompt are involved. Capability, successful local use, organizational approval, and permission for public deployment are separate findings. For authorities in England, the Ministry of Housing, Communities and Local Government’s *Digital community engagement toolkit for planning consultations* distinguishes formal responses, which form part of the official consultation record, from operational feedback about the platform. It says AI may assist with tagging comments, summarizing feedback, and identifying patterns, while officers should review results before drawing conclusions. The toolkit also says it does not replace statutory planning guidance or requirements. These are England-specific directions; planners elsewhere need to check the rules and procedures that govern their own authority. In either setting, an internal draft summary is not itself a public explanation of what residents said, how the authority interpreted it, or how their input will be used. Before any change in use, ask the responsible local owners to confirm the precise purpose, input material, service and data route, and accountable reviewer. Check privacy and security controls, supplier terms, retention and access to records, source traceability, correction and sign-off responsibilities, and how uncertainty will be described. The government toolkit asks authorities to plan for accessibility and inclusion; its guidance on community engagement platforms also says to retain offline ways to participate, check accessibility and data-protection requirements, and explain how feedback will be used. So review whether a public output is understandable and accessible, whether residents can still take part through appropriate non-digital routes, and whether the authority can correct or challenge the analysis. A Greater London Authority decision record shows why context must be stated precisely: it sought approval to use PlanAI for initial analysis of London Plan representations, described safeguards and continuing officer review, and noted that the tool had not yet been tested on a Strategic Development Strategy such as the London Plan. The decision record says London Plan representations would be made available for public inspection and expected to appear in the examination library. That record documents a specific proposed use and its limits; it is not a general permission or an independent finding that another authority’s workflow is ready. The practical boundary is therefore clear: do not upload live material or rely on generated findings in public work until the authority confirms that the exact purpose, inputs, controls, accountability, and participation safeguards are covered. A successful internal test can inform that review, but cannot substitute for it.

Sources: Digital community engagement toolkit for planning consultations; Using community engagement platforms in planning consultations; ADD2869 Use of Plan AI to support London Plan consultation analysis

How should the next approval gate work?

Advance one stage at a time, only when evidence and local permission fit the proposed use. A document test does not earn an “AI-ready” label. It informs whether to stop, narrow or repeat the task, continue with a bounded internal aid, or request review of higher-consequence use. Ask what staff would do with the output, who would see it, and what it could affect; changed answers may require different evidence and approval.

Stop if the service, data route, or material is not clearly permitted. A successful result cannot answer whether the organization may use that tool or upload those documents. Ask the responsible local owner before using real planning or resident material. If permission is clear, compare the task with ordinary staff work on representative, cleared documents. Narrow or abandon it if claims cannot be traced, themes disappear, source errors recur, or checking outweighs the proposed benefit. A reviewer must explain why remaining errors are acceptable; polished language does not establish correctness.

If the test supports a useful internal aid, take its findings through the organization’s governance process. Record the task, inputs, tool and data route, review method, errors, total preparation and correction effort, limits, and accountable sign-off. The NIST AI Risk Management Framework resource describes validation for a specific intended use and continued evaluation. This supports task-specific evidence, not universal approval. OECD’s 2026 review frames government experimentation across performance, public value, feasibility, usability, and risk management. Neither source sets a planning approval sequence.

Treat live consultation data, official records, public summaries, or decision support as a new stage. Design a separate evaluation and obtain local approval: audience, data, accountability, and consequences have changed. UK government guidance on planning and preparing for AI implementation describes evaluating prototypes before progressing toward live data and workflows, with evaluation continuing during implementation. Treat this as an analogy, not a universal rule. Monitor whether task, inputs, users, or failure modes differ from the pilot, and return for review when they do.

One exception may make a new pilot unnecessary: an approved platform and documented procedure cover the exact task, data, audience, and consequence. Follow it and confirm it applies to the actual documents and purpose. The recommendation would also change if robust local evaluation and explicit authorization already covered the proposed public use. A vendor demonstration, test on published documents, or another authority’s result does not establish either condition. The outcome is a reasoned next decision with defined scope and owner, not a blanket readiness claim.

Sources: Planning and preparing for artificial intelligence implementation; AI Risk Management Framework resource: AI risks and trustworthiness; Generative AI Experimentation in Government

What should the planner record before asking for approval?

Before asking for approval, prepare a short record showing exactly what was tested and what you want to do next. Identify the task and intended use, approved service and data route, tool and version, and source documents by title, date, and version. Note which formats or consultation stages were represented and which were absent. This gives reviewers a clear boundary for interpreting the result instead of a general claim that “AI works” for planning. Describe the ordinary staff baseline and criteria set before comparison. Keep the instructions or settings, output, corrections, and source checks together. Summarize the sample, errors found, how they were detected and corrected, unresolved issues, who accessed the material, and total staff time for preparation, checking, correction, and sign-off. Separate observation from interpretation: report what the tested documents showed, then label any conclusion about wider suitability as an inference. Make a narrow sample visible as a limitation. The Digital community engagement toolkit for planning consultations distinguishes analysis support from officer review before conclusions; the NIST AI Risk Management Framework calls for documented, context-specific evaluation. They support a reviewable handoff, not a universal approval form. Follow the authority’s route to involve relevant planning, information-governance, privacy, digital, procurement, or communications colleagues as needed; End with one request: permission to repeat the internal test, a scoped evaluation of a new use, or a decision to stop. The record should help the accountable owner approve, narrow, or decline that next step.

Sources: Digital community engagement toolkit for planning consultations; AI Risk Management Framework

What should the planner do next—and what could change the decision?

Write down one candidate task and ask the organization’s information-governance or data-protection contact which tool and material are permitted. If permission is unclear, do not upload real planning or resident material; ask whether cleared published documents can support an internal test. Record the task and intended use, approved data route, document versions, staff baseline, output criteria and stop conditions, reviewer, full preparation and correction time, source trace for errors, and decision owner. An existing approved platform and documented procedure may remove the need for a new pilot when they cover the exact data, task, audience, and consequence. The Greater London Authority’s PlanAI decision record shows the context boundary: earlier testing in other planning settings did not mean the proposed strategic-plan use had already been tested. Another authority’s result or a vendor demonstration does not establish local permission. A limited test also does not authorize public-process use. The free task checker can map change pressure across the planner’s tasks; it cannot approve a tool or predict job loss.

Sources: ADD2869 Use of Plan AI to support London Plan consultation analysis; Digital community engagement toolkit for planning consultations

Questions readers ask