Build an AI Work Sample That Shows Verification

Why is a prompt-only result weak evidence of verification?

A polished answer to a prompt shows that a tool produced a plausible response under those instructions. It does not show whether the output matches an independent reference, or what happens when inputs change. A stronger work sample makes those steps visible: define a bounded task, explain why the data support it, compare the assisted result with a simple baseline, and record errors, corrections, and limits. A reviewer can inspect the reasoning instead of treating fluency as proof. Prompt design still matters when the target work calls for constructing instructions. Show the prompt and test whether it produces the stated behavior . For other tasks, prompting is part of the method, not the result to evaluate. “Generative AI Projects: From Demo to Evaluated System” recommends baselines, tests, failure examples, and reproduction steps; NIST’s AI Risk Management Framework calls for documenting context, assumptions, limits, test sets, metrics, and tools. These are design principles, not evidence that a portfolio format wins interviews. A small, reproducible sample can demonstrate task framing and checking on its stated inputs. It cannot establish production reliability, employer demand, or an employment outcome. Choose the task and independent check before polishing the demo.

Sources: Generative AI Projects: From Demo to Evaluated System; AI RMF Core

Which public dataset and question can support a checkable claim?

Choose the question first, then check whether the source’s fields and collection process can answer it. With CFPB complaint data, a reproducible count of published complaints for a named product category and received-date period is supportable. A consumer-harm rate, causal account, or company quality ranking is not. Public access makes a dataset available to inspect; it does not make its records representative of everyone affected by a product. The CFPB’s Consumer Complaint Database allows users to filter, download, and analyze published records. Only complaints sent to companies for response are eligible for publication, after the company responds or 15 days, whichever comes first; complaints referred to some other regulators are excluded. The CFPB explicitly says the database is not a statistical sample of consumers’ marketplace experiences, and that low complaint volume does not necessarily mean little consumer harm. These rules describe database coverage, not the prevalence or severity of problems. That distinction should shape the project question before any AI step. “How many published complaints in product X were received during period Y?” can be answered by a filter and count, provided the category and dates are defined. “Which company treats customers worst?” cannot be inferred from raw complaint totals: companies differ in size and market share, and the CFPB itself advises considering those factors when interpreting volume. Likewise, a rate such as complaints per customer needs a suitable denominator, aligned to the same companies, geography, product definition, and period. Even that denominator would not turn a self-selected complaint database into a representative measure of harm. Without a suitable denominator and comparison population, report counts, not rates or rankings. A usable data note should identify the publisher and database, the retrieval date or release, the fields used, the exact filters, date window, and exclusions. The CFPB’s “How we share complaint data” defines Date received, Product, Sub-product, Issue, and Sub-issue as selections identified by the consumer. A label records a submitter’s categorization, not an adjudicated finding that a company broke a rule. The CFPB documents past category changes, and records retain the choices available when each complaint was submitted. For a long-period analysis, check whether labels remained comparable; if they shifted, restrict the period, group categories with rationale, or show separate periods. The same care applies to dates. Date received records when the CFPB received a complaint; publication may occur later, after response or the 15-day interval. A quarterly count should filter on Date received, not publication date. State endpoint and missing-record rules, and note that recent complaints may not yet appear. This keeps the count reproducible and tied to the database’s publication process. The UK government’s “Guidelines and best practices for making government datasets ready for AI” offers a useful, bounded lens: data suitability depends on intended use and on how data are defined, collected, and captured; missing metadata or provenance can invite misunderstanding. That guidance is written for UK public-sector dataset preparation, not as a universal portfolio rule. Applied here, its practical lesson is modest: inspect definitions, collection and publication rules, historical changes, and permitted use before asking a model to summarize records. If those details cannot support the intended conclusion, rewrite the question. A bounded count is a stronger work sample than a polished answer built on a dataset that cannot establish what the question claims.

Sources: Consumer Complaint Database; How we share complaint data; Guidelines and best practices for making government datasets ready for AI

What is a fair baseline for the same work?

Use the simplest credible method that solves the same task on the same records, with the same filters and expected output. For a count or grouping, that may be a spreadsheet pivot or SQL query. For structured extraction, it may be a rule-based parser or current workflow. For a handful of factual statements, it may be a manual check against the source. In the CFPB complaint-data example, suppose the question is how many published records match a specified product, issue and period. A transparent filter and group-by gives a reference result. An AI system asked to count those records should be checked against that result, not another model's answer. The Consumer Complaint Database makes published data available to filter and analyze, but CFPB cautions that the records are not representative of consumers' marketplace experiences. A baseline can verify a count under stated filters; it cannot turn that count into a measure of harm or company quality. Fix the task contract before comparing approaches: which fields count, which dates are included, how missing values are treated, and what output is expected. Give both approaches the same data slice and information. Otherwise a difference may come from setup rather than method. Record the period and category definitions so a reviewer can interpret the comparison against the database's structure. | Task | Baseline | Possible AI role | Check | | --- | --- | --- | --- | | Count records by category | Spreadsheet pivot or SQL group-by | Draft a plain-language explanation | Re-run query; reconcile totals | | Extract fields from records | Existing rules or source review | Suggest a structured summary | Compare fields with source | | Explain a verified aggregate | Human-written description | Draft explanation for editing | Check claims against figure and limits | If the goal is to demonstrate prompt construction, compare prompts on the same inputs and define success in advance. Prompt design is then part of the method being evaluated. But if the claim is that an AI-assisted workflow calculates correctly, a deterministic query is a more independent check than asking another generative system to endorse it. Make the model's role explicit. After a query produces a verified aggregate, a language model might help draft a readable explanation. Label that contribution as drafting or interpretation, not calculation. Check that the prose preserves the query's scope and does not imply prevalence, causation or performance conclusions the data cannot support. The NIST AI Risk Management Framework Core recommends understanding intended use, comparing capabilities with appropriate benchmarks, and documenting test sets, metrics and tools. These are voluntary organizational risk-management outcomes, not a portfolio rubric or evidence of hiring preference. A small work sample should make its own comparison fair and legible. “Generative AI Projects: From Demo to Evaluated System” also recommends testing a baseline on the same data as the improved system. This supports the comparison design, not claims about employer preferences. A reviewer should be able to see whether AI added a useful step, restated a deterministic result, or introduced an error. If the baseline already solves the task, explain why AI was unnecessary there or identify another step worth evaluating. The comparison demonstrates judgment about this task and dataset, not broad system reliability or a hiring outcome.

Sources: AI RMF Core; Generative AI Projects: From Demo to Evaluated System; Consumer Complaint Database

What should the project record so another person can reproduce it?

A reproducible work sample should let another person recover the same data slice, understand its transformations, and compare the result with the same expectation. Record the publisher, dataset name, retrieval date or version, applicable terms, fields, filters, exclusions, transformation steps, and output expected before the AI-assisted step. Include instructions to repeat the work: an API query, spreadsheet formulas, or code and commands, plus software or model details when they affect results. The goal is a traceable path from source to conclusion. For a bounded count using CFPB complaint records, name the downloaded file or retrieval date, received-date interval, product and issue fields, treatment of blank categories, and formula or query used to group records. If a model only drafted an explanation of the resulting table, say so; keep the count tied to the deterministic calculation. Linking the publisher’s field documentation can prevent a label from being mistaken for a broader concept. The CFPB’s Consumer Complaint Database describes available fields and publication limits, while its data-use documentation explains when complaints appear and what information is not published. Those details affect what the records mean. A compact README can cover source and permitted use; question and data slice; reproduction steps; and known limits. Attach a small input snapshot when terms and sensitivity permit, or explain how to retrieve the same slice from an API. Keep formulas or code versioned, identify manual edits, and distinguish original input from derived files. For a larger dataset, a query and retrieval date may be more practical than copying everything. If a later retrieval changes the result, record the difference so a reader can distinguish a changed source from a changed method. Public availability does not mean every row should be republished. If aggregate fields are sufficient, leave complaint narratives and identifiers out of the portfolio output. State which columns were omitted and retain only what is needed to reproduce the analysis. Check use conditions before sharing an extract. This makes the artifact’s choices inspectable and reduces unnecessary exposure of record-level detail. NIST’s AI Risk Management Framework supports documenting system context, assumptions, limits, test sets, metrics, and tools. GOV.UK’s dataset-readiness guidance highlights metadata, provenance, quality issues, rights, and whether data can be reconstructed. That guidance is written for UK public-sector data preparation, so it is a documentation lens, not a portfolio standard or legal determination for every dataset. A command without field definitions may repeat a mistake consistently; a caveat without a repeatable procedure cannot show how the result was produced. Reproduction has a narrow but valuable meaning. If another person follows the recorded steps and obtains the stated output, that supports the claim that this version of the pipeline is repeatable under those conditions. It does not prove the records represent everyone affected, the interpretation is true, or the method works on another period or source. Nor does it show that an AI system’s public-data examples were absent from training material. Put the audit trail beside the result and the limits beside the conclusion. A reviewer can inspect what happened without inferring what changed between the source and the chart.

Sources: Guidelines and best practices for making government datasets ready for AI; AI RMF Core; Consumer Complaint Database

How do you choose checks that can catch a convincing wrong answer?

Match each check to the kind of answer the project produces, and define the expected result before asking the system to produce it. A count can be reconciled against an independent spreadsheet or SQL query; extracted fields can be compared with labelled source records; a generated factual claim can be checked against the cited passage that is supposed to support it. A second language model can repeat the first answer without supplying an independent reference. Begin by separating format checks from meaning checks. A response may contain valid JSON, use the requested headings, or include a citation link and still be wrong. They can catch malformed output, but cannot establish that a number used the right records or that a sentence follows from its source. In the NIST AI Risk Management Framework Core, the Measure function calls for documented test sets, metrics, and tool details; it also says that performance should be demonstrated under conditions similar to the intended setting and that limits on generalization should be documented. For a portfolio project, this is a design principle, not deployment assurance. Fix the check set before the final run, including ordinary cases, boundaries, and plausible failures. For a public-data count, ordinary cases can confirm that familiar records enter the chosen grouping. Boundary cases should exercise the exact date cutoff, category filter, or treatment of missing values. Plausible failures might include a shifted date range, a changed category label, or a duplicate record. Record the evidence expected, the output, and whether they matched. If a CFPB complaint analysis uses received dates, its data-use documentation explains that publication timing follows a separate rule: complaints are published after a company response or 15 days, whichever comes first. The query must state which date it uses. For text outputs, treat each material factual statement as a checkable unit. Identify the source passage or record that supports it, then mark whether the statement is supported, contradicted, or absent from the evidence. Include one answerable question and one the records cannot answer; the latter tests whether the system withholds claims rather than fills gaps. The guide “Generative AI Projects: From Demo to Evaluated System” likewise recommends test cases and failed examples, including citation accuracy and abstention for grounded answers. This is project advice, not evidence that a format improves hiring outcomes. Keep a short discrepancy log: case identifier, expected evidence, actual output, error type, correction, and retest result. Classify errors such as wrong records, boundary handling, arithmetic, unsupported claims, or failure to abstain. Do not replace the log with an accuracy percentage unless you have a defined set, a counting rule, and a meaningful denominator; a percentage over a handful of selected examples can imply more certainty than the check supports. A colleague's clarity feedback improves presentation; it tests accuracy only when applied through a stated rubric against an independent reference. The result is more persuasive when a reviewer can see not only that a check was performed, but how it could have exposed the wrong answer.

Sources: AI RMF Core; Generative AI Projects: From Demo to Evaluated System; How we share complaint data

What do failures and metrics let a reviewer conclude?

Report outcomes at the level of the task, alongside failures that changed your understanding of it. A single score can make a small demonstration look conclusive while hiding what was counted, which mistakes mattered, or whether the check could catch them. The NIST AI Risk Management Framework (AI RMF) says measurement should document test sets, metrics and tools, include uncertainty and compare results with relevant benchmarks. It also says performance should be demonstrated under conditions similar to the intended setting, with limits on generalizing beyond those conditions recorded.

Start an error log before polishing the result. For a public-data task, useful categories might include selecting the wrong records, misreading a field label, mishandling a date boundary, making an aggregation error, adding a claim the source does not support, or omitting a caveat that changes the interpretation. Record the input or case, expected result, actual result, how the discrepancy was detected, the correction, and what happened when the same check was run again. If the system appropriately declines to answer because the source lacks evidence, record that as an abstention and assess whether the refusal was warranted.

Choose a metric because it answers a question a reader might reasonably ask. For a bounded category label, exact-match can show how many checked labels matched their references. For a count, reconcile the result against a separately implemented query or a manually checked subset. For structured extraction, report field-level correctness so a correct date does not conceal a wrong amount. For a source-grounded summary, inspect whether each material claim is supported by the cited passage and whether unsupported claims were withheld. These checks need defined references and scope: a second model agreeing with the first is not, by itself, an independent reference. The metric should name its denominator and what counted as a match.

Averages can conceal errors that matter. If most records belong to one common category, an overall accuracy figure may look strong even when the system misses a less frequent category that is important to the task. Show the distribution or report results by relevant category when the sample supports that breakdown. For a small project, describe the cases actually checked rather than presenting an unstable percentage as a general performance estimate. Do not combine unlike failures into one score unless the weighting has a defensible reason; a missed caveat and a formatting defect do not necessarily carry the same consequence.

The School of Core AI’s project guide similarly advises showing test cases, failed examples, reproduction steps and limitations, while presenting them as portfolio guidance rather than evidence of hiring outcomes. Put the metric next to its scope, then include a short failure table or log with corrections and unresolved errors. A reviewer should be able to conclude what the workflow did on the stated records, how you checked it, and where it failed. They cannot infer broad reliability, performance on unseen data, or workplace impact from that bounded result alone.

Sources: AI RMF Core; Generative AI Projects: From Demo to Evaluated System

What does a public-data test prove—and what remains unknown?

A public-data test can show that a named workflow passed or failed stated checks on the records and conditions you report. It cannot, by itself, show that the dataset represents everyone affected, that a language model had never encountered those records, or that the same result will hold on new data or in production. Reproducibility means someone can rerun the documented pipeline; evaluation validity means checks measure the claimed task; representativeness concerns whom the data describe; generalization asks whether performance carries to unchecked cases. One does not settle the others. Public availability makes inspection and reuse possible, but it does not make a dataset a neutral picture of reality. The Consumer Complaint Database says its published records are not a statistical sample of consumers’ marketplace experiences and may not represent all consumers. Its publication process includes only complaints eligible to be sent to companies, and excludes some referrals to other regulators. The Bureau also cautions that a low complaint count does not mean little or no harm. So a project that reproducibly counts published records can support a bounded statement about those records and filters; it cannot turn that count into a harm rate or a company-quality verdict. Document use conditions and minimize what you reproduce: if aggregates answer the question, do not copy narratives or identifiers into a public portfolio. There is a separate uncertainty when a language model is evaluated on public material: the evaluator generally cannot inspect its full training history. “An Open-Source Data Contamination Report for Large Language Models,” published in Findings of EMNLP 2024, analyzed more than 15 models across six multiple-choice question-answering benchmarks. It reported contamination estimates from 1% to 45%, varying by benchmark, and found that contamination did not always raise measured performance: reported gains reached 14% on C-Eval and 7% on Hellaswag, while MMLU showed only a minimal increase. This shows benchmark overlap can complicate interpretation and its effect varies. They do not show that CFPB records, your selected cases, or a particular model are contaminated. “How Contaminated Is Your Benchmark? Measuring Dataset Leakage in Large Language Models with Kernel Divergence,” an ICML 2025 paper, likewise reports experiments in controlled contamination scenarios for a proposed leakage measure. This makes leakage a distinct concern, not something diagnosed by an ordinary work sample. A later time period or separately sourced cases can test whether a workflow handles a different slice, but this is a robustness check, not proof that the examples were unseen during pretraining or that results will hold in deployment. UK public-sector “Guidelines and best practices for making government datasets ready for AI” emphasizes intended use, definitions, metadata, provenance, quality and rights. Use it as a documentation lens, not a universal legal checklist. In your sample, state the dataset version or retrieval date, filters, checks, observed failures and limits. Then make the narrow claim those records warrant. Reproducibility makes your process inspectable; it does not establish representativeness, production readiness, safety, hiring demand or a job outcome.

Sources: An Open-Source Data Contamination Report for Large Language Models; How Contaminated Is Your Benchmark? Measuring Dataset Leakage in Large Language Models with Kernel Divergence; Consumer Complaint Database; Guidelines and best practices for making government datasets ready for AI

Which learning move fits the work you want to demonstrate?

The right learning move depends on what the sample should demonstrate. To use AI more thoughtfully in your current field, begin with a task you understand and modest tools: a spreadsheet or short SQL query can establish a baseline, while a focused AI step might classify, extract, or draft an explanation. Show how you frame the question, define an expected result, check errors, and explain limits. An elaborate model adds little when the task is a count that a pivot table can verify. If the AI step adds no useful capability, show that finding clearly instead of forcing it into the workflow. For an upgrade within your role, use domain knowledge to choose a meaningful question, notice misread labels, and decide which mistakes matter. Make that judgment visible: explain the task, baseline, checks, and where a person reviews the result. NIST’s AI RMF Core calls for documenting context, assumptions, limitations, test sets, metrics, and tools. It supports a disciplined record; it is not a hiring rubric and cannot show that an employer will value the artifact. For an adjacent data, operations, research, or analyst task, identify the one missing skill blocking the sample. It might be writing a query, cleaning a file, tracing results to source records, or making a clear chart. Learn that piece through a focused course, documentation, or guided practice, then apply it. A course can provide structure and feedback; a certificate can document completion. Neither alone demonstrates application to this task. The sample shows an example of application and makes your decisions inspectable. If the target role expects software development, include code, tests, and setup to rerun the work. If it expects machine-learning engineering or research, a spreadsheet is unlikely to show the required depth; you may need programming, statistics, data handling, and model evaluation, followed by projects matching that work. A degree can provide sustained study and a recognized credential, but takes substantial time and money. Consider one when the route requires it or you need its depth and structure. For a narrower gap, a project, course, or self-study may fit better. Check prerequisites and credential requirements for actual roles and locations rather than assuming one path fits every market. “Generative AI Projects: From Demo to Evaluated System” recommends baselines, tests, failures, and reproduction steps. Apply them at a scale that fits your aim. A reviewer should see what you did and what the result supports, not infer that polish proves job readiness. The sample can show one task-specific capability and your verification approach; it cannot replace relevant experience, a required credential, or evidence across a role. It does not establish hiring demand or promise an outcome. Choose the least costly next step that closes a real gap in the work you want to do.

Sources: AI RMF Core; Generative AI Projects: From Demo to Evaluated System

What is the smallest useful next step?

Choose a task you understand and write one question a public dataset could answer. Before prompting, read the publisher’s field definitions, collection notes and use conditions. A count of published complaints within stated filters is not a rate of consumer harm. The Consumer Financial Protection Bureau’s Consumer Complaint Database explains what its records include and why they are not representative of all consumer experiences. Let that limit shape your question. Write the expected result and simplest baseline. A spreadsheet pivot or SQL query may be enough for a count; show filters, date range and treatment of missing values. Add AI only if it contributes something the baseline does not, such as drafting a source-grounded explanation. Check fixed cases against the source or an independent calculation. Record one discrepancy, how you handled it and one limitation. NIST AI Risk Management Framework’s Measure guidance supports documenting test sets, metrics, tools and system limits; a small portfolio check is not production assurance. Keep the deliverable small: question, source note, baseline, checks, result and reproduction steps. Ask a colleague familiar with the target work whether they can follow the evidence and see what the conclusion does not say. Unsure which work tasks to examine first? The free [task checker](/ai-job-risk-checker) offers task-level change-pressure signals, not a validated probability of job loss. Use the sample to show one checked capability, then choose a proportionate step: apply the method at work, study a gap, or expand the project if your target role calls for it.

Sources: Consumer Complaint Database; AI RMF Core

Questions readers ask