Adding AI to a workflow changes more than the tool used to complete a task. It can change which information is processed, how a decision is explained, who notices an error and whether a customer can reach a person. The first design question is therefore not “Which model should we use?” but “What exact work are we trying to improve, and what harm could a wrong result cause?”
Write a one-sentence purpose
Name the user, task and intended improvement. “Help a support agent find relevant policy passages faster” is testable. “Use AI for customer service” is too broad to define data, evaluation or responsibility.
Record what remains outside the system. An assistant may draft a reply while a person remains responsible for checking account-specific facts and sending it.
Classify the consequence of an error
Use a simple risk table:
- Low consequence: an internal suggestion that is reviewed before use.
- Material consequence: an incorrect output could waste money, mislead a customer or expose confidential information.
- High consequence: the output may affect access, employment, credit, safety, legal rights or another significant decision.
As consequence rises, strengthen human control, evidence, access restrictions and specialist review. Some uses should not proceed merely because a technical demonstration works.
Map the information before connecting it
List every input, where it comes from, who owns it, why it is needed and whether it contains personal or confidential information. Remove fields that do not contribute to the stated purpose. Decide whether data may be retained by a provider or used to improve its services, and configure the service accordingly.
ICO guidance emphasises considering data protection from the design stage and limiting personal information to what is necessary for the purpose. If the proposed use of personal information is likely to create a high risk to people’s rights and freedoms, the ICO says a data protection impact assessment must be completed. Confirm the assessment with an appropriately qualified person for the organisation’s circumstances.
Design the human review point
“Human in the loop” is meaningful only when the person has time, authority and enough information to disagree. Define:
- which outputs always require review;
- what evidence the reviewer sees;
- how uncertainty or missing information is shown;
- how corrections are recorded;
- when the item must be escalated rather than approved.
Do not measure reviewers only on speed. That encourages automatic approval and removes the safeguard the process claims to provide.
Create an evaluation set from real work
Collect representative examples, edge cases and known failures without using more personal data than necessary. Define the expected outcome and have appropriate subject knowledge involved in judging it.
Measure more than average accuracy. Break results down by task type, language, incomplete input and groups that could be affected differently. Include tests for fabricated facts, unsafe instructions, prompt injection, data leakage and refusal when the system should not answer.
Make uncertainty visible
An AI output should not appear more authoritative than its evidence. Where practical, show the source material used, label generated drafts and let the user inspect the underlying record. Do not invent citations or present a plausible sentence as a verified fact.
Build a failure route before launch
Decide what happens when the model, provider or integration is unavailable. Queue work safely, switch to a manual process or tell the user the service cannot complete the task. Silent substitution with an untested model creates a new system without approval.
Log enough information to investigate while avoiding prompts or records full of unnecessary personal data. Alert on failure rates, unusual volume and repeated human corrections.
Tell people what affects them
If AI materially shapes an interaction or recommendation, use clear language about its role and provide an appropriate way to ask a question or reach a person. Explanations should describe the actual process, not marketing language about intelligence or accuracy.
Set review and retirement rules
Name the business owner, technical owner, review frequency and events that trigger reassessment. These can include a model change, new data source, repeated incident, regulatory update or significant shift in the people affected.
Define how the workflow can be paused and how its data, credentials and retained outputs are removed when the use is retired.
Sources and further reading
- ICO artificial intelligence guidance and tools — UK data-protection guidance for organisations developing or deploying AI.
- ICO data protection by design and by default — the source for purpose limitation, data minimisation, lifecycle controls and high-risk DPIA requirements.
- NIST AI Risk Management Framework — a voluntary framework for managing AI risks across design, deployment and operation.
Xapner’s automation and AI integration service starts with the workflow, controls and exception path before selecting a model or provider.