Designing the Confirm Step: The Human-in-the-Loop That Is Also Your Legal Shield

The sharpest mental model here comes from the Court of Justice of the European Union. In Case C-634/21 SCHUFA Holding (Scoring), decided on 7 December 2023, the Court held that automated credit scoring "must be regarded as an automated individual decision prohibited in principle by the GDPR, in so far as SCHUFA's clients, such as banks, attribute to it a determining role in the granting of credit" (CJEU Press Release No 186/23). Read that carefully. The test is whether the machine's output plays a determining role, not whether a human is nominally present at the end of the chain.

That inverts the comfortable assumption. A bank officer who rubber-stamps whatever the score says has not converted an automated decision into a human one. If the model's output determines the outcome, the decision is still, in substance, automated. Move the logic to your business. If your AI drafts a quote and your team approves it without genuinely reading it, the AI made the decision. The button is decorative. Worse, it is decorative in a way your logs can prove.

The standard for real human oversight is not vague. The Article 29 Working Party Guidelines on Automated individual decision-making and Profiling (WP251rev.01), endorsed by the European Data Protection Board, state that to escape the "solely automated" classification, human oversight must be "meaningful, rather than just a token gesture," and must be carried out by someone with the authority and competence to change the decision (EDPB / Article 29 Working Party guidance).

Two words carry the weight: authority and competence. The reviewer must be able to change the outcome, and must understand it well enough to know when to. A confirm step routed to someone who cannot alter the price, or who has no basis to judge whether the price is right, fails the test no matter how many times they click. For an SME, this often works in your favour. The owner approving a quote usually does have both the authority and the commercial context. But authority is not competence, and neither is worth anything once the review collapses into reflex.

Set the data-protection question aside for a moment, because the confirm step earns its place even when Article 22 is irrelevant. An AI that drafts quotes will, eventually, draft a wrong one. A misplaced decimal. A stale discount. A unit confusion. If that quote goes out under your business's name and the recipient accepts it, you may be bound by it. The confirm step is the gate that catches the hallucination before it becomes a contract.

This value owes nothing to any regulator. It is pure operational risk control, and it applies to every business sending priced output, in every jurisdiction. The design implication follows directly: the confirm step should surface exactly what a human needs to catch a pricing error fast. The number. The basis for it. The deviation from your norms. A confirm step that shows a wall of polished prose and a single green button is optimised for speed, not for catching the rare quote that is wrong.

Here is the uncomfortable engineering truth. The more reliable your AI gets, the worse your human reviewer gets at catching its errors. That is not a motivation problem. It is a documented human-factors phenomenon. In her 1983 paper "Ironies of Automation," Lisanne Bainbridge described how, as an automated system becomes more reliable, the human monitor becomes less able to catch its rare failures (Bainbridge, Automatica vol. 19 no. 6, 1983).

Applied to your confirm step: a model that is right most of the time trains the reviewer to approve the next output without looking. The button quietly converts from a control into a rubber stamp, and the timestamps record the conversion. If every approval lands within a couple of seconds of the draft appearing, you have built a machine that manufactures evidence against your own claim of meaningful review. The defence is design, not discipline. Vary what the reviewer must attend to. Force a second look when the output deviates from your norms. Make the confirm step harder, on purpose, exactly when the stakes or the anomaly are higher.

The two regimes ask for different things, and a confirm step serves both without being a magic word in either.

In the EU, beyond Article 22, there is a transparency duty that does not care whether the decision is significant. EU AI Act Article 50 requires that providers and deployers of AI systems intended to interact directly with natural persons inform those persons that they are interacting with an AI system, unless that is obvious to a reasonably well-informed, observant, and circumspect person (EU AI Act, Article 50; the Act is Regulation (EU) 2024/1689). So if your AI is talking to a customer directly, disclosure is a separate obligation from anything Article 22 covers.

Singapore takes a different shape. The PDPA contains no equivalent to GDPR Article 22 and gives individuals no standalone right against solely automated decisions. The PDPC's Advisory Guidelines on Use of Personal Data in AI Recommendation and Decision Systems instead map existing PDPA obligations, Consent, Notification, Accountability, and Accuracy, onto AI systems, and treat human oversight as risk-proportionate good practice rather than a blanket mandate (PDPC Advisory Guidelines). The oversight vocabulary Singapore uses, human-in-the-loop, human-over-the-loop, human-out-of-the-loop, originates in the Model AI Governance Framework (Second Edition, January 2020) and was extended to generative AI by the AI Verify Foundation and IMDA's Model AI Governance Framework for Generative AI on 30 May 2024 (AI Verify Foundation and IMDA).

The common thread across both regimes is accountability: who decided what, and on what basis. A confirm step that logs an authenticated human decision against a specific output is the artefact both regulators recognise.

One limit is worth stating plainly, because it is widely misunderstood. A human approval at the output stage does not cure a defect in the underlying data. If the personal data feeding your AI was collected unlawfully or is inaccurate, the GDPR Article 5 violation occurred at the point of processing, that is, collection, inference, or storage, not at the point where someone approved the draft (GDPR Article 5). The confirm step governs the output. It does nothing for an upstream data problem. Treating "a human approved it" as a blanket cure is the same error as treating the button as an Article 22 shield. It assumes the human's presence launders everything beneath it.

Origin Pi builds the agent-ready business layer, a governed business brain in which an agent's actions are constrained by, and recorded in, a structured, machine-readable record of who decided what and when. The argument of this post is the argument behind that thesis. Governance does not live in the presence of a confirm button. It lives in the logged, authenticated, replayable record the button produces. A confirm step designed this way is either your defence or, if it logs nothing but reflex clicks, the prosecution's exhibit. If you are wiring AI into quoting and customer replies, see how we approach this in sales automation and how the audit trail underneath it is built for AI compliance.

• The CJEU held in SCHUFA that automated credit scoring is an automated individual decision prohibited in principle by the GDPR where clients attribute to it a determining role; the test is whether the output plays a determining role, not whether a human is nominally present.
• EDPB-endorsed Article 29 Working Party Guidelines (WP251rev.01) require human oversight to be meaningful rather than a token gesture, by someone with authority and competence to change the decision.
• EU AI Act Article 50 requires informing natural persons that they are interacting with an AI system unless obvious, independent of whether the decision is legally significant.
• The EU Artificial Intelligence Act is Regulation (EU) 2024/1689.
• Singapore's PDPA has no equivalent to GDPR Article 22; the PDPC Advisory Guidelines map existing PDPA obligations onto AI systems and treat human oversight as risk-proportionate good practice.
• Singapore's human-in/over/out-of-the-loop oversight taxonomy originates in the Model AI Governance Framework (Second Edition, January 2020) and was extended to generative AI by AI Verify Foundation and IMDA on 30 May 2024.
• A human approval at the output stage does not cure a GDPR Article 5 accuracy or lawfulness defect; the violation occurs at the point of processing.
• Automation complacency means a more reliable automated system makes the human monitor less able to catch rare errors, so a static approve button decays into a rubber stamp.