Back
Blog / 
Strategy & ROI

What regulation changes in a customer-service AI project

written by:
David Eberle
A magnifying glass over a blue speech bubble shows a small stack of checked papers, the record behind an answer.

Published · Updated

A bank, an insurer or a medical-device maker wants the same things from AI in customer service as any other company. It should answer in minutes instead of days, take routine cases off the team and help customers when the office is closed. What differs is the second test the project has to pass. Months after go-live, an auditor, a regulator or your data protection officer may ask how a particular answer came about, who was responsible for it and where the customer's data went.

Whether you can answer depends on a handful of choices made before the pilot: who the AI is talking to, what it may do without a person, what its replies contain and what evidence you keep. Make them early, and the approval review checks a design you can explain. Leave them open, and the review sends you back to rebuild a setup that already runs. The paperwork for that review is covered in What to prepare for customer-service AI approval. The design decisions come first.

Know who is writing before the AI acts

In some sectors, what a customer writes can create a duty for the company. Gambling operators may have to act on signs that a player is at risk, and financial services have rules or guidance on customers in vulnerable circumstances. Imagine a message to a bank's chat late at night: "Since my husband died I can't keep up with the loan payments." If the chat was anonymous, the bank may have a duty it cannot act on.

There are two workable designs. You can ask people to identify themselves before the conversation starts, which costs some convenience for everyone. Or you can allow anonymous questions and decide in advance what happens when a message like that arrives. For example, the AI hands the conversation to a person and asks for contact details as part of the handoff. What doesn't work is discovering the gap after the first such conversation.

Identify first

  1. Identify before chat
  2. Start the conversation
  3. Hand over with contact details already known

Trade-off: every customer meets this step at the start, including those with a simple question.

Anonymous first

  1. Allow anonymous questions
  2. Human help is needed
  3. Collect contact details during the handoff

Trade-off: easy to start, but you depend on a planned handoff that asks for details, and the customer can decline.

Two ways to make sure a person can follow up. Contact details collected in a handoff are not a verified identity, and neither path guarantees a follow-up or resolves a crisis on its own.

Decide what counts as verified, too. An email address or a logged-in session isn't always a confirmed identity, and reading a balance needs less certainty than changing a payout account. Set the level for each type of request before the AI may act on it. The permissions guide shows how to enforce that per action.

Identity also runs the other way. Since 2 August 2026, Article 50(1) of the EU AI Act has applied to providers of AI systems that interact directly with people. They must design those systems so that people are informed they are dealing with an AI system. The duty doesn't apply where that is obvious to a reasonably well-informed person from the context, and there is a narrow exception for authorized law-enforcement systems. The information has to be given at the latest at the first interaction. The design duty sits with whoever provides your system, but customers see it in your chat and email wording, so check that wording with your vendor.

Let review capacity set what the AI covers

Some organizations must, or choose to, have a person review AI conversations: every one, or a defined sample. That review only works if the reviewer understands what was said, including nuance a translation can lose.

So the reviewers set the AI's reach. Say your service team reviews in German and French. An AI that answers in every language would then produce conversations nobody responsible can check, so either limit it to those languages or plan review for the others. The same goes for channels and volume. If a person must approve every AI reply, that time belongs in the staffing plan, not in the hope that review will soon become unnecessary.

The handoff is a review moment as well. When a person takes over a case, they read what the AI said and did. A short, accurate summary in the case record makes that check faster and leaves a better trail.

Decide in advance what the AI hands over whole

Some limits in a regulated service lower the automation rate on purpose. Write them down as rules before the pilot, with an owner, so nobody later treats them as failures to fix.

  • Several requests in one message. Suppose a customer asks an insurer to update their address, confirm a payment and explain a rejected claim. The AI can do the first two. It can answer those and pass on the third, or hand the whole message to a person. The partial answer helps the customer sooner but splits the case between the AI and a person; the whole-case handoff keeps one accountable person. Decide which you want, and how each counts in your resolution figures.
  • Topics where judgment carries risk. A message can read like a routine request and still report an injury or a safety hazard. Route such topics to a person by rule, and decide who may change that rule.
  • Rules with exceptions. If the right answer depends on conditions the AI can't check, it shouldn't pick one. Let it explain what applies in general, say that individual cases can differ, and pass the specific case to a person.

Keep personal data out of replies, and know where it is processed

The GDPR requires personal data to be "adequate, relevant and limited to what is necessary" for the purpose (Article 5(1)(c)). Apply that to what an AI-sent reply contains, not only to what the AI reads.

Many email systems quote the customer's earlier message below the answer. If that message held a policy number and a diagnosis, the reply sends both again. Decide whether AI replies may quote earlier messages at all, or whether automatic removal of identifiers is reliable enough for you.

Then ask where processing happens, not only where data is stored. Under Chapter V of the GDPR (Article 44 sets the general principle), personal data may go to a country outside the EEA only under the conditions that chapter lays down. Data stored in the EU can still be sent elsewhere for an AI model to process it. Ask your vendor where each processing step runs, including the model, and record the answer in the data flow your reviewers will check.

The same question applies to AI you already license. An assistant built into software you use every day may be ruled out for customer service by where it processes data, however capable it is.

Build the audit evidence while you pilot

Auditors and contracts ask you to show that a process is under control. That evidence takes time to build, so start it with the pilot rather than after it.

  • Run in parallel first. Let the AI work alongside your team while people check its output before customers rely on it. Add knowledge sources and system connections one at a time, so each change can be judged on its own.
  • Keep some human steps on purpose. A step can be ready to automate and still be worth keeping manual through your next audit. A period of recorded human checks shows the process is stable before you hand it to the AI.
  • Ask whether validation applies. Quality systems in some regulated industries, such as medical devices, require software used within the quality system to be validated. Ask your quality team early whether customer-service software falls within that scope. If it does, validation is a project with requirements, test cases and sign-offs, not a form.
  • Name what you can't control. A customer-service AI may contain parts you can neither configure nor trace, such as the step that routes a request between AI agents. Ask for them to be described in the contract in plain language, so reviewers know where your oversight ends.
  • Agree how problems travel both ways. The vendor's monitoring covers what the vendor sees. Agree response and resolution times, weekend cover if you need it, and how your team reports something it notices.
  • Keep changes traceable. If business users or an AI assistant can change the AI's instructions, record who changed what and why, and keep business rules out of reach of those changes.

Who decides what before the pilot

DecisionExampleDecide with
How the AI learns who it is talking to, and what it does when a message triggers a dutyIdentify before chat, or hand over and collect contact detailsCompliance, service lead
What counts as verified for each requestReading a balance versus changing a payout accountService lead, IT security
How people learn they are talking to an AIWording in chat and email templatesService lead, legal, vendor
What reviewers can coverLanguages, channels, volume and review timeService lead
What the AI hands over wholeMessages with several requests, safety-relevant topicsProcess owner, quality
What replies may contain and where processing runsQuoted earlier messages, model locationData protection officer, IT security
What evidence you keepParallel running, human steps through the next audit, validation scope, contract termsQuality management, procurement

When you get to individual actions, What should a customer-service AI be allowed to do? goes into permissions, and Who owns the case after an AI handoff? covers the moment a case moves to a person.

These recommendations draw on Typewise conversations with service teams in regulated organizations in 2026 and on the legal texts cited above. They are not legal advice; read your own obligations with your compliance team.

FAQ

Which decisions should we settle before a pilot in a regulated service?

How the AI learns who it is talking to and what counts as verified, what your reviewers can cover, which cases the AI hands over whole, what replies may quote, where processing runs, and which human checks you keep through your next audit. The table above maps each to the people who usually decide it.

Do we have to tell customers they are talking to an AI?

For AI systems that interact directly with people, yes in most cases. Since 2 August 2026, Article 50(1) of the EU AI Act has required providers to design them so that people are informed they are dealing with an AI system, unless that is obvious from the context. Agree the wording in your chat and email templates with your vendor and your legal team.