An AI support launch needs more than a date. It needs an agreement about what the system may do, who is watching, and what evidence will justify giving it more responsibility.
Our recommendation is to separate three decisions: whether the system behaves correctly in testing, whether it can operate acceptably on live requests, and whether the business case works. A good answer in a demo does not settle all three.
This is a suggested pilot design, not a report of completed launches or a promised supervision period.
Agree the scope and the permissions
Start with a pilot brief that your support lead and implementation owner can both sign off. Make the boundary clear enough to classify an incoming ticket.
Record:
- Scope: ticket types, channel, languages, and customer groups included, plus explicit exclusions.
- Knowledge: approved sources, who maintains them, and what happens when information is missing or conflicting.
- System access: what the AI may read and which actions, if any, it may take.
- Ownership: who decides whether the pilot advances, who investigates problems, and who can pause it.
Separate permission to send a message from permission to change a record. A draft-only setup should not quietly make production changes in the background. Check permissions at the connected system as well as in the instructions.
A useful pilot hypothesis is specific: can this setup resolve the chosen requests at the agreed quality while reducing total handling effort? If the scope is still vague, first estimate how much of your customer service could be automated.
Give each testing stage a clear purpose
Consider offline testing, supervised live handling, and limited autonomous handling: distinct modes with explicit permissions, not a mandatory timetable.
Offline testing checks whether the proposed setup can handle relevant cases before it affects customers. Use authorized historical tickets or realistic test cases with known expected outcomes. Keep evaluation cases separate from examples used to tune the system. Use test accounts or blocked write actions so a replay cannot change a real order or contact a customer.
Supervised live handling checks new requests while a person reviews the answer and any proposed action before release. Record what the reviewer changes and why. An approved draft shows that the reviewer accepted it; it does not establish that the request was handled without human effort.
Limited autonomous handling tests only the scope approved to run without per-case review. Continue sampling outcomes, tracking exceptions, and keeping a person available for escalation. Approval removal for one intent does not authorize every intent or every action.
Build the test set around the work you expect, including exceptions: ambiguous requests, missing customer information, conflicting policies, failed lookups, repeat contacts, and out-of-scope cases. Check that the AI asks for clarification or hands over where needed, rather than producing a plausible answer at any cost.
Choose sample size for the volume, variation, and consequences involved. A quiet period may miss important failure modes. If you lack historical tickets, use our guide to launching AI support without historical tickets to plan the starting evidence.
Plan approval coverage before live traffic arrives
Human approval is an operational workflow. Someone needs time to do it, a clear queue, and a fallback when they are unavailable.
Name the primary reviewer and backup, staffed hours, and an acceptable review delay. Decide what happens outside those hours: keep requests in the normal human queue, hold drafts until coverage resumes, or restrict the pilot to staffed periods. Do not let a missing approver silently trigger automatic sending.
Give reviewers a rubric: correct answer and customer context, permitted action, appropriate tone, and a usable handover. Capture failure categories so recurring problems can be fixed and retested.
If the approval backlog grows, reduce the incoming scope or add coverage. Removing approval solely to clear the queue avoids the readiness question.
For workflow options, see human override and approval patterns.
Decide what permits expansion and what stops it
Before live testing, agree the evidence needed to expand. Write the thresholds into the pilot brief rather than choosing them after seeing the results.
- Quality: correct answers and actions, policy compliance, and error severity, not just a draft-acceptance rate.
- Customer outcomes: repeat contacts, reopenings, complaints, and satisfaction where measurable.
- Operational readiness: working handovers, manageable review delays, dependable integrations, and maintained knowledge.
- Evidence coverage: enough representative cases across the approved scope, with unresolved failure categories documented.
Set stricter stop conditions for consequential errors. Define who receives an alert, which cases or actions must pause, and how affected customers will be helped. There is no single approval-rate threshold that substitutes for these decisions.
Test the rollback before you need it. The owner should be able to disable autonomous sends and actions, route new and pending cases to people, preserve the conversation history, and prevent duplicate processing. Reverting the mode does not undo a message already sent or a completed action; include remediation in the plan.
Expand only the scope that meets the criteria. Keeping approval for a difficult case type can be a deliberate operating choice, not a failed launch. A calendar deadline is a review point, not proof of readiness.
Make the commercial decision separately
A technically acceptable pilot still needs a comparable business baseline. Record the same channel, case mix, volume, and quality standard before and during the test.
Count total human time, including review, corrections, handovers, and maintenance. Add software and usage costs, integration effort, and setup costs over the chosen evaluation period. Distinguish freed capacity from cash savings that can actually be realized.
Then make two explicit decisions: is this scope ready to operate at the proposed level of autonomy, and is continuing or expanding it commercially worthwhile?
The pilot should end with a documented decision to expand, revise, keep the current mode, or stop. That is more useful than proving the team can switch off an approval button.
