Back
Blog / 
AI Agents

Use draft mode to improve AI before it replies to customers on its own

written by:
David Eberle
A hand replacing one line of an AI-written reply on a blue reply card before it is sent.

Published · Updated

Autonomous customer service offers the prospect of resolving requests without a person writing or approving every reply. For a support team, the value depends on the AI doing the work reliably enough that customers and colleagues do not have to repair it afterward.

An incorrect reply can deny a valid return, make a promise the business cannot keep or leave a customer asking again. Removing human review too soon can turn errors in a draft into customer problems and extra work for the team. Keeping review indefinitely has a cost too, especially if people correct the same mistakes every day.

Draft mode provides a supervised path toward greater responsibility. People control what is sent while the team uses recurring corrections to identify causes, test fixes and assess which requests the AI can handle more reliably. The improvement comes from that deliberate work; an edit is evidence to investigate, not something the AI automatically learns from.

Imagine a customer asking to return a jacket bought six weeks ago. The AI says the return window is 30 days. A support team member checks the policy and changes it to 60 because the customer is a member.

Counting that draft as "used" misses the important question: did the AI have the right policy and enough information to apply it?

Five things people change in AI-written replies

Group edits by what changed, then investigate why. These categories describe the edit, not its cause, and a single change can fit several categories.

Facts. A date, amount, policy term, product detail or status was wrong or missing. Check for an outdated source, the wrong passage, missing case details, an invented answer or a mistaken human correction.

Relevance. The draft answered a different question, missed one of several, or ignored the customer's situation. Compare it with the full request before deciding what needs to change.

Length. Text was cut or compressed. The default length may not fit the channel, but check whether the extra words were hiding an irrelevant answer.

Tone. Wording was softened, warmed, formalized or made more direct. Instructions and examples are a useful place to start testing a change.

Next step. A support team member added a question, a handoff, an action in another system or a commitment. This may reveal missing workflow support or a promise the AI should not make.

One of our customers, a consumer-products company, told us its Typewise AI assistants initially asked one troubleshooting question per message. Customers reacted badly, so the team instructed the AI to group the relevant questions into one email.

When reviewing edits, look for changes to the conversation as well as the wording. Repeatedly grouping troubleshooting questions may point to an instruction that needs changing.

Review draft mode one request type at a time

A retailer we spoke with was testing another provider's AI. Its support team reviewed and edited suggested replies, tracking how often they changed them to help decide which requests could be automated. It was still gathering data before judging the results of that automation.

Our recommendation is to start with one request type, such as returns or delivery updates. This makes it easier to see which problems recur and whether a change fixes them.

Take a random sample of edited and unedited drafts. Twenty a week is an illustrative starting point for finding problems, not a readiness threshold. Keep the model, instructions and policy fixed during each review period; start a separate count when a version changes.

Turn recurring edits into a tested fix

Use the review to propose a specific change, have the person responsible for the AI approve it, and check the next version.

  1. Compare the draft and sent reply. Record what changed, allowing several categories, "unclear" and "no change." A label alone does not tell you whether either reply was correct.
  2. Check the request, source and context. Read the customer's original request and the relevant policy or case details. Check unedited drafts too: an incorrect reply can be sent untouched.
  3. Identify the recurring cause. For the jacket example, was the membership policy missing, did the AI select the wrong passage, or was the customer's membership unknown? Those problems need different fixes; mapping your support sources helps separate them.
  4. Propose a change for administrator review. The fix might be a clearer instruction, a corrected source or a workflow that checks missing information. The administrator should review the evidence and approve the change before it is applied.
  5. Retest the changed version. Check whether the original problem is fixed, then review fresh requests of the same type for mistakes or missing actions. Keep results separate from the previous version so you can see what changed.

A language model can help suggest edit categories, but a person still needs to check the underlying facts and review proposed fixes. Use only an environment approved for customer data, with identifying details removed where your policy requires it. Check suggested labels against human review and recheck when the model, instructions or request mix changes.

For each request type, report edited drafts divided by all sampled drafts, with the sample size alongside. Separately record incorrect drafts that were sent unchanged. Combining every request type into one rate can hide a recurring problem in a smaller group.

When to move beyond draft mode

Mostly cosmetic edits are a reason to evaluate more responsibility for that request type, not proof that the AI is ready. Check whether replies actually resolve requests and whether the AI has the permissions and safeguards for the work.

A reply that needs no editing can still leave the customer needing help; see what counts as a resolution.

Start with one request type, choose the recurring problem with the greatest impact, and take one proposed fix through administrator review and retesting.