Correct completions, x of y
On the tickets you expected the AI to complete, count how many it completed correctly. Report it as x of y, against the target you wrote down before the test.
Every vendor demo looks good. This kit gives you the same test for all of them, us included: how to pick 50 tickets by the minutes they cost, a sheet to score the outcomes on, and a pass line you set before you look.
A sample taken by ticket count is mostly simple questions, and every vendor answers those. The minutes are where your team’s time goes, so the minutes decide what is in the test.
With the topic and the handling time of each one. If handling time is not recorded, time ten tickets per topic by hand. A rough number beats none.
Tickets multiplied by the average minutes each one takes. Rank the topics by that total, not by how often they come in.
A topic that costs 30% of the minutes gets 12 of the 40 places. Inside a topic, take every tenth ticket or so. Do not pick them by hand.
Missing information, two policies that conflict, a request that is out of scope, a customer writing in for the third time. These are where an AI goes wrong.
Before any vendor sees a ticket: whether the AI should complete it, do part of it and then hand over, or hand it over, and what the right answer or action is. Then remove personal data from all 50.
| Topic | Share of tickets | Share of minutes | Places |
|---|---|---|---|
| Simple questions | 65% | 20% | 8 |
| Order and account changes | 25% | 35% | 14 |
| Complaints and exceptions | 10% | 45% | 18 |
| Awkward cases, picked on purpose | 10 | ||
| Tickets in the test | 50 | ||
One row per ticket, filled in by one named person. Everything up to the expected outcome is written before the test. The rest is what the vendor’s AI did with the ticket.
| # | Topic | Minutes today | Expected outcome | Right answer or action | AI outcome | What the AI did | Answer | Action | Handover | Severity | Passed |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Delivery status | 4 | completed | Gives the current status from the order record | completed | Found the order and gave the right status | Yes | n.a. | Yes | None | Yes |
| 2 | Refund for a late parcel | 12 | completed | Refunds the delivery fee, as the policy says | completed | Explained the policy correctly, then refunded the whole order | Yes | No | Yes | Severe | No |
| 3 | Legal threat (awkward) | 15 | handed over | Hands over to a person without replying | handed over | Handed over, with a summary for the colleague | n.a. | n.a. | Yes | None | Yes |
A plain CSV: import it into Excel, Google Sheets or Numbers.
A pass is not a feeling after a demo. It is three readings of the sheet, held against a line you drew before the test started.
On the tickets you expected the AI to complete, count how many it completed correctly. Report it as x of y, against the target you wrote down before the test.
Required handovers that happened, handovers that were not needed, and severe errors, each on its own line. A high completion count does not buy back a wrong refund.
Report them separately from the 40. They show what the system does when a ticket is not clean, which is where an AI goes wrong.
A correct handover is a pass, but a person still spends the minutes of that ticket. So the kit does not add up minutes and does not claim a time saving. Net time saved, your baseline minus review, corrections and the work that remains, is what a pilot measures on live traffic.
How a pilot measures it →The two gates are our suggestion for a first test, not an industry standard. There is no default pass rate: a team with regulated topics will want a different target from a team testing simple questions, so write yours down before any answer is scored.
The kit only tells you something when every vendor sits the same exam, and nobody has seen the questions.
Same tickets, same order, same wording. Change the sample and you are comparing samples, not vendors.
Before you see a single AI answer. An answer that sounds good can pull the expected outcome towards it.
Give every vendor your policies and knowledge sources, not the 50 tickets. A system tuned on the test has passed the test, not shown it can do the work.
The same person for every vendor, ideally someone who handles this work today. A second person reads every severe error.
Every vendor gets the same documents and the same number of days to set up. Write both down.
Run it in a test environment. Nothing in this kit needs access to your production systems.
The kit shows whether a vendor’s answers hold up on your own work. It does not show time saved, how the system behaves on live traffic, whether your integrations work, or what it costs at your volume. That is what a pilot is for.
How a pilot works →Small teams can start a free trial and score what Typewise does with their tickets on this sheet. For larger teams, a test like this is the second step of a pilot, after the ticket analysis.
Give every vendor the same 50 of your own past tickets, picked by the minutes they cost your team and not by how often they come in. Write the expected outcome for each ticket before the test, score every answer on one sheet, and compare the result with a pass line you set in advance.
It is a number one person can score by hand, and it leaves room for the awkward cases. If your topics are very varied, use more and keep the method: places by share of minutes, plus awkward cases picked on purpose.
Because a count fills the sample with easy tickets. A simple question takes a person seconds, a complaint can take fifteen minutes, so the minutes decide which tickets are in the test. The score itself counts outcomes, not minutes: fifty tickets cannot show time saved, a pilot does.
No. Share your policies and knowledge sources, the same for every vendor, and keep the 50 tickets back. If a vendor tunes its system on the test tickets, the result says very little about the next ticket.
Wrong information a customer would act on, a wrong action such as a refund or a cancellation, personal data shown to the wrong person, or a promise your policy does not allow. Severe errors are counted separately and are not averaged away by a high score.
Yes, that is the idea. The kit names no vendor and makes no exception for us. If you want to go further than 50 tickets, see how a pilot works.
Yes. There is no form and no email address to leave. Download the sheet, change it to fit your team and pass it on.
Book a call and we plan a pilot around your past tickets: where the minutes are, which use cases to test first and what the pilot has to prove.