The open test kit

Test any AI support vendor on 50 of your own tickets.

Every vendor demo looks good. This kit gives you the same test for all of them, us included: how to pick 50 tickets by the minutes they cost, a sheet to score the outcomes on, and a pass line you set before you look.

See the method
Free, no form, no emailWorks for any vendorYours to change and share
Part one

Pick 50 tickets by minutes, not by volume.

A sample taken by ticket count is mostly simple questions, and every vendor answers those. The minutes are where your team’s time goes, so the minutes decide what is in the test.

  1. Export last quarter’s tickets

    With the topic and the handling time of each one. If handling time is not recorded, time ten tickets per topic by hand. A rough number beats none.

  2. Add up the minutes per topic

    Tickets multiplied by the average minutes each one takes. Rank the topics by that total, not by how often they come in.

  3. Give 40 places out by share of minutes

    A topic that costs 30% of the minutes gets 12 of the 40 places. Inside a topic, take every tenth ticket or so. Do not pick them by hand.

  4. Add 10 awkward ones on purpose

    Missing information, two policies that conflict, a request that is out of scope, a customer writing in for the third time. These are where an AI goes wrong.

  5. Write the expected outcome for each

    Before any vendor sees a ticket: whether the AI should complete it, do part of it and then hand over, or hand it over, and what the right answer or action is. Then remove personal data from all 50.

How the 50 places are given out
TopicShare of ticketsShare of minutesPlaces
Simple questions65%20%8
Order and account changes25%35%14
Complaints and exceptions10%45%18
Awkward cases, picked on purpose10
Tickets in the test50
An illustration, not customer data. By ticket count the simple questions would fill 26 of the 40 places. By minutes they get 8.
Part two

Score every answer on the same sheet.

One row per ticket, filled in by one named person. Everything up to the expected outcome is written before the test. The rest is what the vendor’s AI did with the ticket.

open-test-kit-scoring-sheet.csvWritten before the testScored after
#TopicMinutes todayExpected outcomeRight answer or actionAI outcomeWhat the AI didAnswerActionHandoverSeverityPassed
1Delivery status4completedGives the current status from the order recordcompletedFound the order and gave the right statusYesn.a.YesNoneYes
2Refund for a late parcel12completedRefunds the delivery fee, as the policy sayscompletedExplained the policy correctly, then refunded the whole orderYesNoYesSevereNo
3Legal threat (awkward)15handed overHands over to a person without replyinghanded overHanded over, with a summary for the colleaguen.a.n.a.YesNoneYes
Scroll sideways to see the scored columns. Example rows to show how the sheet is filled in, not results. The download has these columns, one to mark the awkward cases, a notes column and 50 empty rows.

The columns that need a word

Minutes today
What the ticket costs a person now. It decides which tickets are in the test. It is not a score.
Expected outcome
Completed by the AI, partly done then handed over, or handed over. Written before the test and not changed after.
AI outcome
What actually happened: completed, partly done, or handed over.
Handover
Yes when the AI handed over where it should, and did not where it should not.
Passed
Yes when the AI outcome matches the expected one, and the answer and the action are correct, or do not apply.

Error severity

  • NoneNothing wrong.
  • MinorIncomplete or badly worded. The customer would have to ask again, but nothing is harmed.
  • SevereWrong information a customer would act on, a wrong action such as a refund or a cancellation, data shown to the wrong person, or a promise your policy does not allow.

A plain CSV: import it into Excel, Google Sheets or Numbers.

Part three

What a pass looks like.

A pass is not a feeling after a demo. It is three readings of the sheet, held against a line you drew before the test started.

Reading 1

Correct completions, x of y

On the tickets you expected the AI to complete, count how many it completed correctly. Report it as x of y, against the target you wrote down before the test.

Reading 2

Handovers and errors, counted apart

Required handovers that happened, handovers that were not needed, and severe errors, each on its own line. A high completion count does not buy back a wrong refund.

Reading 3

The ten awkward cases, on their own

Report them separately from the 40. They show what the system does when a ticket is not clean, which is where an AI goes wrong.

What 50 tickets do not show
Time saved

A correct handover is a pass, but a person still spends the minutes of that ticket. So the kit does not add up minutes and does not claim a time saving. Net time saved, your baseline minus review, corrections and the work that remains, is what a pilot measures on live traffic.

How a pilot measures it →
Two gates we suggest. The target is yours.

Write the line down before you look.

  • 0severe errors
  • Alltickets that should go to a person were handed over
  • x of ycorrect completions: you set the target for your scope before the test

The two gates are our suggestion for a first test, not an industry standard. There is no default pass rate: a team with regulated topics will want a different target from a team testing simple questions, so write yours down before any answer is scored.

Part four

Rules for a fair test.

The kit only tells you something when every vendor sits the same exam, and nobody has seen the questions.

  1. The same 50 tickets for every vendor

    Same tickets, same order, same wording. Change the sample and you are comparing samples, not vendors.

  2. Expected outcomes are written first

    Before you see a single AI answer. An answer that sounds good can pull the expected outcome towards it.

  3. The test tickets stay with you

    Give every vendor your policies and knowledge sources, not the 50 tickets. A system tuned on the test has passed the test, not shown it can do the work.

  4. One named person scores

    The same person for every vendor, ideally someone who handles this work today. A second person reads every severe error.

  5. Same material, same time

    Every vendor gets the same documents and the same number of days to set up. Write both down.

  6. No real customer, no real system

    Run it in a test environment. Nothing in this kit needs access to your production systems.

What the kit does not tell you

Fifty tickets are a first filter, not a go live decision.

The kit shows whether a vendor’s answers hold up on your own work. It does not show time saved, how the system behaves on live traffic, whether your integrations work, or what it costs at your volume. That is what a pilot is for.

How a pilot works →
Run it on Typewise

Hold us to the same sheet.

Small teams can start a free trial and score what Typewise does with their tickets on this sheet. For larger teams, a test like this is the second step of a pilot, after the ticket analysis.

Questions about the kit.

Give every vendor the same 50 of your own past tickets, picked by the minutes they cost your team and not by how often they come in. Write the expected outcome for each ticket before the test, score every answer on one sheet, and compare the result with a pass line you set in advance.

It is a number one person can score by hand, and it leaves room for the awkward cases. If your topics are very varied, use more and keep the method: places by share of minutes, plus awkward cases picked on purpose.

Because a count fills the sample with easy tickets. A simple question takes a person seconds, a complaint can take fifteen minutes, so the minutes decide which tickets are in the test. The score itself counts outcomes, not minutes: fifty tickets cannot show time saved, a pilot does.

No. Share your policies and knowledge sources, the same for every vendor, and keep the 50 tickets back. If a vendor tunes its system on the test tickets, the result says very little about the next ticket.

Wrong information a customer would act on, a wrong action such as a refund or a cancellation, personal data shown to the wrong person, or a promise your policy does not allow. Severe errors are counted separately and are not averaged away by a high score.

Yes, that is the idea. The kit names no vendor and makes no exception for us. If you want to go further than 50 tickets, see how a pilot works.

Yes. There is no form and no email address to leave. Download the sheet, change it to fit your team and pass it on.

Ready to go past 50 tickets?

Book a call and we plan a pilot around your past tickets: where the minutes are, which use cases to test first and what the pilot has to prove.