Inside The AI: how we build our autocorrect, emoji and text prediction engine

Although it’s Typewise’s unique hexagonal keyboard that tends to catch people’s eye at first, it’s our advanced autocorrect and text prediction AI under the hood that provides the power behind our product. George Roberts, Senior AI Engineer at Typewise, reveals how we build our AI to beat the competition.

‍

What are the problems that Typewise’s autocorrect & text prediction AI is trying to solve?

The days of extremely inaccurate autocorrect fails may be behind us, but despite all the advances in smartphone keyboard autocorrect tech, even today you’ll find plenty of discussions on Reddit and such highlighting how many problems still exist with autocorrect. No one, even the big tech giants, have yet created an autocorrect and text prediction system that works perfectly.

One of the main problems that we at Typewise are trying to fix, is changing correctly typed words to incorrect words. Over-aggressive autocorrect is still an issue on some of the big tech platforms (Google, Apple, Microsoft); you'll see for example, somebody wanting to type ‘heaven and hell’ and their phone keyboard will continually correct ‘hell’ to ‘heal’.

These sorts of errors are extremely irritating and that’s one of the reasons why we built our autocorrect undo button into our system, so you can revert a bad autocorrection in one tap, and in doing so train our AI so that it doesn’t happen again.

There’s also an oddity in the autocorrect world, in that you’d expect Google’s Gboard or Apple Keyboard to get better and better each year, but lots of people will tell you otherwise; for example they say that Gboard autocorrect five years ago was the best autocorrect and then it just got worse and worse.

With something like autocorrect, Big Tech’s attitude can be; “don't break it”, which can mean that new ideas don’t get implemented. If you're a big company and you change things like this, you're going to upset far more users than if you're a small, agile startup like Typewise and you can just rapidly iterate and improve.

How does Typewise’s autocorrect and text prediction AI differ to competitors?

Comparing the nuts and bolts of Typewise’s AI to the downloadable keyboards from the big tech players – Google, Apple and Microsoft – is a challenge because they don't publicly share how they build their autocorrect AI.

So, to measure how Typewise performs against competitors, we have created a set of benchmarks which allows us to compare the quality of our autocorrection and text prediction AI, to them.

Part of the way we optimised our keyboard app was to use hexagonal keys rather than regular square or rectangular keys. This innovation means that Typewise users make fewer typos, because they're less likely to hit the wrong key.

So, we’ve improved upon the smartphone keyboard in two major ways; firstly with the hexagon layout of the keyboard itself, but also the AI that powers it. When we benchmark the autocorrection, we compare the algorithm to our QWERTY layout, rather than our hex layout (and we come out on top). When we compare the big tech's QWERTY layouts to our hex layout, we perform even better.

Does the text prediction AI take into account only the last word typed, or the entire sentence?

Currently our AI considers the previous word typed, but we’re planning an update to make it analyse the entire sentence, giving it more context.

The goal is to improve the AI’s ability to accurately predict the next word or words. So if somebody pastes a huge chunk of text into the autocorrection, Typewise should be able to figure out the most likely words to follow.

This type of contextual mapping allows the AI to probabilistically calculate what they might have typed. But equally perhaps they wanted to type something that was not very likely, perhaps a sentence we've never seen them type that in that context. So our aim is to balance these factors and offer the world’s best autocorrect AI.

What else does the AI use to predict the next word typed or suggest an autocorrection?

The next thing the AI examines is where the user actually hit the keys. It looks at the surrounding letters and tries to work out whether the user meant to hit the letter they did, or whether they were actually aiming to hit an adjacent key, depending on how close they were to the edge of the key.

To learn, the AI must be trained; what does Typewise use to train it’s AI with natural language?

There are many datasets you could use to train a text prediction or autocorrection engine. We used large sets of publically available material to train our autocorrection because we need a clean (i.e. – with no typos) dataset for that.

The data sets we use have been transcribed into numerous languages, and are based on ‘spoken word’ rather than written word, which is more natural for the way people text on their phones.

For next word prediction, the dataset doesn't need to be clean because you want to have some natural typos in there. So we take a big subset of this material and apply ‘noise’ to it. Noise is intentionally inserting errors such as deleting or substituting characters.

This gives us a clean set of data and a noisy set of data, which we can then use as a comparison and this enables us to train the AI.

How are neural networks involved in keyboard apps and autocorrection?

In simple terms, a neural network takes input numbers and applies a number of different mathematical operations to them in order to get an output.

In the context of text prediction and autocorrection it means that you can take a number of inputs – like the word they have entered, previous words, previous sentences, and where keys are hit, and then you generate outputs, which would be a suggested word or next word prediction.

And all this takes place on the phone itself not in the cloud, ensuring Typewise is a 100% privacy keyboard and that users’ typing data never leaves their phone.

How does Typewise tackle the ongoing evolution of language?

New words and phrases are constantly appearing, and the meanings of existing words are constantly changing. So whilst we learn new words as the user types, we also will update our dictionary with the latest terms that have appeared. For example, three years ago, most people had never written a text with the word ‘COVID’ in it, and if you did, autocorrect would probably change it to ‘Vivid’. Since the pandemic, few people would not have used the word COVID in a text.

So, it’s very interesting to watch how quickly language evolves, and which new words enter our everyday texting language. We’re aiming to update our word lists with datasets from sources like Twitter, where new words quickly spread. This would be on top of our functionality that already learns from each individual user.

‍

George Roberts is Typewise's Senior AI Engineer, an expert in Machine Learning, holds a PhD in Quantum Physics from The University of Cambridge, and is a massive Tolkien fan.

‍