Table of Contents
Today, we are officially open-sourcing our mobile keyboard testing framework we call Kebbie.
While mobile virtual keyboards may appear straightforward on the surface, they are packed with cutting-edge technologies under the hood that vastly enhance user typing efficiency. For example: autocorrection, auto-completion, next-word prediction, and swipe gesture recognition are other NLP features that users typing on mobile use everyday.
At Fleksy, innovation is one of our core focus. We strive to constantly improve the quality of our virtual keyboard technologies. But this drive for innovation shouldn’t come at the price of regressions. Recognizing the need for rigorous testing in order to keep the highest level of product quality without hindering innovation, we searched for a standardized methodology to test and benchmark the various NLP features of our keyboard.
With no less than 82 languages to cover, fast iteration cycles with frequent releases and millions of users to cover, an efficient and automated testing framework was necessary.
We surveyed existing tools for testing the NLP features of a mobile keyboard, but found them lacking in features as well as not comprehensive enough for our needs.
While libraries for testing spellchecking exist, as well as datasets with common typos errors (Twitter Typo corpus, Github typo corpus), they often fall short with critical limitations such as:
- Usually cover only spellchecking, other tasks such as next-word predictions or auto-completions are not supported
- No focus on mobile platforms, which comes with its own, unique challenges (fat finger syndrome, etc…)
- Limited to specific types of typos
This is why we wrote a Fleksy framework with the following features :
- Comprehensive coverage of the four most important NLP features of a virtual keyboard
- Auto-correction
- Auto-completion
- Next-word prediction
- Swipe gesture recognition
- High performance, by leveraging multiprocessing to minimize testing time
- Customizability, enabling the use of any text dataset, thus supporting multiple languages easily
- Mobile platform as first-class citizen : entirely centered around mobile virtual keyboards
Comparing competitors
This framework covers our testing needs very well for internal usage: we can track how the score evolves, for each language and for each task. It turns out to be extremely useful to find out regressions between versions. But even at a smaller scale, this tool allows testing individual PRs and better understand the impact of each changes we bring to the codebase.
However, it lacked the ability to compare results with external competitors such as GBoard or Apple keyboard, which do not offer accessible API. Still, benchmarking against competitors is crucial for assessing where we stand and identifying areas for improvement. This is also important for our clients : it ensures Fleksy is competitive with the leading keyboards available in the market.
To address this issue, we extended our original testing framework to interact with any keyboard installed on a simulator.
The framework interacts directly with the keyboard in the simulator, and mimics a real user typing : sentences are typed by tapping each character one by one. Occasionally a typo is introduced, and we observe and analyze the behaviour of the simulated keyboard :
- Which words are auto-corrected
- What are the suggestions for auto-completions and next-word predictions
For now, swipe gesture recognition isn’t supported for simulated keyboards
After extracting these predictions, our testing framework will compute various metrics for the simulated keyboard.
Although this approach is more time-intensive (since we need to interact with a simulator and type sentences character by character), it enables direct comparison with other keyboards.
And because our testing framework provides detailed metrics, it allows us to analyze the strengths and weaknesses of our competitors compared to our own keyboard (for example, our keyboard may have lower recall for a specific type of typo, but the overall precision may be higher, etc…)
Goals
In the previous section we saw what motivated us to create our own testing framework for mobile keyboards, and how we expanded this framework to compare with other keyboards.
Recently, we’ve decided to give back to the community by open-sourcing it and make this framework available to anyone.
In the following sections, we’ll expand a bit on the reasons behind this choice, and our hopes for the future.
Transparent results
Our decision to open-source this framework stems from a commitment to transparency.
The trust our clients put in Fleksy’s work is our most important asset. Transparency not only fosters trust with our clients but also promotes reproducibility and accountability.
By making our framework open-source, we empower everyone to verify and understand our testing methodologies, ensuring that our shared results are credible and reliable.
Foster healthy competition
One of the reason we had to write our own framework was the lack of existing tools. By open-sourcing our framework to the public, we aim to save time for others, but also and most importantly cultivate unity in testing practices.
By establishing a common benchmark, we can collectively improve upon it and achieve new standards for mobile virtual keyboard testing. Well-established standard testing framework are enabling better comparisons, and improve our understanding of new approaches and technology advancements.
Build better together
Beyond merely sharing the source code, we invite contributions from the community to enhance and refine our framework collaboratively. External perspectives and insights are invaluable in crafting better tools.
It is our hope to foster a robust community, centered around mobile keyboard and their improvements.
Want to contribute?
Kebbie is already available on Github for exploration and contribution here: https://github.com/FleksySDK/kebbie
Everything you need to get started is provided in the documentation here: https://fleksysdk.github.io/kebbie
Should you encounter any issues or have suggestions for improvement, please feel free to open a GitHub issue or join the discussion forum.
We can’t wait for your contributions in shaping the future of mobile keyboard testing and development.
Let’s build better keyboards, together!
