“Alexa, what should organizations know about voice app testing before they start development?” “Alexa STOP, scratch that…it’s probably just easier if I explain.”
Following the success of Google Home, Siri, Echo and others, new “voice-first” applications are flooding the market in what many are calling the first phase of the voice era. As a matter of fact, Canalys predicts that by the end of 2018, U.S. shipments of smart speakers are expected to reach 38.4 million units, and Amazon Alexa alone had more than 30,000 U.S. skills available through Q1’18. However, these voice apps require an entirely new developer skill set than what their teams utilize for desktop and mobile. In fact, voice apps, a powerful new way to connect with and better serve customers, are significantly more difficult to master.
To ensure a 5-star user experience, many organizations are turning to new automated testing methods to speed development and ensure optimal performance. Before embarking on a new voice app, consider these key points.
Just because you have a highly rated mobile app, doesn’t mean it will translate into a voice app.
If you’re a Starbucks or Domino’s Pizza fan, you may recall when they first rolled out their voice apps. Both were plagued with issues and had a 2.5-star rating (compared to the iOS 5-star ratings). With most voice applications, vendors are forced to use the host device’s voice services when creating the application (such as Amazon Alexa), and it often results in having less control over the quality of the application. In addition, the frequency of new platform updates can be aggressive, causing voice applications to break if not constantly updated and tested. Automated testing solutions can help overcome these challenges, ensuring a smoother user experience.
Testing a Voice User Interface (VUI) is different from testing a Graphical User Interface (GUI).
Voice apps obviously use a VUI as the primary interaction between human and machine. Whereas a GUI is designed to limit choices for easy navigation, a VUI offers almost unlimited possibilities, which means the added burden of an almost indefinite number of test cases when developing the app. To help overcome this and streamline the process, testers should consider advanced testing technique such as Pairwise Testing, which helps narrow down testing to a subset of input combinations that maximizes the probability of detecting problems.
Lack of standards across device platforms complicates voice application testing.
The ability to execute a test against any kind of voice app running on any device is key to ongoing performance. While there are some common concepts in building voice apps across platforms, they’re simply not the same. This makes establishing a set of tests that can be executed to verify the application across different platforms critical to success. To help standardize this process, developers need to be sure they plan in advance for automated cross-platform testing, which can be applied to Alexa, Google Home, voice-enabled websites or web apps, virtual assistants (such as SIRI, Cortana, etc.), call center apps, and much more. Streamlining voice app testing processes across platforms not only makes it easier for DevOps teams to manage these apps, but it also presents customers with the same great experience regardless of what device they’re using.
Continuous speech recognition and multi-turn conversations will continue to pose testing challenges.
Although speech recognition technologies are reaching human parity, it’s still a technical challenge to get smooth continuous speech dictation through voice apps, due to the significant time investment required to train the engines. Further complicating this task are multi-turn conversations, which are designed to keep a conversation going as far as possible (and can also be used to deliver different variations of the same answer to make the app seem more human). An automation testing framework needs to be smart enough to deal with multi-turn conversations and speech variations so the test execution can be more robust and meaningful.
Whether you’re a seasoned application developer and tester, new to the game, or just new to voice-based applications, it’s an exciting time to be in the industry. The work happening today will have a huge impact on how people interact with voice technology for years to come. Listed above are just some areas worth considering as you start your voice app dev and testing journey. What other areas should be considered? I’d love to see your comments below.
