My recent work with one of South Africa’s largest commercial banks highlighted a common issue many banks have when it comes to testing their software applications on mobile devices. That is, how to automate regression testing when you have hundreds of different devices and device permutations to test.
While large banks can afford to buy (and test their software on) multiple physical devices, none that I know of have the manpower or budget to cover all the devices that need testing. Take the latest iPhone, for example. You can test your apps on the phone running the most recent version of iOS, but you also need to test it running the previous version, and the version before that.
The challenge gets even more complex with legacy devices running even older versions of iOS, and multiplied exponentially when you account for the ocean of Android devices and their multiple hybrid OS versions that are still very much in circulation.
Previously, many banks resorted to testing a small subset of available devices, and generally did so manually, which lead to significant risk that applications would glitch or fail on untested devices/OS combinations when new features were added. Moreover, few had the means to automate their regression testing, so even partial testing was time consuming and costly.
But there is light at the end of this tunnel. I’ve been fortunate to work with a bank that prides itself on bleeding-edge technology, and as such, was open to finding a creative solution to the very real issue I described above. The solution: creating a Cloud-based mobile device farm and finding and configuring software that would not only automatically test every device in the farm, but also integrate seamlessly with bank’s tightly controlled and regulated software testing and control framework.
The first part was easy thanks to a combination of the bank’s willingness to invest in the physical devices we needed to make the solution work, and Inspired Testing’s own device farm that we’ve built as part of our comprehensive testing service for clients. It’s one thing having physical devices, and another entirely to have all the devices we need available remotely so we can run our tests simultaneously, without human intervention.
The second part was a trial-and-error proof of concept using two well-known testing software platforms: Headspin and Perfecto. Both allowed us to do overnight automated regression testing on mobile devices, taking into account the various parameters and settings that typically make automated testing on mobile devices so difficult.
Ultimately the bank chose Headspin, not necessarily because Perfecto is notoriously costly, but because of Headspin’s integrated machine learning features that generate detailed reports on software issues, bottlenecks, load times and real world statistics. This gives us – and the bank’s development and management teams – better insight on what needs fixing before it becomes an issue, and also guides the future development of the software.
Using Headspin, we could run our automated regression scripts on the Cloud-based mobile device farm, connected via Appium, schedule new tests to run as and when new features were released, with the entire process integrated into the bank’s continuous delivery pipeline.
Not only is this a first for the bank, I believe it’s the first time that automated and continual regression testing on an entire farm of Cloud-based mobile devices has been successfully achieved in South Africa.
This adds tremendous value, not only for the bank but also its users. The bank saves hundreds of man hours of manual regression testing when those very valuable resources can instead be put to work on testing specific devices or features that aren’t part of the regression testing remit. Users are then assured that applications on their devices have been fully tested prior to release, which keeps quality high and troubleshooting to a minimum.
Now that we’ve successfully demonstrated the value of the automated testing solution on a non-core application, plans are in play to expand the concept across all mobile applications for the entire bank, including the bank’s major frontline, user-facing applications. The benefit of freeing up 50 or 60 test squads in the bank that currently spend at least eight hours a day running scripts is immeasurable, and will have a significant impact on the performance and quality of delivery going forward.