In the mobile testing world, there has been a raging debate pitting the use of virtual, cloud-based testing against testing on physical devices. As with any good debate there are valid arguments on both sides, but the answer really depends on the situation.

Cloud-based virtual testing has become not only faster and more efficient but also more cost effective as we gain new tools and simulation advancements. On the flipside there’s still no substitute for testing on physical devices when it comes to getting the most accurate user experience of a particular device.

Let’s take a closer look at the main advantages of each approach as well as the factors you should keep in mind when choosing an approach.

Virtual testing

Virtualisation has come a long way, and using emulators can speed up your testing significantly, but not all emulators cover all devices. More recently, cloud-based device farms have gained in popularity, but not all device farms have all the devices you’ll want to test against.

The main advantages of virtual mobile testing include:

• Forward-thinking – Virtual testing makes it easier to stay up-to-date in the fast-changing mobile environment of new devices, operating systems, and browsers.
   
• Flexibility – Virtual testing can be done on dozens of different devices with different operating system (OS) configurations concurrently without having to switch between devices.
   
• Cost – It is significantly less expensive to test virtually than on physical devices, both in terms of manual labour cost and time-per-test cost. Maintaining physical devices – upgrading OSes and fixing device faults – is also more expensive.
   

Physical testing

Physical testing will give you the most accurate understanding and visibility of the quality of your mobile application on a given device. You will be able to immediately see and identify how the app responds, reacts, and works on the device screen without having to wait for the test results.

The main advantages of physical mobile testing include:

• Accuracy – Testing on an actual device is the only way to see how an app will react in real life. Intangible aspects such as battery life, Wi-Fi strength and device-related issues are difficult to reproduce virtually.
   
• Consistency – Some features like voice input aren’t supported for all devices by all emulators and would have to be tested on physical devices. Moreover, only physical testing can give you consistent results on actual device performance, for example .
   
• User feedback – It’s difficult to carry out User Acceptance Testing on virtual devices , so physical devices will always be needed for this last but crucially important step in the testing process.
   

How to choose the right mobile testing approach

You don’t have to choose between virtual and physical mobile testing. In today’s connected world, a hybrid solution that accounts for both types of testing is even more optimal, especially as we adjust to a world where remote, cloud-based testing is fast becoming part of the ‘new normal’.

Before you take on your own testing project, ask yourself some important questions:

• What types of devices do I need to test? For example, are they mostly iOS-based or Android-based, do I need to test on Symbian devices, and which OS versions do I need to consider for my application?
   
• How many devices do I need to test on? For example, do I only need to test on recent iPhones and Samsung Android phones, or do I need to cast a wider net in terms of legacy models and less popular Android devices?
   

Once you’ve answered these questions, you’ll have a better idea how many devices and test configurations will be required for testing. Use this as a guide to determine how practical it would be to test physically or, in the case of more widespread testing, what type of virtual platforms will be needed. For example, would a basic emulator for a few devices suffice, or will you need access to a well-stocked virtual device farm?

Finally, remember that you don’t have to walk this road alone. Working closely with a qualified and experienced testing partner, who has access to all the different options at different price points, is the best way forward for the majority of test applications.

About Morgan

Morgan is professional Principal Consultant with more than 17 years of experience in the testing industry. He is trained in both automation and performance testing and passionate about DevOps. Morgan has spent some time as a DevOps technical coach and is fluent in most of the open-source and preputial testing tools. He has also head non-functional teams and created strategies for a host of industries.