Reviewing the data surveyed over the past four years by the Capgemini group in their series of reports – World Quality Report – we can see a startling gap in the ability to actualize Test Automation and techniques which are required to see value coming out of automated tests. The figures paints a picture of an industry (Software Development) which has an idea about the outcomes it desires, but has little to no idea about how to get from point A to point B. When it comes to automation there is a visible backtracking trend which suggests that as the industry learns or attempts Automated Testing it finds that it is more complicated than it first appears.
The period of 2015-2016 shows this trend in the answers provided to a survey which asks about challenges regarding agile development. In 2015 only 23% of respondents claimed that they experienced an inability to apply appropriate test automation, whilst in 2016 this figure went up to 41%. 1 I believe that this is not related more companies adopting automation, but rather companies discovering that automation is not as simple as it was sold to them.
2
1 Mamnani, Deepika. “Agile and DevOps.” World Quality Report 2016-2017 8 (n.d.): 33. Accessed August 23, 2019.
2 Mamnani, Deepika. “Agile and DevOps.” World Quality Report 2016-2017 8 (n.d.): 33. Accessed August 23, 2019.
This is typified by responses between 2016-2017 which focus on the tools being used in automation. In 2016 41% of respondents claimed to not have the right tools for automation testing, and this figure remained high in 2017 at 39%.3
4
These figures, however, do not represent my experiences within Test Automation since 2016. Inspired Testing (formerly part of DVT) has been working with a variety of tools – based on customer need but focused on the Open Source toolset to save clients money – and in my experience there has been little that we could not automate. Returning to the figures we can see that in 2017 42% of respondents answered that their automation solution did not support mobile testing. From an Open Source perspective this does not make sense. Tools like Appium have been around since as early as 2011 and have gone from strength to strength. The problem – I believe – comes back to the issue with the answer “We don’t have the right automation tools”.
There is a long running trend in Automation where the enterprise Automation tool giants claim that their tools are the “easiest” to use, boasting features like record and playback which allow anyone to automate. This is obviously attractive, as the idea of having your manual test effort reduced without effort is just a win, but the figures tell us this is clearly not the case. If the tools were the solution we wouldn’t be seeing failure to automate, or complaints about having the right tools.
Another figure which helps to explain the difficulty curve of automation is that between 2016-2017 we see a growth of respondents (25% in 2016, 41% in 2017) coming to terms with not having the ‘right’
3 Murray, Will, Prabhakaran Karuppiah, and Chrisopher Stancombe. “Test Automation: On the Way to Smart, Intelligent, and Cognitive QA.” World Quality Report 2017-2018 9 (n.d.): 28. Accessed August 23, 2019.
4 Murray, Will, Prabhakaran Karuppiah, and Chrisopher Stancombe. “Test Automation: On the Way to Smart, Intelligent, and Cognitive QA.” World Quality Report 2017-2018 9 (n.d.): 28. Accessed August 23, 2019.
automation testing process or method. 5 This could likely be a result of buying in to a tool, and not to Automation as a concept. The idea that tools were the primary focus rather than the discipline is highlighted over the period of 2016-2018, where we see the challenge of ‘Lack of skilled and experienced test automation resources rise from 34% in 2016, to 36% in 2017, to 46% in 2018. 6
7
Again, I do not believe that there was a change which meant Automation resources lost their abilities, but rather respondents started waking up to the challenges which Automation presents that cannot be solved by buying an out of the box tool.
So now what?
5 Murray, Will, Prabhakaran Karuppiah, and Chrisopher Stancombe. “Test Automation: On the Way to Smart, Intelligent, and Cognitive QA.” World Quality Report 2017-2018 9 (n.d.): 28. Accessed August 23, 2019.
6 Throvagunta, Sridhar, Brian Olsen, and Antoine Aymer. “Test Automation: The Single-Biggest Enabler of Maturity in QA and Testing.” World Quality Report 2018-2019 10 (n.d.): 29. Accessed August 23, 2019.
7 Throvagunta, Sridhar, Brian Olsen, and Antoine Aymer. “Test Automation: The Single-Biggest Enabler of Maturity in QA and Testing.” World Quality Report 2018-2019 10 (n.d.): 29. Accessed August 23, 2019.
I firmly believe that the solution to these problems is not to purchase a tool, rather there is a need to engage with the right people or services to solve these problems.
How can Inspired Testing solve these problems?
I will start with what I believe is the biggest challenge to an Automation project – experience. An in-sourced automation tester will build experience, but its limited to the Applications Under Test – which in a closed company environment will be limited. Inspired Testing – due to its outsourced nature – has scripters with experience in up to 6 different client environments within their first two years. This means that the Automation testers have knowledge and learnings from all kinds of systems and environments which can be brought to any of clients Automation projects. The Inspired Testing Global Testing Center also recruits young programmers and future BA’s who make Automation their specialty – rather than going through years of general testing to be given the chance at Automation, Inspired Testing cultivates Automation testers from grass roots and allows them to focus and hone their craft. This allows Inspired Testing to provide testers with varying levels of experience based on the need of the client, the project, and the complexity of the Application Under Test.
This article has focused a lot of negative energy towards tooling, but it does play a huge role in Automation. The approach that Inspired Testing takes towards Automation tools is that it uses the tool that fits the client need – not the other way around. To this end Inspired Testing is proudly tool agnostic, but we do lean towards Open Source, as often a primary client need is to reduce costs! Another tool related complaint is that there are too many different automation tools. From experience the Inspired Testing Global Test Center can recommend the tool that will best cater for all the needs of the project, rather than having multiple tools.
One of the benefits of using a specialized outsourcing partner for Automation is that Inspired Testing can provided guidance on how, and what to automate. This alleviates the issues of not having the right testing process/method because Inspired Testing can bring lessons learned from other projects and how similar industries have successfully implemented automation. Because of the focus on Automated Testing the Inspired Testing Global Test Center can boast a level of expertise which allows us to help guide clients on their journey through Automation towards DevOps.
In Conclusion
According to the world Quality Report survey regarding the proportion of automated tests there is plenty of work that requires skilled and experienced Automation Testers. According to respondents only 16% of functional test cases are executed with test automation tools, and this figure hasn’t changed since 2017. 8
8 Throvagunta, Sridhar, Brian Olsen, and Antoine Aymer. “Test Automation: The Single-Biggest Enabler of Maturity in QA and Testing.” World Quality Report 2018-2019 10 (n.d.): 27. Accessed August 27, 2019.
9
If your test team is also in this situation, facing similar challenges to those outlined in this article and you would like to do something about it reach out to Inspired Testing for a solution to your Automation needs.
9 Throvagunta, Sridhar, Brian Olsen, and Antoine Aymer. “Test Automation: The Single-Biggest Enabler of Maturity in QA and Testing.” World Quality Report 2018-2019 10 (n.d.): 27. Accessed August 27, 2019.
