Episode 1

Expert duo Jehan Coetzee, Head of Consulting and Architecture, and Dawie Greyling, Principal Consultant at Inspired testing discuss how to choose the right test automation tools, sharing valuable insights and tips along the way.

Transcript

JC: Good morning, Jéhan Coetzee, head of Consulting and Inspired Testing and I'm here this morning with David Greyling, one of our principal consultants. David, something we’re faced with every day, choosing the right tool for the job, right in your consultancy role I'm sure that's a problem that you have to solve daily. How do you approach that? What are your thoughts, initial thoughts when it comes to that?

DG: So the funniest thing is with a lot of organizations that I talk to around automation and implementing of automation none, or not none but most of them actually don't know the value. So the goal that we want to achieve with the tool that we want to actually implement and I think that is the most important thing to look at. If you go to Google you usually see ‘is this automation tool a fit for my organization’ or ‘what is the initial cost for the implementation, what's the budget, do we have the right skills?’ But a lot of times we miss the main concept which is actually what is our goal with implementing this tool because then we can actually understand what value we're going to get out of this tool. Which means you can start looking at return on investments on these tools. Another subject that I always love is people tend to when looking at costing, they tend to focus only on the initial cost. So how much it will cost to implement a framework, even if it's in Selenium or if it's a tool like Tosca from Tricentis or Smartbear or any of those tools, they focus on what is the initial cost? But most of the time some of these actually have in different environments, different maintenance and costs. That is a long running cost because you've got the resources that you need to either have in-house or training to and being capable of doing the automation. And that I think the most things that people miss when they actually look at tools.

JC: All right, so you say a couple of interesting things. So, let's build that a little bit. Right? So the first one is that the tool needs to fit into your strategic plan for your strategic landscape plan, right? So not only finding out what is the right tool for this environment now but also which of these tools will actually support what I'm trying to achieve from an IT landscape perspective, right? How do you try and evaluate that? So all the tools out there, you do a quick Google search as you said, 1s you find the Seleniums and the Playwrights and the commercial tools. But how do you make that decision then from a strategy perspective? Because all of them are automation. So what are the factors that you look at when it comes to that?

So I usually look at what is the goals and deep dive a bit about it. So a lot of environments these days, people hear about automation and the first thing they hear is it's faster, we save time. But there's hidden times around it as well with maintenance, with things like that that can cost you dearly in the long run. And then a lot of organizations that I've seen implement tools. Yes, initially it makes it faster in the start, but as you progress through your development and your tests become not one test but 1000, 20,000, 30,000 tests that's actually running constantly, sometimes you sit with developers that is actually frustrated more because they need to wait for these things. So when I do an evaluation of a tool or evaluation of something that we need to implement to improve the current situation, because that's what we strive for, is to actually improve the current processes continuously and make sure that we get to a place where we assist and help our people or our employees to actually do their job better and faster. And the thing is, with tool selection, and you said it now the goal I always look in the environment, yes, 90% of any dev teams currently need to develop faster, needs to improve the process, to make it quicker. And I've got this funny saying that we've gone into this stage where we've gone so fast and so quick that quality of products have actually decreased a lot. And the funniest thing is we're just fixing the bugs faster instead of looking at the initial quality of the product. And with tools, that is the main thing as well. We need to go and have a look at what do we want to achieve. Meaning that do we want to improve the quality of the product and go faster? Which means then we need to go and have a look. Do we automate everything or what is this tool going to do? Is it going to be just automation? Are we going to use it for data development or data management or data creation? But there's various things that we need to look at and it's very important to actually look at that way. Also the skills that sitting in the team.

JC:So let's just look at that again. So are you saying some of the tools are better lent towards speeding up the process and speeding up the testing? Some of them are better suited to look at the data creation and set and it's assisting the manual process. And based on that, you decide which tool is right for the environment. Now…are the open-source versus the commercial tools… so are you saying that commercial tools are necessarily suited for one of these scenarios while open-source tools are suited for the other? Or can you use one or the other?

DW: Absolutely not. The funniest thing is I've always been a front runner for free tools. Tools where I have the capability and freedom to actually adapt it to how I want it. It is an important thing because you've got that freedom where… if you get into a problem, there's always your development team that can actually assist if needed. And it's resources that I've got already in the team that can just take some of their time, which also costs the company, but it adds value. And I've got that… how can I say got that capability within my organization, which is much better. But…Outsource or bought tools have in the past few years actually shown me their value. Where if I don't have the capability within my testing team to be able to write a great framework, modular that I can actually adapt, and change quickly and make sure that we use correct automation practices to develop these frameworks with a mind of reusability and maintainability. If I don't have that insight there is tools Tosca from Tricentis is a good example. They have done all these, how can I say, the initial thoughts around a framework and how to develop that framework into a point of understanding that it is reusable and maintainable in the future and also the integration parts. There's a lot of libraries that you need to actually incorporate into your framework and these guys have already done it. So they've already looked at how do I integrate a React native implementation or how do I integrate a dynamic flow or the anchoring, the scanning of elements and things like that. They've done it which is valuable. Yes, there is some points where I'm concerned that a lot of these tools come and then after a while their language, their base language that is behind them actually gets stale or old and we lose that capability within our knowledge base in the industry where people are actually moving past it. I mean, some of the tools are still using old coding languages and to chain for them to upgrade that into newer languages is difficult. But then yet if you have a bought out tool I can easily train somebody to do a manual test to actually start automating. There is skill differences and there's certain things that I still need an automation engineer to understand. I mean, automation readiness of a test case is important and those skills is a bit lacking if you use your normal staff to actually just record and play back because then you're going to go into a situation where I'm recording and playing or recording tests automatically and it's the same thing over and over and over again. So when we get to a maintenance stage, these guys haven't thought of maintenance in the future.

JC: So it sounds to me what you're saying is, especially with commercial tooling, that yes, your initial cost will be higher because you have to buy the tool or technology, but then from there to get to a point of implementation and actually activation and using the tool might be short because they've done all the modularization. They work within certain landscapes a lot better. So that thinking part has been done well. If you go open source, not necessarily because they are frameworks, but if you go open source that you still need to do that thinking and architecting a little bit before you get to that point. So what are you going to pay? On the one hand, you're going to in tooling and technology. On the other hand, you're going to pay that in resourcing costs to actually create the framework and get everything into place. And then from maintainability perspective, there are pros and cons again, right. So with the commercial tooling, yes, you'll have to go away to the vendor to help you with that, which you don't necessarily have the skills. But from a maintainability perspective, if it’s an in-house solution, you have those skills, but you'll have to pay for them again in resourcing costs. So how do you then make that call? If I'm sitting there and I have to start with a new tool or technology tomorrow, how do I buy those things? Which, are the top two or three factors that I really have to consider before I pull the trigger?

DG: So the first thing that I look at, and it's a strange thing it is on resourcing. So how much would it cost me to get an automation engineer with, let's say we do a Java Selenium integrate or framework. How much would it cost me to get a really well-skilled automation engineer, senior level, to be able to do the architecture? Or if I get a consulting firm like we've got on our side in Inspired Testing, we've also got our framework that's already predeveloped as well. How much will it cost me to get a resource in with that capability? And then I weigh it up against, yes, initial cost from licensing or initial cost from getting this person, hiring this person and things like that. But I also have a look at the license fees because how much is it going to cost me for licenses? Because if I've got a free tool currently, I've developed it, I've maintained it, I've got the resources within my organization. It's mine. I don't need to pay anybody license fees after that. With a lot of these bought out tools, or paid tools, as we say in the industry, there is those costs that we need to keep in mind as well. And the funniest thing is most when we do implementations, we look at the initial cost, we look at that, okay, so to install to this and then we look at the license fees, but we don't go and evaluate it against how much will it cost me to have a resource? Because the funniest thing is it's not just the license. We also going to have the resources. Yes, they're going to might be a bit cheaper, but do that actually scale out. And what happens in like I said before, a lot of these tools, we've seen a lot of tools coming and going. CBTA with SAP, we've seen them coming and then two or four, three years later they…they scale off. The partnerships disappear between SAP and them, and then what happens then? I'm sitting now with projects that we are trying to convert them to Tosca because Tosca is the new partner, and that makes their license fees a bit better because they've got the license fee or a better how can I say, deal with SAP. But now how long is that going to last? And then we need to convert from CBTA to Tosca. How much does that cost? With the in-house development tools and things like that? For me to shift between there is no real shift because the tool has been obsolete or it's my library and my IP that I'm keeping within my organization, and it's actually growing as we continue. Yes, the maintenance might become more expensive or it feels like. And the funniest thing is a lot of the development managers that I speak to, they always say the one thing and it is: Yeah, but now I need to maintain another project. It's just another product that we're developing in the company, and we're not supposed to see it. We develop a lot of tools within our organization for a lot of things, for monitoring, for business points. That's a product that we're developing. It's not just our front customer using products, but within testing. Our testing teams also needs tools so that they can be developing faster or testing faster within their teams.

JC: I think that's the important point. So we spoke a lot about implementation and set up and maintenance. But the other part, the biggest part I suppose is usability and you just mentioned that now testing teams need to use them, right? So is it necessary that if you have one of these commercial tools that it's easier to use and someone without all the technical skills can still use it and add value while the open-source solution generally requires a little bit of a high skill set to use as well? So, do you think that that's one of the great benefits of the commercial tooling is that it is easier to use and you don't need to reskill your whole testing team?

DG: The funniest thing is I've worked in many organizations and a lot of times I would sit as a tester, sit with the developers and a lot of people or they will be developing. And the first time I actually look at some of their code and say but I'm sorry, you've missed something here or there's a better standard to actually developing this. They go with this fright of, you know, coding. “You actually can code?” Yes, I've got a few languages that I can actually code and I'm not a senior developer, but I understand code and I can read code. And a lot of testers out there actually has that capability and we don't even know it because they are stuck in a world where they are seen as, you know, what they are testers and that's all they do. But the funniest thing is these days testers spend so much time with developers, they see standards, coding standards, they sometimes read the code, they understand the code. So we might not even know of the resource and capabilities within our teams around coding. And yes there is free training tools out there like the Test Automation University that can teach these basic things to them. Where yes with the bought-out tools most of that back-end is done already so you've got that initial but it is only a small part of the maintenance throughout because a tester in my team I want them to be able to use this tool daily. If they see somewhere where it can add value. I've used it a few times in all places where I just create customers on my databases for testing and it adds a lot of value. And if they can code or they've got that capability, the funniest thing is you can start using your testers then within your develop team to go and review their code because to see what unit testing has been done on those codes.

JC: All right, so again, it's a skill thing, it sounds like, right. So with commercial tools, yes, it's great. You can get up and running and anyone can use it. The maintenance part becomes tricky and I see that as well. If everyone uses, everyone uses it and there's no real architecture around it and no real structure around it sometimes and we end up with thousands of scripts that are duplicated and that you need to maintain. Right. So I get that. But from a business perspective, surely the value is there as your testers are able to automate, right. Your time to production or time to deployment is shorter. So surely that's a great benefit that you don't have with open source tools.

DG: Is it? Really? Sorry. The funniest thing is you just said our goal is to automate. That's not our goal. Our goal is to actually have a better quality product. Our goal is, yes, to move faster within because a lot of teams say that testing is a bottleneck, but it's not really in most of the environments. Because most of the environments SDLC is not set up in a way that we give- and our planning for Sprints and if we work in Agile and things isn't set up in a way that we can actually go faster, we jippoing or we taking away from testing, we all know that. But our goal with automation is not to automate. Our goal with automation is to ensure that the coverage that we have - I think I said to you the other day as well- automation for me is a change management system. It's to make sure that I've got confidence that if we're pushing something to production or something, there's nothing else that we've missed that we've done before. That is the only goal of automation, really. Because if I do it in sprint and we automate, it actually takes me longer than a person sitting, just looking and actually testing it and saying, yes, this is good. But if a person does that, we're losing that continuous quality on the product where we have got coverage on the environments or our products that we are developing. And that's the most important. That's our value that we're going to add.

JC: It gets back to your first opening statement, right, about why do we do this? So the tool needs to support why we are doing something and quicker and everyones using it’s not necessarily the answer. There are business cases where that is the need and then you look at the right tool for that need. If the need is to increase quality and coverage absolutely. Then look at the right tool and I think that really brings it together. So there are many options out there. But when it comes to tool selection, first look at what you want to do and why do you want to implement something. If you understand that, find the right tool or technology that supports the business need or the engineering need. Yes,

DG: It's like testing. Find something that adds value. And not just value because somebody said to you that you need to do automation or it's the new trend out there. It is literally value to our business. What value do we want to get out of this? And that is then adding to how do we look at the costing of it. Because if I get value that, you know what, I've got coverage. Which means is that my risk in production defects are going to decrease. Which means is that we're actually not just adding value by going faster and my testers are reduced by so many people. It's literally looking at, you know what, these are the bugs that we've -I've done it before as well in my environment- where I go and say, okay, what is the common bugs that we found within a sprint? And then calculating what is sitting with the business analysts and stuff and saying, okay, what would have been the risk, the cost, the real cost of these defects within production? Because now I'll know, okay, cool. This is how much a bug will cost me if it goes to production. So this is the value of testing within the environment. Now, is this defect a thing that can reoccur? Because there I'm getting the value of what is my value of automation and that is how you justify the value or the cost of what you want to implement and then we go back to what is the value. The funniest thing is sometimes we miss hidden values even with… I'm talking about Tosca as well because I'm busy with implementations within Tosca. And it's strangest thing is I'm sometimes actually using their training material for my manual testers because the way they've set up their actual libraries and templates and things like that I would love that manual testers when they start developing their test script or their test cases actually think of that as well, because that's reusability. So sometimes with these tools even their training is valuable but with coming back to the main concept-sorry I was deviating now- but yes, it's the value of what I want to achieve and making sure that that value is actually a value that will improve the quality of the product.

JC: So, what we're saying and what he's saying here David is don't go out there looking at all the tools and decide which tools to implement. Decide what your strategy is. Decide why you need automation, what the benefits are to you in automation. Understand your landscape and skill set and then go look at which tool or technology whether it’s open-source or paid for will fit into that. And then weigh up the initial cost and the maintenance cost and the resource cost.

DG: The other thing that I also do is I always so I would say okay cool we do now we limit it down to two or three tools. I usually have like one bought tool and then one or two open-source tools or the other way around depending on the environment. But the funny thing is I always go and I go narrowing it. What I usually do in a project is I would go and have a look at: What is my goal firstly, so what is the goal that we want to reach? And and then I will go and say okay, we narrow it down and we've narrowed it down to maybe two paid tools and two free open-source tools, which is usually that I then use to do my calculations on what is the cost most cost effective in our business and for the return-on-investment things. But I always add a rewrite. It's the strangest thing because then we will go and say okay, let's take these four tools, let's do a POC on them and see. Let's do a test case, one of these or difficult and easy one. We tend to use different skill levels to actually see how easy it is to use these tools. Then I actually do a rewrite budgeting, which means everything that I've done in my POC I wipe. Because if as soon as I've done the POC and I've selected a tool which means as it works with the products that we products or the platform that we are using. So the technology in our organization, we make sure that we've got the skill level in it. So all of those aspects we've seen that, yes, with the POC, they are all in line and this is the tool for us. I would go and literally have a product project, just like if we developing a product for a customer and say okay, cool, now let's do architecture on this. If it's a paid tool or a non-paid tool, let's do architecture and design this thing with reusability and maintainability in the future in mind. And then we set up. Because the funny thing is, in a paid tool you will still have a framework. So you still have a framework within a paid tool. You still need to organize your folders and how you do your scanning of your elements or how do you store it, your data management within it. But it's important then to do the architecture around it to make sure that this is not just we're going to do it now, because that's the problem that we have with our normal products as well, is we tend to do a POC. And then the POC runs into this long running project. And then we start just adding and adding and adding and adding and adding, and later on, it doesn't actually add value. Where if we go and we actually do the design well and the architecture well, it's going to continuously add value in the future. So I think, Jéhan, in conclusion, what I've seen in various industries and things is. We're not just taking the tool to just implement the tool. We need to really go and look at the values. And if we've got that value lines and we say that, you know what, I want to increase my coverage or I want to actually make sure that my testers have extra time because we're spending a lot of time on regression, then those things will affect the tool that I'm actually selecting. It then the cost. If we go to the cost, it is important to look at all the costs, not just the initial cost, not just what the license fees that actually look at every cost there is and weigh it up against resources that you have in your team or actually getting people with the capability and understandability. Because if you hire a person that is a good architect for these tools, developments or whatever, they can actually be valuable in your paid tools as well, because they know the standards, the automation standards and stuff.

So, in conclusion, there's a lot of sites on Google that can actually tell you these are the five or six things that you need to look at. These are how you evaluate it. I know Michael Bolton did, and I did nice chat around testing the test tool at one of the conferences I was at. But we need to understand that all of this we are weighing up to what is the value that we're going to add and is it a short term value or is it a very long term value? And if it's a long term value, how is the maintenance going to be on this and is it still going to add value or is it actually going to be a burden to people around me? And that is, I think, the biggest thing that we need to look at when we select the tools that we use.

JC: Thank you, Dawie. Yeah, it's been very insightful. I think I'm walking away with a lot of things to think about. It's not as simple as just googling and implementing. You mentioned a couple of factors and thank you for that. Appreciate your time. Thank you.