Is There Such a Thing As Too Much Test Automation

    Automation is a good thing, right? How can you possibly have too much of that? What many teams sometimes seem to forget, however, is that automation itself comes at a cost, too. Test automation is a means to an end, not the end goal in and of itself.

    Having been involved in testing and test automation for some 14 years now, including as a test automation engineer, consultant and trainer, I’ve seen and worked with a lot of organizations that are at some stage of their transformation towards a continuous delivery-like model of software development and delivery. As part of this transition, development teams within these organizations typically spend a lot of time and effort building automated tests.

    Now, test automation in itself is definitely a good thing. When done well, it’s an excellent means to getting insight into specific quality attributes of the software that you’re developing. It provides a great safety net for developers during refactoring, and offers a way for testers to focus on the tasks in software testing that make their work interesting, too. However, in their quest to make the software development, testing, and delivery process as efficient as possible, many of these teams  fall into the trap of “too much automation.”

    You may wonder, how is that even possible? Automation is a good thing, right? How can you possibly have too much of that?

    What many teams sometimes seem to forget, however, is that automation itself comes at a cost, too. Not only does it need to be built in the first place (test automation is software development—even when you’re working with low code tools), but it also needs to be maintained in order to keep it running smoothly and to ensure that it keeps delivering the information that you’re looking for. This means that it might be a better idea to work towards an optimal amount of automation, rather than blindly trying to maximize your automation coverage. Or, as the well-known management consultant Peter Drucker put it so eloquently:

    “There is nothing so useless as doing efficiently something that should not have been done at all.”

    Automation is a means to an end, not the end goal in and of itself. We should use automation to get faster feedback and increase the speed with which software can safely go from a developer’s machine to production, while making our lives as testers easier in the process. In the remainder of this article, I’m going to dive into two questions that can help you in scaling down your test automation efforts and in preventing your automation suite from becoming a liability rather than the asset it should be.

    The first question I would like to see asked more often, as a countermeasure to the ubiquitous “automate all the things” attitude that is so prevalent these days, is the question of “what to automate?” The problem here is that if you ask “what should we automate?” the answer will often be geared towards doing more automation, while forgetting to consider whether or not that’s even useful. For this reason, I’ve learned to rephrase this question as:

    “Are we comfortable with not having the information provided by this test?”

    Remember that testing is all about uncovering and providing information, and test automation is a means to perform specific testing tasks more efficiently. Would you be comfortable to deploy, deliver, or use your product not knowing the outcome of a specific automated test? If so, you might want to consider not spending the time and effort it requires to write (and subsequently having to maintain) that test. What use does test automation have when it provides information that you do not really need?

    This question can also be extended to help you decide what tests to include in a continuous delivery pipeline:

    “Are we comfortable with not having the information provided by this test before we deploy?”

    It might very well be that you have some tests that you would like to see run from time to time, but not necessarily on each new increment. These might be costly tests (in terms of time consumed, for example), or tests whose result falls into the “nice to have” category of information.

    Oh, and asking the above questions can help you weed out the clutter in your existing test automation suites, too.

    The second way to scale down your test automation efforts in order to make them more efficient is not directed towards “what” to automate, but rather to the “how” of automation. Even though most people involved in testing and test automation have been aware of the fact that there’s more to test automation than mimicking end user interaction through full-stack test automation, many teams are still struggling from an overabundance of tests that exercise the application “from DOM to database.” One way of opening up the discussion about whether or not a test should really be a full-stack test is asking yourself, your fellow testers and your developers:

    “Is this the most efficient way to get the information that we’re looking for?”

    This question, when applied on a regular basis, can open up all kinds of opportunities for better and more efficient and effective test automation. Possible outcomes of applying this question might be:

    • Identification of full-stack tests that only cover backend business logic and that can be tested more efficiently through API-level integration (or even unit) tests.
    • Identification of bottlenecks with regards to dependencies that could be virtualized for more efficient testing and automation.
    • The decision not to automate some tests that were thought ripe for automation, because having them performed by testers turns out to be more efficient.

    All of these outcomes will, in turn, lead to a test automation suite that’s more efficient, gives faster feedback and more valuable information, and requires less maintenance.

    So, instead of simply striving for the highest possible coverage through maximum application of test automation, why not try scaling down your tests and see what happens?