Automated Testing: The Cornerstone of CI/CD

This post was contributed by Ferdinando Santacroce.

In the Italian language, there is a single word for computer science: Informatica. It is a portmanteau of “informazione automatica” and, as you may have already spotted, it means something like information automation.

Automation is a pillar of information technology. Among experienced software professionals, it becomes instinctual. Every time you find yourself doing the same job over and over again, the desire to write a script and automate a repetitive task asserts itself.

This attitude is key for software developers, but sometimes things get overcomplicated. Then, automation becomes difficult, and manual work becomes necessary. This article describes what automated testing is, the path to a fully automated test suite, and the challenges that lie along the way.

What is automated testing?

Like any other material good, software needs to be tested before reaching the customer. The obvious way to test something is to use it for a while to make sure it behaves as expected. This is the way most developers in the past used to check their work before shipping it.

After all, you have written it, you have it on your computer, you know how to run it, and you know how to quickly test it as you make changes.

The flip side is that you as the creator are biased. You might forget to test some parts or might not realize the ramifications of a change. Also, you can’t be 100% sure that your code will work in an environment that isn’t your development machine.

Even if you do everything right, testing is a boring task — a time-consuming activity where skilled, creative professionals are forced to do mindless drudgery. Historically, the countermeasure was hiring people to test, making developers happy and removing biases.

This seemingly win-win solution makes things worse more often than not. Developers lose the overall picture and awareness of all the moving parts when they aren’t involved in testing. Testers, on the other hand, spend an enormous amount of time just making things work and have to bother developers whenever they find something they don’t understand.

The role of a QA team when testing is automated

Testers are part of what is known as the Quality Assurance (QA) team. They are paramount for shipping high-quality products, so it’s important to let them do their job efficiently. They do not have to test everything every time, nor catch unhandled exceptions or 404 errors. They are there to help developers and companies to deliver good quality software. Otherwise, the “QA is doing the testing anyway” mindset settles in, giving rise to dysfunctional behaviors.

The inflection point for automated testing

In 1999, the book Extreme Programming Explained laid the groundwork for a revolution, which later resulted in the Manifesto for Agile Software Development. In this text, we find the cornerstones of modern software development:

1. Rapid feedback loops.
2. Simple and safe ways to modify code. As Kent Beck said, “make the change easy, then make the easy change.”
3. A happy and satisfied team when the work is done.

In this field, automated testing plays a key role. Let’s have a look at some kinds of tests we can put into use.

From unit tests to end-to-end tests

The topic of testing is very broad and there are dozens of different types of tests, depending on the needs of the developer. Let’s suppose, for example, that we want to create software that can manage prices in a supermarket. The code below is an example of a unit test:

@Test
 public void When_I_add_an_apple_the_system_charges_50_cents() {
 CashRegister register = new CashRegister();
 register.add("apple");

 int expectedCost = 50;
 int actualCost = register.getAmount();

 assertEquals(expectedCost, actualCost);
}

In this test, we are verifying that our cash register charges 50 cents for an apple. Although simplified, the example gives a good idea of what it means to test a “unit of code” (the cash register), in a very focused way. If the price of an apple changes or the system stops pricing apples as expected, this test will alert us. A unit test validates an assertion on a specific portion of code.

There are other types of tests, aimed at ensuring the expected behavior of the application as a whole, from the graphical interface to the database. These kinds of tests are called acceptance and end-to-end (E2E) tests.

Our range of tests, therefore, goes from unit tests, which are focused on the technical details, to acceptance tests, which instead show that the required business objectives are being met.

Feature: Supermarket Bundles        
           To increase the revenues as a Supermarket Manager, 
           I want to apply discounts and offers to my customers

Scenario: if customer buys 3 apples, he pays $1.00
    Given the apples costs 50 cents each
    Given the “buy 3 apples, pay 2” promotion 
    When the cashier scans 3 apples
    Then the cash register charges $1.00

In the excerpt above, we see an example of a specification written in Gherkin, a language made famous by Cucumber and used in Behavior Driven Development (BDD). Despite appearing as natural language, there are automated tests hidden below the statements in this scenario.

The characteristics of a good test

All good tests have some features in common.

A good test is deterministic

A good test is deterministic and idempotent. It doesn’t matter where and when it runs, it should always produce the same (verifiable) outputs given the same inputs. Nothing is more frustrating than an unreliable test that fails randomly due to unknown or uncontrollable conditions.

Writing deterministic tests is not always easy, especially if the codebase was not designed to be testable from the beginning. Good programmers should take all the time they need to write deterministic tests.

A good test is fully-automated

A good test must be endlessly and effortlessly repeatable. As a developer works to implement new features, an external server can work to ensure the expected behavior. This is achieved by continuously running all the tests without any additional effort.

Machines are good at repetitive tasks, so let them do what they were designed for. Leave the developers with the higher and more rewarding task of satisfying users by unleashing their ingenuity.

A good test is responsive

A good test provides quick, honest feedback. Experienced developers know that testing is primarily about feedback. Having a test suite that offers feedback after hours or days is a big problem because it breaks cognitive flow. A developer can more quickly and effectively fix bugs the sooner they are discovered.

Build a robust test suite

A solid test suite has tests separated according to how long they take, their complexity, and the resources available. Unit tests run very fast — usually a matter of milliseconds — while end-to-end tests are slower, and more often than not they require staging environments to be properly installed.

In the end, a good developer should build a well-organized test suite over time, and write plenty of unit tests to run continuously. Here’s where a CI/CD pipeline comes into play.

The heart of Continuous Integration (CI)

At regular intervals, developers integrate changes with those of the other team members, using a versioning tool such as Git. A well-established team has a strong Continuous Integration policy and a CI pipeline that, upon noticing the changes, will take the new code from the repository and run tests on it — from quick unit tests to more rigorous acceptance tests. More costly tests can be run less frequently, such as once or twice a day. This lets the developers know immediately, with a good degree of confidence, if something stops working.

As in many aspects of life, there are trade-offs in testing strategy. It’s up to the developers and testers to negotiate the best compromise, ensuring quality and efficiency.

Simplicity and automation are prerequisites for safety

As mentioned earlier, software professionals must ensure the quality of their own work. Quality means low defectiveness, and low defectiveness means a vast amount of testing. Thus, writing good tests and building a system that can execute them quickly and accurately isn’t optional. Teams must commit to building a safety net with CI/CD to ensure smooth development.

The Myths of Test Automation

Despite the growing awareness achieved by companies and software developers in recent years, test automation is still plagued by some myths.

Testing slows down development

Those who claim that testing slows down development are both wrong and right. They’re right because in some cases, especially in TDD, writing tests makes developers take the time to think ahead about the result they want to achieve, setting aside the frenzy of writing code for a moment. In this case, slowing down is an act of will, a precise means of focusing on the problem ahead.

In the long run, however, those who think that writing tests are a waste of time are wrong. We must think of the time spent writing tests as an investment because a failure-safe environment makes it possible to make changes and experiments at a very high speed. Developers can come to a solution in a fraction of the time it would take if they constantly had to be careful about where they put their feet.

It’s like walking home on the sidewalk versus walking on a tightrope stretched between two buildings: tightrope walkers are not famous for their speed.

Abraham Lincoln used to say “Give me six hours to chop down a tree and I will spend the first four sharpening the axe“. Testing automation is like having a little helper that constantly sharpens your axe, so you can “hack” down any problems that pop up without spending hours sharpening it yourself first.

Testing is only for finding bugs

Another myth is that testing is only for finding bugs. Going after bugs is one possible purpose, but certainly not the only one. The real advantage lies in making bugs easily detectable and fixable after every single change. That is what test automation is all about: making it easier for a developer to alter code because they know they have a safety net.

You must achieve 100% coverage

Modern development environments allow you to calculate a metric called code coverage. It represents the amount of code covered by tests. It is quite natural for the non-test-savvy to think that 100% coverage guarantees the total absence of bugs in their codebase. This belief, apart from being incorrect, is a harbinger of bad practices.

Coverage only guarantees that there is at least one test that puts a certain portion of code under scrutiny. It does not guarantee that said test makes sense, that it verifies all the possible combinations of inputs, or that it will always behave as expected.

Writing tests is a profession somewhere between art and science, and coverage is certainly not the tool that guarantees the quality of a test suite. Furthermore, the drive for 100% coverage leads developers to test trivial portions of code, wasting time and money, and feeding an unhealthy perception of procedural infallibility. Setting a more reasonable threshold, such as 80%, might be a good compromise.

How to define a good testing strategy?

At this point, the question arises: what are the steps to creating a good test suite? Let’s try to define a roadmap for a company or a team that wants to change course and start testing its codebase.

Start testing

No, this is not a joke: the first thing you need to do is start testing. Usually a simple query on a search engine like “ testing tools” provides plenty of hints and tutorials to get started.

But even if you’re using outdated or abstruse languages, you can always find a way to test your product. If there are no frameworks or ready-made tools on the market, you can always write tests using pure programming languages or with bare shell scripts.

To paraphrase Walt Disney, “If you can run it, you can test it.“

Identify a meaningful part, and write an automated test

In codebases that have developed with no inclination to testability, the road is uphill at first.

The first tests written should offer a good balance between value and affordability, i.e. code that is sufficiently easy to approach, but that at the same time can provide you with some value. Testing a part that never breaks, a trivial feature, or one that almost nobody uses is not helpful. Moreover, going straight to the main feature of your product usually ends with brittle tests and frustrated developers.

Most of the time, we can start by writing some end-to-end tests that try to emulate the behavior of the user. If we have a web application, for example, we can begin by training a browser with tools like Selenium or Cypress. Let’s not, however, limit ourselves to tools; we also have to be ready to manually write our test routines, shell scripts, and whatever else is needed to make the test automatic.

In the journey we take, we always discover useful details for increasing our understanding of the codebase, details that will be valuable when we go to modify or refactor bits of code. Once we understand the basic principles of testing, we can start learning about testing frameworks, mocking libraries, and all the tools that make a developer’s job easier.

Create your safety net

A good test suite is a safety net. In the beginning, when you don’t have tests, every change is an unknown risk. Over time, and with the addition of tests, the risk decreases, confidence increases, and development accelerates.

It’s a long process. You can’t hope to solve all the accumulated problems in a few weeks. The important thing is to keep adding tests, and reinforcing testing practices in the team.

Make automation an integral part from the beginning. If a new project is about to take off, don’t waste the opportunity to start off with the right foot with a well-rounded testing strategy and a solid set of CI tools. Once the right procedure for conducting a test has been identified, the test must be automated, and then it must be run in a secure, isolated environment: a CI/CD pipeline.

This is about feedback: the longer you wait, the later you will know if your automated tests are as isolated and idempotent as you want them to be, or if they “just work on your computer”.