How to Enhance Your Deployment With Continuous Testing in CI/CD

Digital transformation has gripped us all! Most of the businesses are moving towards a cloud-centric environment and leaning towards ‘Shift-left’ DevOps practices. This rapid paradigm shift in Software Development Life Cycle (SDLC) is creating a need to integrate continuous testing to build out resilient CI/CD pipelines. Balancing quality and speed is a major challenge and continuous testing in CI/CD can act as a catalyst in the whole process.

To ace the competition, businesses are looking for better ways to perform CI/CD Pipeline Automation Testing. One of the approaches adopted by many is an integration of continuous testing in the entire product development life cycle, right from the start until the end.

What Is Continuous Testing?

The basic goal for testing any software is to isolate errors or bugs that will impact the quality or will compromise the security framework. In the case of legacy waterfall development cycles, the release cycle was long, and the software tester / QA would spend time manually simulating and testing various scenarios. However, as organizations are adopting DevOps ways of working, the frequency of release is changing from months to every hour and it would be inefficient as well as a humongous task to manually test each time. Also, manual testing might not cover all the corner cases, leaving scope for production defects.

Continuous testing in DevOps is an uninterrupted process of testing throughout the SDLC where test cases are included in every stage in the CI/CD pipeline. Right from code commit, to build, to provisioning, to deployment, and everything that happens in between—all the steps are tested automatically. In case there are any bugs encountered in the process, instant feedback is provided to developers ensuring rapid fixes of failure before the final release.

Test Automation vs. Continuous Testing

Test automation and continuous testing are often used interchangeably. However, these two are not the same. Wikipedia defines Continuous Testing as “the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.” However, this definition is not accepted by many. Lisa Crispin, an influential Agile Testing Professional, highlights that automated test is an integral part of continuous testing framework but the terms cannot be used interchangeably as continuous testing in CI/CD is not just about test automation.

Continuous testing in DevOps encompasses and is a collaborative effort involving processes, teams, and tools along with automation. It is an evolution of test automation and aims to provide validation and feedback at every step of the software delivery pipeline - right from design to production.

Why Do We Need Continuous Testing?

Continuous testing is beneficial in more than one way. Following are some of the benefits of CI/CD testing:

• Shortening and amplifying the feedback loop – If the automated Unit, functional & regression test cases are integrated into the build pipeline, feedback is available almost instantly for each step. Feedback from the test tasks provides immediate and useful insight to the whole team on the stability, quality, and consistency of the product.

• Quality never takes a back seat and the software is tested continuously on the run, from the start, and until the end.

• Maximum test coverage as CI/CD Pipeline Automation Testing as a practice is incorporated right from the earliest stages of development.

• Boosts the morale of the team and also provides confidence before delivery and deployment to stakeholders regarding the quality of the overall product.

Include and Integrate Continuous Testing in CI/CD

Now that we’ve looked into what continuous testing is, and how it is aligned with the test automation, let’s focus on how we can bring it all together and fit in the DevOps pipeline.

In a typical scenario, a developer writes a piece of code and commits it. The code then goes into version control. This is the first step in DevOps. In the next phase, continuous integration, the code is merged into the main branch and then Jenkins / Git is used to build and test. Following this, as part of continuous testing in CI/CD, QA writes automated test scripts and executes them.

As part of the continuous testing in DevOps, test cases are introduced even before the developer starts writing the piece of code, i.e. during the design phase, and until delivery. If the test fails, the failure notification is sent to the developers and if the test passes, the code is pushed for Continuous Deployment. All the aspects of CI/CD testing like Unit, functional, security, and performance which were manual before should now be automated and made as atomic as possible to help reduce refactoring effort. Having said that, we need to ensure we try and automate enough to be able to push something from the design and development stage into production automatically or with minimal effort possible.

Let’s understand how these tests should be selected, written, and executed.

Choose the Test That Best Fits the Use-case

Multiple types of CI/CD testing can be integrated into the pipeline as it is important to have coverage on all the software aspects. Some of them are listed below:

1. Unit Testing – These are short tests written from a developer’s perspective predominantly on single classes and functions i.e. the smallest individual unit of the code.
2. Integration Testing – These tests are written to test if multiple classes and functions are working together.
3. Functional Testing – These tests are written from the user’s perspective. The test cases ensure that all the functionality is behaving as the user is expecting it to be.
4. End-to-End Testing – These tests are also known as workflow test as it tests the entire functionality of the application from backend to frontend. These tests are larger as it tests the entire workflow of the application.

The list is not comprehensive however, it touches on key testing types. Apart from the ones mentioned above you will hear API Testing, Performance / Load Testing, Stress testing, regression testing, security testing, UI testing, etc. Each of these testing variations can be performed using multiple CI/CD Test Automation Tools which allow concurrent test execution like Selenium and Cypress.

Make Your Tests Independent

For the tests to run smoothly in a continuous integration (CI) environment, it is important to run them independently. Let’s consider that you are building an application to help find restaurants in the city. Some of the important workflows for this application would be, users should be able to search the restaurants, login into their accounts, and check out the different restaurants based on selected criteria. So while writing the test workflow focus on creating independent test cases to run the test on each of the features. This will help reduce the CI/CD testing time and will also help identify failures at a feature level.

Run Parallel Tests

In serial CI/CD testing, the tests are run in a sequence one after the other. However, in parallel testing, multiple tests run side by side, and hence the overall execution time is shorter. For a parallel test execution suite to run smoothly, factors like the number of processors and the capability of the processors play a very important role.

Evaluate and Decide the Testing Type

Identifying the types of tests that should be included and automated in the CI/CD pipeline of the software is a difficult task. Agile Testing Quadrants is helpful for the teams to identify and plan the types of tests that should be included, and the resources that would be required for implementation. This model was introduced by Brian Marick, an agile testing consultant, in the year 2003. It combines both the aspect of testing, business, and technology.

There are four quadrants as depicted in the picture below. The bottom of this matrix is tech-focussed, while the top is business-facing. The left-hand side focuses on tests that guide the development of the product whereas the right-hand side critiques the product.

Quadrant 1 (Lower left corner) - All the technology-facing tests that guide the programming team are part of this quadrant like unit testing, integration testing, and component testing. These tests are automated and provide regular feedback to the developers on the code quality and if the software is behaving as expected. The test cases for CI/CD Pipeline Automation Testing are written alongside development to confirm that the functionality being built is moving in the right direction.

Quadrant 2 (Upper left corner) - Partially automated and partially manual, this quadrant is business focussed and the tests are performed to validate product specifications, simulate or build a prototype. Automated functional testing which is done to ensure the product is aligned with the business specification is part of this bracket.

Quadrant 3 (Upper right corner) - This quadrant is focused on business-facing tests that critique the product. It primarily includes manual testing like exploratory testing, A/B Testing, User Acceptance Testing (UAT), and usability testing. These tests are more user-oriented and are helpful to assess aspects like ease of use, flaws in application workflow, etc.

Quadrant 4 (Lower right corner) - This corner is for technology-facing tests like performance testing, load testing, and security testing of the software application. Several tools are available today which can help automate this type of testing like Apache JMeter, LoadNinja, etc. The tests in this quadrant should be prioritized based on the application-specific need. For example, for reporting software containing non-sensitive data type information, fast refresh and loading of the page is of utmost importance so implementation of performance testing would add value.

Following the order in which quadrants are listed or ensuring at least one test from each quadrant must be included is not necessary. The choice of type of CI/CD Testing and the number of tests to be included in the software is circumstantial. So, assess the need and leverage the test or the quadrant which is relevant for your specific use case. These quadrants should be used more like a guiding framework so that the whole team understands the importance of testing and starts thinking about it right from the start.

Testing Pyramid: Give Structure To Test Automation Strategy

Coined by Mike Cohn, the testing pyramid is a test automation framework that gives a graphical representation of testing types and the ideal composition of each of them for better coverage. However, it’s not a complete guide and there are other tests too that should be included apart from the ones part of the pyramid.

The Pyramid calls for automation testing at three different levels. Unit testing is the foundation of automation test strategy followed by service-level testing and user interface testing. Service here refers to testing the application’s response based on the set of inputs.

CI/CD Test Automation Tools

It is important to involve the complete team in the test automation process. Business stakeholders can help classify high-priority features, tests can help the developers identify the edge cases, and the developers can help write the test scripts. Keep the automation suite lean and create a strategy on how much automation is possible and what tools will help achieve that.

Automation Tools

After defining your automation strategy and deciding what automated tests to incorporate, the next step is to select the automation tools. As a first step in selecting the tool, focusing on defining two things:

1. What tests need to be implemented?

2. Which is the preferred language in which test cases should be written?

Once you have clarity on the above two, start looking for automated tools that best suit the requirement and promote cross-functional collaboration between testers, developers, and business stakeholders.

Some of the popular test automation tools are:

Selenium
Selenium is an open-source end-to-end testing framework with a user-friendly UI. It supports parallel testing and the test cases prepared can be tested using any OS. The framework also allows the testers to write complex test cases in RUBY or HTML.

Selenium is compatible with most of the popular programming languages, including Java, JavaScript, C#, Python, Perl, and PHP.

There are few limitations of Selenium too like It does not support CI/CD automation Testing for desktop applications/windows applications. Also, the Selenium testing framework depends on other third-party frameworks like cucumber for reporting.

QuerySurge
QuerySurge is an automated data testing and validation tool. It is used for testing Big Data warehouses, Data migration products, Extract Transform Load (ETL) solutions and Business Intelligence (BI) reports.

CI/CD test automation for data solutions can be done using QuerySurge’s Restful API or Command line. The process can be automated to execute unattended and post-execution, it shares all the reports of pass/fail results and data anomalies detected over email.

One of the limitations of QuerySurge is that it does not point to the exact location/cell where data discrepancy exists. However, this can be overcome easily by writing a small utility on top of the tool to automatically detect the exact location where the data mismatch is.

Apache JMeter™
JMeter is an open-source application that is used primarily for load and performance testing. It can be used on not only static but even dynamic resources, including Web dynamic applications.

This DevOps load testing tool can also be used to perform automated functional testing. The stress testing helps identify the point where the software application can fail. JMeter supports multiple plugins like basic graph and Response Times Over Time Listener.

The application is designed on a multithreading framework, allowing concurrent sampling on multiple threads and simultaneous testing of varied functions on separate threads. You can also visualize the test results in multiple different ways like tree views, XML, graphs, JSON, etc.

Apart from the ones listed above, there are many other continuous CI/CD testing tools available. For example, there is Nunit for C#, JUnit and TestNG for Java, and Embunit for C++ development. Choose the tool that best supports your continuous testing requirements.

Continuous Testing Best Practices

Apart from including the multiple types of CI/CD Testing and enhancing the testing coverage, when designing the continuous testing framework focus on some of the key points listed below:

1. Adopt “Shift left” testing and integration testing right from the initial stages of development and into each branch in the pipeline.
2. Testing should be done on the staging environment which is the replica of the production server so that it’s easy to identify the false negatives.
3. Continuous testing in DevOps is more frequent and by multiple owners. Hence it is better to do the testing in a virtualized environment.
4. Leverage CI/CD Test Automation Tools to help scale testing.
5. Identify and remove “flaky tests” from the Continuous Testing Pipeline.
6. The test results should be carefully assessed to ensure there are no failures in the production.
7. There should be a well-crafted roll-back plan in case of any failure in production.
8. Build dashboards to constantly monitor the availability and performance of your application while in production.

Changing Responsibilities of the Test/QA Teams

DevOps is also cross-referenced as agile on steroids. To keep up with the pace, testing needs to be automated more than ever now. The scope of testing is not limited to re-running unit test cases, functional tests, workflow, bug finding only but also includes security testing, load, and performance testing. Hence it is crucial for the testing teams to:

1. Understand all integrations critical to user workflows to design a Continuous Integration Testing framework.
2. Prepare and move an automated smoke test into the Continuous Integration (CI) build process.
3. Build comprehensive regression suites and include them in the CI build process.
4. Run smoke and/or regression tests on virtual environments and multiple VMs.
5. Optimize the test cases embedded in Continuous Testing Tools to improve quality and velocity.
6. Make use of containers and/or virtualization for data and replica of the production environment.
7. While 100% automation may not be achievable but attempt to automate the maximum possible and build CI/CD Test Automation Tools.
8. Choose API over GUI as GUI testing takes longer and also requires a lot of rework.
9. Think beyond automation and design intelligent systems by leveraging Artificial Intelligence (AI) that perform heuristic thinking on every CI/CD pipeline.

The trick to successful continuous integration and continuous delivery is continuous testing. Continuous testing in CI/CD helps isolate failures and provides a critical feedback loop to move fast and build effective software solutions. Most importantly, it helps build a quality CI/CD pipeline and reduces the risk and wastage in the software development lifecycle.