Organizations can swiftly respond to rapidly changing market needs faster than their competition and excel user experiences only by maintaining continuous delivery of high-quality software releases.
In this article, let’s highlight some of the key metrics that should be monitored and improved to optimize the performance of CI pipelines. To measure the success of a continuous integration pipeline, we’ve curated these three metrics to monitor: change failure rate, test coverage, defect escape rate.
Why not just use the DORA metrics?
The DevOps Research and Assessment (DORA) research program provides guidance for how DevOps teams can continuously improve their processes and capabilities. In their book, Accelerate, they identified a set of metrics that they claim indicates software teams’ performance. The following metrics are known as DORA metrics: Change Lead Time, Deployment Frequency, Mean Time to Resolution, and Change Failure Rate.
The DORA metrics make DevOps measurable in means of the full development life cycle. In other words, these metrics help engineering teams take data-driven decisions. The DORA metrics help deliver software faster reliably.
When we focus on specifically optimizing a continuous integration pipeline, we take a slightly more particular approach when it comes to the metrics that we need to monitor, and take into account.
Let’s look at how we approach CI performance monitoring by tracking change failure rate, test cycle time, test coverage, and defect escape rate.
Change Failure Rate
The change failure rate is the percentage of code changes that lead to failures in production. It is code that needs to be fixed or called back after it has been deployed. Defects in production caused by code changes comprise this metric.
A Change Failure Rate dashboard from AWS DevOps Monitoring solution
While tracking a change failure rate, we only take into consideration the failures that happen in production but not the ones caught and fixed during the testing phase. In addition to that, defects may happen at the users' end that are no fault of the developers. Change failure rate metric should only be counted when there’s a change in the code which can be anything from new features to quick fixes.
How to Improve Change Failure Rate
The change failure rate can be improved with a holistic and continuous effort. Anomalies and defects should be monitored carefully not only in the production environment but also during the testing phases.
It's pretty hard for code reviewers to know the impact of the PR in production. They want to know and focus on the parts of a code change that matter most to the end users. Hence, it's difficult for code authors to ensure their testing strategy is in alignment with how their applications are actually used. Moreover, change management processes are mostly manual and daunting.
Furthermore, code reviewers have a direct impact on this metric. This metric reveals the quality of the code reviews. More precisely, pull requests should be rated by the risk impact that they might have on production. Code reviewers should get automatically clued in on the risk level that the PR they will review. This way imprecise reviews will be annihilated.
The best way to improve the change failure rate is when errors, bugs, or failures are caught during the testing phase in CI workflows.
Erroneous workflows dashboard from Foresight
Test Coverage
Test coverage should not be confused with code coverage. Code coverage is the percentage of code that is covered by test cases through testing frameworks and suites. Test coverage is the percentage of how many software functions and features are covered by tests or test suites. Both code coverage and test coverage relate to software testing that improves code quality and, therefore, your CI pipelines.
Change impact analysis dashboard from Foresight
Test coverage has many practices such as unit, functional, integration, and acceptance testing. These tests focus on the different business positions and business values of the software. This means that test cases can be created before or without the knowledge of the code itself.
Testing everything for every commit is a timesink, old-school practice. Long testing cycles slow down and delay releases. Without quickly debugging test failures, detecting flaky tests, identifying slow tests, and visualizing performance over time to identify trends, you’ll waste so much time.
Long-running test suites and frequent failing tests are the most common reason for slowing down build times and hence reducing deployment frequency. You should have visibility into test runs to quickly debug test failures, detect flaky tests, identify slow tests, and visualize performance over time to identify trends.
Defect Escape Rate
Implementing DevOps practices like building CI and CD pipelines to achieve excellence in software delivery does not always bring a hundred percent success. Software teams can face challenges such as defects escaping from their sight even if they follow the best practices of continuous integration and continuous delivery.
The defect escape rate is a measure of the number of bugs and defects reported by end-users or customers of a product.
Test monitoring dashboard from Foresight
To improve the defect escape rate, software teams should take into consideration of the below practices:
- The QA and testing environments should be closely monitored and erroneous tests or tests with unexpected latency should be examined carefully.
- Debuggers should be introduced in the pre-production phase to catch any potential bugs or mistakes.
- The testing strategy should be strong, strict, and wide enough to make sure any part of the code is tested well. Test automation is a must, but manual quality assurance should also be a part of the QA strategy.
Observability of CI Pipelines
Improving your CI performance through metrics means understanding your continuous integration process, architecture, runtimes, engineering teams, and development process.
Having clear observability of your CI pipeline means being able to track, monitor, and gain insights into CI performance. Monitoring, error tracking, or CI/CD tools and platforms today do not solve this. One of the best ways to prevent production regressions is to have an observable CI pipeline.
CI Observability consists of monitoring workflow resource metrics such as CPU load, memory usage, disk, and network I/O, tracing processes at Kernel level to monitor inside workflow steps, monitoring test suites and tests, understanding the impact of the code changes on the production environment, and prioritizing tests at every CI workflow run to optimize performance. These are some of the key abilities that Foresight provides. You can try Foresight’s GitHub application yourself.
Save the date! 🕊️
On the 28th of September, we are launching Foresight on Product Hunt 🎉🍻
Take your seat before we launch on Product Hunt 💜
Top comments (0)