Simple ways that make it easier to debug tests in your CI pipeline
If you develop microservice applications that run on cloud environments, you may be wasting more time than necessary debugging tests in your CI pipeline.
In this article, we share 3 ways that can make debugging tests in your CI pipeline easier:
- Instrument your tests to get test analytics
- Implement distributed tracing to pinpoint the root causes of a failure
- Debug on the CI server rather than try to replicate everything locally
Bonus Section: How to debug your tests in CI before production
The Need to Debug Tests in the CI Pipeline
Looking for an answer to why my tests fail in CI
As developers who write our own tests, we all ask the questions below:
- Is my build successful? Did it fail?
- Which tests failed? Are they ignorable ones?
- How can I debug a failed test?
- This is wasting time. How can I accelerate the build process?
Yes! Our main concern is time management.
The real problem is that testing software is a time-consuming process and it becomes a nightmare to understand tests when they fail or take forever to finish.
When reviewing the results of Electric Cloud’s survey, it is obvious that developers spend considerable time waiting for tests and builds to reach completion.
Additionally, Tidelift and The New Stack’s joint survey reveals that testing code takes up an important amount of time for software development.
What Challenges Are Faced When Tests Fail
What should I do if my build fails?
You should not let your pipeline deploy to production.
It is very challenging to implement CI practices within a software team, but following some strict rules can help you avoid common mistakes and keep the CI process in a healthy place.
Your build most likely fails because your tests fail. Most CI pipelines today, like Jenkins, Circleci, GitLab, TeamCity, Bamboo, and GitHub Actions, are configured to automatically cause the build process to fail when tests fail.
There may be several reasons for a build failure. The main reasons are listed below.
Test Failures*:
Failures of unit tests, integration tests, functional tests, or performance tests can cause build failures.
Compilation Failures:
Code failures during the compilation process and test classes’ compilation failures may cause build failures.
Bad Code Quality:
Low test coverage, coding conventions, and code duplications can cause build failures.
*According to this study, more than 87% of build failures originate from various test failures.
How Google Tests Software by James Whittaker, Jason Arbon, and Jeff Carollo is an exceptional book about how a tech giant transforms the testing bottleneck into an accelerator for its developers.
Let’s discuss the following question to dive a bit deeper.
How can I debug a failed test?
Aggregate tests: We need to understand which tests failed first because we may have run anywhere from a handful up to hundreds of tests. To do this, we need to instrument the tests and the tested applications’ backends in order to get test analytics such as traces, metrics, and logs.
At the commit level, test runs and statuses should be aggregated to point out which tests passed, failed, or skipped.Distributed tracing: After detecting exactly which tests failed, we should be able to identify which service and request caused the test to fail. While there are a lot of external services and dependencies used in tests, we should nevertheless be able to understand whether a unit test or an integration test failed, and we should be able to determine the root cause of the failure without getting lost in log piles.
Code-level information: We should see contextualized errors of the failure by examining source code and structured logs and events. The error might be thrown by our service or a dependency, which means we need to have per-test code coverage for each individual test run.
Debug remotely: Since the tests and the tested applications are instrumented, we need to be able to selectively record the test failures. This way, we can debug the failures in the environment where the tests actually run rather than trying to replicate everything locally.
How long do my tests take in the CI pipeline?
When the total build time in CI crosses a limit (approximately ten minutes), we would want to take a look at exactly which tests we ran that caused the latency.
Unfortunately, production observability tools don’t work successfully for CI like they do in production. There are key differences between production and CI environments that explain why this is the case. A production environment is a lasting system that can handle transactions, but a CI environment sometimes goes through four seasons in one day as several types of tests are run on it. Because of this, it is very difficult and impractical to use production tooling in a CI environment.
We should still have observability in our CI pipeline. And the only way to have that seems like instrumenting our tests and gathering observability data such as metrics, logs, and traces. Contextualizing every error and following the traces down to the root cause looks like the only way to shift the observability paradigm left.
GitLab’s survey showcases developers' thoughts about the delays in the software development process.
Test failures, long-running tests, and lack of visibility in the CI pipeline are common reasons for slow CI processes.
Every code change triggers an automated build-and-test sequence within the CI pipeline and provides feedback to the developer who made the code change.
Ideally, the entire CI feedback loop should run in less than ten minutes.
How can I optimize my slow CI pipeline?
- Performance monitoring: We should have the ability to monitor the performance of tests, such as which tests are running and for how long, in addition to being able to compare test durations with previous performances. We should also know which tests pass or fail, and after how long of a time.
- Finding the bottleneck: We should make sure we understand where bottlenecks are occurring in our slow-running test suites. If we have a microservices application that is being tested, we should know which dependencies are used with test doubles and which are real cloud dependencies, with a clear view of which dependencies’ responses take longer than expected.
Test configuration optimization:
The design of tests is important when it comes to performance because it can lead to unnecessary resource consumption. We should make sure that we are not running clean-up methods redundantly, as this will unnecessarily extend the build time.
We should make sure that we do not run the tested application and then stop it after each test run. This will significantly increase test durations as well as infrastructure costs.
Let’s explore three ways of how to make debugging easier for tests in your CI pipeline.
3 Ways to Make Debugging Your Tests Easier
Instrument your tests to get test analytics
We need observability in order to be able to debug microservices applications. Unlike monoliths, the scope is not a single process in microservices.
Observability requires instrumentation to collect, measure, and analyze signals such as metrics, traces, logs, and more.
In today’s world of automation, OpenTelemetry has become one of the preferred ways to collect data at scale from modern applications which output diverse streams of data.
An application's distributed traces, time-series metrics, and logs can be collected using instrumentation APIs or structured data formats.
There are some basic steps required to implement automatic instrumentation for our test.
Automatic Instrumentation:
- We need to add some language-specific dependencies in order to enable automatic instrumentation.
- We need to configure instrumentation via environment variables. Some configurations may include data source-specific, exporter, propagator, and resource configurations.
Implement distributed tracing to pinpoint the root causes of the failure
When we implement distributed tracing enhanced with additional context, it becomes easy to understand the application’s behavior during test failures, and then debug and maintain operational excellence, especially in microservices applications.
Distributed tracing vs. logging:
When leveraging telemetry, the goal is to have actionable data, whether it’s in the form of traces, metrics, or logs. Logs are usually the first option for monitoring. But as your applications scale, using logs can become expensive to manage, hard to organize, and impossible to aggregate different streams.
The new way of debugging:
Distributed tracing helps us analyze transactions transmitted across multiple networks to reveal information about the communications that occur between processes.
In order to track transactions between applications, instrumentation needs to provide contextual information, such as transaction IDs between processes, in a standardized way.
Let's say we make a remote procedure call request. It should share certain tracing information in meta headers to enable us to determine which trace the call request belongs to.
By using context-sharing on the instrumentation level, we can follow a request across the whole system when we troubleshoot a test failure.
Debug on the CI server rather than trying to replicate everything locally
When a test fails in CI, we get a log dump from the CI automation tool. If we decide to debug the issue, we generally try to reproduce it on our local environment to understand the root cause.
But this is so daunting. So instead, why don’t we just debug on the CI environment?
While debugging is still fairly easy when the environment is in your control on your local environment, it’s not as easy as it is on cloud environments where the infrastructure is managed by cloud vendors. On your local environment, you have to connect to the server where your code is running to debug your service, and that can be grueling, to put it mildly.
Traditional debugging techniques are not ideal in such circumstances because hitting a breakpoint pauses the application and blocks it from progressing into further execution. This therefore prevents the system from being available for other requests, which can in turn stop the CI pipeline and block your colleagues’ ability to push code to the master branch. The best way to perform remote debugging on a cloud environment is to add non-intrusive breakpoints that capture the application’s context (such as variables, stack trace, etc.) when they’re hit, without interrupting the application at all. This provides all the relevant information necessary to understand and fix an issue. In an ideal world, you could even immediately apply a hotfix to verify the impact of the change without having to go through the whole CI/CD process each time.
Bonus Section: How to Debug Your Tests in the CI Pipeline
Thundra has introduced a CI observability tool called "Foresight"
How to Debug Your Tests
- Aggregate tests: We need to understand which tests failed first because we may have run anywhere from a handful up to hundreds of tests. To do this, we need to instrument the tests and the tested applications’ backends in order to get test analytics such as traces, metrics, and logs. At the commit level, test runs and statuses should be aggregated to point out which tests passed, failed, or skipped.
- Distributed tracing: After detecting which tests failed, we should be able to identify which exact service and request made the test fail. While there are a lot of external services and dependencies used in tests, we should still be able to understand whether a unit test or an integration test failed and determine the root cause of the failure without getting lost in log piles.
- Code-level information: We should see contextualized errors of the failure through source code and structured logs and events. The error might be thrown by our service or a dependency, and we need to have per-test code coverage for each individual test run.
- Debug remotely: Since tests and tested applications are instrumented, we need to be able to selectively record test failures. This way, we would be able to debug failures in the environment where the tests actually run rather than trying to replicate everything locally.
What is Shift-left Observability?
Just like the shift-left testing approach, shift-left observability means having full visibility earlier in the software development life cycle.
Why is observability an important issue in testing?
Testing today’s distributed systems, especially when they’re coupled with microservices architectures, is costly and time-consuming.
It’s just like a black box inside the CI server – and nothing sheds light on it except Foresight!
Most developers say: “I don't have any visibility into my tests. I need to know whether the build process that I triggered is successful or not. If it fails, I need to know about it immediately.”
Most development leaders say: “I don't know what's going on in the overall CI process, and I need overall visibility to see how it's going. I want to check the trend, see which tests are slower than usual, and know how often they fail, why they fail, and the overall performance trend over time.”
Tracing requests through different services and understanding the processing they undergo from end-to-end can be difficult, especially when it is hard or even impossible to reproduce the exact environment.
Make your CI pipeline observable for a faster and optimized build process
“There is no tool that we can leverage to quickly understand why my build process fails.”
Unlocking visibility into the CI pipeline by spotting test failures in no time.
Foresight empowers developers and development leaders to build successful CI pipelines by providing deep analytics and debugging capabilities. Foresight is unique with its tracing capabilities for shift-left observability in tests. By providing observability into the CI process, Foresight helps optimize the build duration, enabling more frequent deployments, higher development productivity, and lower CI costs.
With Foresight, we can gain visibility into our builds in the Continuous Integration process that we trigger in order to push our code to production.
Foresight enables us to identify frequently failed and long-running tests, which then guides us in prioritizing the build process to optimize the time spent for tests.
We can easily identify the root cause of test failures and troubleshoot immediately with distributed traces.
Explore how Foresight works with your projects now.
Save the date! 🕊️
On the 28th of September, we are launching Foresight on Product Hunt 🎉🍻
Take your seat before we launch on Product Hunt 💜
Top comments (0)