Importance of CI/CD and parallelisation in test automation

Oryginally posted at: https://kodziak.com/blog/importance-of-ci-cd-and-parallelisation-in-test-automation

Nowadays, products change very often. We're create new versions of applications on daily basis. That means we need to know fast if everything works. To achieve this, we could run tests manually before every release or...?

Let's automate this process.

How can CI/CD help with test automation?

Let's be honest. If we tell the developers "you need to run the tests manually", they probably won't do it too often. Why? Because they are time-consuming. The execution usually takes more than an hour. Maybe it will work a few times, but I'm pretty sure there will always be some kind of excuse. But that's life.

Then we're coming to the problem that we don't want to merge untested code, ever. We want to run unit, integration, and end-to-end tests (of course, if we have all of them:)). In a convenient way of course. Without any extra work from the user.

The worst we can do, is to make something dependant on us. Continuous processes require knowledge from the developers how to maintain the tests and that's very close to writing them. When we all work on one product, we're all responsible for its quality. And to achieve that, we should work closely, together. Who said that only testers could write E2E tests? Anyone from the development team can do it.

Last but not least, in continuous processes we can block the deployment when a test suite fails. Then the author needs to fix the feature or maintain the tests, if changes affect them. Of course, deterministic tests are needed here, which we all know - is hard to reach.

Loooong time execution...

Yeah... The problem we all know. Many times I've heard that tests takes one or two hours to finish.

We should avoid that.

Try to stick with 10-20 minutes to execute the core ones. To reach that, we can use parallelisation.

We can distinguish two types:

• run tests by groups,
• run multiple tests at a time.

Tests by groups

First, we should split our tests by groups/criticality. Recently I've written an article about that: What is smoke, sanity, regression testing? How to CI/CD with them. I've split the tests by types. And with that, we could run the groups sequentially.

For example, yours pipeline could looks like this:

An approach like that, makes sure that the most important user flows are always gonna work. Then, on production environment we could check these less important ones. That's fine, until we're aware that something could be broken.

You'll secure our time-to-market, while still checking everything you're supposed to.

Multiple tests at a time

Secondly, use parallel nodes. Basically, most of the tools are configured to run one test and then another. With a strong machine and properly configured tests you can run a few of them at a time.

One of the out of the box tools which solves this problem, and makes the execution faster is Knapsack Pro. A very good tool to get the best execution time you can imagine.

Spread knowledge about tests

But what if a test suite fails? What happens then?

There are two solutions:

• you can always monitor what happened and try to find a solution,
• you can make it easier for people to get the information about what happened,

And I really love the second option.

I think everyone who works on software should be able to write, maintain, and fix the tests. You're working together to deliver the best product quality you can. That means you should avoid the stereotypical division between testers and developers. You're working together, as a development team.

You can also implement automatically generated reports. Showing people, business, and other stakeholders what the tests cover, builds trust. And the trust in tests is what you need.

Conclusion

You should deliver the best product quality you can in reasonable time. It's good when you can do it fast and with fine user experience (our internal users). People won't run tests manually, so you should automate it. That's why you could use continuous processes.

Combining it with parallelisation, you'll secure the time-to-market.

Spreading knowledge about tests makes more people feel responsible for them. If something fails - someone from the team will need to fix or maintain it. The more people work on tests, the better these tests are.