If you practice continuous integration, you rely on a CI/CD pipeline to build, test, and package your product. That means that the speed of your CI/CD build directly impacts the pace of software development — if your CI/CD pipeline runs faster then productivity will increase.
How fast is fast? In 1999 Kent Beck, in the landmark book Extreme Programming Explained, set the bar to a maximum of 10 minutes. Everything — from building to deployment — should occur in that timeframe.
Is your build up to par? If you need help speeding it up, here are five ways to run your CI/CD build faster.
1. Use more powerful machines
A CI/CD pipeline is like a train: it takes your code and delivers a deployable package. Add horsepower to the train and it will be able to do more work in less time. The same happens when your CI machine has more memory and CPU; by vertically scaling it, you'll find that processor-bound tasks, like building executables, building Docker images, or performing end-to-end tests, run faster.
If your CI/CD is running on-premise, it's up to you to perform hardware upgrades. With cloud-based CI/CD platforms such as Semaphore, choosing a more powerful machine is as easy as changing a setting.
There is a third option available: mix different types of machines on a more fine-grained basis. You can achieve this with self-hosted agents. With self-hosted agents, you can choose which server, either in the cloud or on-premise, runs each part of the pipeline. For instance, you can execute the build on a beefed-up on-premise server and parallelize the tests on many small cloud VMs.
2. Increase parallelization
Another way of improving CI/CD time is by spreading the load among many machines, this is also known as horizontal scaling. Cloud-based platforms like Semaphore can provision machines on-the-fly, so learning how to set up your pipeline to take advantage of parallelization is critical for getting a fast build.
Parallelization is the fastest way of improving your build time. Still, it comes at a premium as more machines mean higher bills. You can manage costs by running your own servers, either on-premise or in the cloud with self-hosted agents. You can even spin up and shut down servers as needed with the Semaphore-maintained AWS autoscaler.
3. Use artifacts and cache
Cache and artifacts are two types of storage features that can dramatically speed up your build time.
The cache is meant as temporary storage. It has three defining characteristics:
- It's free.
- It has a size limit of about 9.6 GB per project.
- It has a 1-month expiration time.
The cache speeds up the build time because you don't have to download files from the Internet on every run. Since old files are automatically deleted as needed, its primary purpose is to keep dependencies such as the node_modules folder or intermediate build files.
Artifacts, on the other hand, are like S3 buckets. They are meant as persistent or semi-persistent storage to keep data and executables for a longer time. While free during the current beta period, artifacts will eventually have a per-usage charge.
There are three categories of artifacts, which allows you to choose an optimal retention policy:
- Job artifacts: used to collect job-specific data such as debug info, logs, and UI screenshots.
- Workflow artifacts: suitable for passing data and builds from one job to another, permitting you to build once and test many times.
- Project artifacts: used for storing final deliverables.
4. Optimize tests
A test suite can easily contain hundreds or thousands of tests, so this is where most of the CI/CD time is spent. The trouble with tests is that adding more power doesn't always help. It's like driving a roadster in a congested city — horsepower doesn’t let you ignore traffic.
The solution is to debug and optimize tests. For that, we must first identify slow or flaky tests, which is easier said than done. The easiest way of finding the bottlenecks in your pipeline is to configure test reports, which gives you aggregated statistics about your jobs across all runs.
Identifying slow jobs is only the first step of the plan, however. To remove them, you should:
- Decide if the test is really needed or not.
- Run a profiler on the job to see what's causing the slowdown.
- Once the problem is identified, optimize the job to remove the bottleneck.
5. Change CI/CD solution
If, despite all your efforts, the builds are still slow, it may not be your fault. It may be that you're using the wrong tool.
There are many CI/CD solutions to choose from, with different features and use cases. Some, like Jenkins, require you to set up, configure, and manage hardware. You are responsible for the reliability of the system and scaling it up as your project grows. While this allows you a great degree of control, you might not have the time or inclination to micromanage everything in some situations.
Other CI/CD solutions, like Semaphore, are cloud-based. There is no management required; you can sign-up and get to work immediately. These solutions are the fastest and most flexible, as they can scale up automatically as needed.
Keep your builds fast
Speeding up your CI/CD build is never wasted effort. On the contrary, every second shaved from the pipeline is compounded by how often you commit code into the repository, which translates into many productive hours reclaimed over a typical week.
Keeping your CI/CD pipeline well-oiled and fast will ensure that your development team reaches its maximum potential.
Top comments (0)