Save precious build time with custom Docker registry

At our company we have a relatively new product in development. On the DevOps side, we have a pretty simple workflow for the testing stage. It involves:

1. Git repository triggers Jenkins build.
2. Latest pull is taken from the repository
3. Frontend is built using npm
4. Backend Docker image is built (which contains frontend files)
5. The image is pushed to Dockerhub
6. Using Ansible, we run a script in the target server.

On the target server (on premises):

1. Pull the latest image.
2. Restart all related containers using Docker Compose.

This is a pretty simple CI/CD system. Nothing too complex. Our codebase is consistently growing and each build took around 20 minutes. We have around 5 builds per day on average with 6 devs working on the project. So, we realised it's high time we optimised the build system a bit to handle more builds per day and reduce time for building priority images.

So, we looked into our Jenkins' build log and found out that pushing and pulling from Dockerhub took the most time (around 10 minutes). Since our test servers are on premises, we decided that having a custom registry (available as a Docker image) is the best solution to save time. So I looked into the Registry documentation (which is very well written 😀) and deployed it on one of the test servers. Then, I tweaked the build script a little bit to accommodate for the changes and Voila! The build time reduced to 12 minutes flat!!

One of by-products of this effort was reduced image size. I noticed that I was adding node modules and git folders (newbie mistake 😛) into the image which were bloating the image resulting in unnecessary pushes and pulls of the layers of the image. The image size reduced to 5GB from 8GB. This decreased the build time to 10 minutes flat.

Some lessons learned:

1. Sometimes cloud is too expensive in terms of time (network latency 😞 ).
2. Always inspect your image sizes and keep them as low as possible.
3. In certain situations, managing your own services instead of cloud is not a bad solution.
4. Monitor builds periodically for rapidly changing codebases for redundancies which can be potentially delaying your builds.