At Woovi, we prioritize both quality and speed in our software development processes, as we've previously discussed in our article on software quality. This commitment entails running over 100,000 tests daily and deploying multiple times to production without disruption. Achieving this level of efficiency requires substantial investment in CI/CD infrastructure. In the past, we relied on CircleCI as our preferred platform, which served us well but came with significant costs.
In this article, we'll provide an overview of our journey toward building our own CI/CD platform. This platform will not only enhance our development experience but also result in a remarkable 87% reduction in CI/CD costs within the first 30 days of implementation, compared to our previous setup with CircleCI. We're excited to share our progress and insights with you.
Requirements
The requirements for our CI/CD pipelines quite simple but not easy to achieve:
GitOps
It must follow gitOps culture, all infrastructure must be described in a git repository, that way we can have auditable changes and updates not only for our software but for infrastructrure aswell.
Automation
Changes in software must trigger CI/CD pipeline automatically, without human intervention.
Conditional Testing
Efficient resource allocation is crucial. Therefore, only tests relevant to the modified portion of the software should be executed. This minimizes unnecessary resource consumption and time spent.
Full Suite Testing
Must be able to run full test suites, our monorepos have 7-10k tests each. We use jest as our test framework, that means we have to optimize for it.
All self hosted
No cloud services, no paid platforms.
Cloud Native
Taking advantage of the k8s api and operators, we can use the controll loop to manage our CI/CD applications. Tekton is a very powerfull cloud native tool we will be using for this project.
Design
The CI/CD flow components are:
Events
The events are generated from github, each push of new code a developer performs have to trigger a test pipeline. Deploy requests are created from a pull request with the release tags.
Pipelines
One pipeline for each repository, the pipeline will run tests, build applications, update registry and bucket with the new version of software if the tests pass and deploy in the correct enviroment based on the event who triggered the pipeline.
Applications
Pipelines will interact with applications directly, deploying new versions, performing rollbacks if something goes wrong and sending notifications about overall status.
Tekton and ArgoCD
Tekton is a very powerfull cloud native CI/CD framework, is open source and has very strong comunity support. We choose mainly because it has all we need in terms of features, with some still in development, and the comunity is very active and helpfull.
Argocd is a declarative GitOps tool, it allow us to implement track our helm charts from repository changes. We also implement the deploy stage of our production enviroment using argocd apis, that from a task in tekton can sync applications in kubernetes after we update our registry. Its also very complete with many security features out of the box.
We have a dedicated server for our staging enviroment that runs proxmox hypervisor, there we run our kubernetes cluster in a VM. We will do a deep dive on upcoming articles for each building block and hope to show a little of how we setup the platform.
Initial Impact
We saw a whopping reduction of R$40,000.00 in monthly CI/CD costs!
The primary expense in our CI/CD setup lies in the development pipelines. In our initial implementation, we opted to retain the production pipelines while migrating only the development ones. The outcome? An impressive 87% reduction in costs.
Not all impact was positive, our pipelines are taking 2x the time run as off the last implementation, we have a lot to develop in order to match circleci features. Performance too took a big hit, turns out running so many tests using jest cost alot of resources. Our staging server when running full test pipelines is always on 95% capacity.
Next article we will be covering, the steps involved in setting up Tekton and Argo CD within Woovi's Kubernetes cluster or any k8s cluster.
In the meantime, we've just published our tools and manifests on GitHub WooviOps. Feel free to contribute, and let us know if we can improve anything.
Woovi
Woovi is a Startup that enables shoppers to pay as they like. To make this possible, Woovi provides instant payment solutions for merchants to accept orders.
If you want to work with us, we are hiring!
Photo by Erlend Ekseth on Unsplash
Top comments (0)