Azure, CircleCI, GitHub, Gitlab: a POC

A limitation in GitHub Actions, despite being completely workable, made us question whether it would be better to change our CI tool.

The tech lead was very experienced in Microsoft tools, so he suggested Azure DevOps. I also like Azure a lot, but I have* a crush on Gitlab. The team's "DevOps"** had experience with CircleCI and suggested it.

In order not to have an unecessary discussion and not decide based only on documentation (we did this once and I don't know if it was the best choice, but it working very well), I offered to do a POC (Proof of Concept) with the CIs mentioned.

As we were already running the website build and tests on GitHub Actions, it was just "translating" GitHub's yml to the yml of other CIs. Said no one ever…

*had. I explain below

**Just to give another name than developer.

CircleCI

I started with CircleCI because one of the things we talked about was to keep the code on GitHub. That way we wouldn't need to do migrations, although that's pretty easy.

From the documentation, I was delighted with CircleCI. The "Orbs", which are like GitHub Actions, the possibility to choose machines with up to 128 GB of RAM, the possibility to choose graphics acceleration. I almost closed the deal with CircleCI here. But then came the construction of the pipeline…

My experience with CircleCI is only from studies. So my pipelines in CircleCI are pretty simple.

When I translated the GitHub Action to CircleCI yaml, it was the CI that took me the longest time to start. It took me a long time to understand the structure of the workflow, but that might have been a limitation of myself.

One point I didn't like were the examples in the documentation. They only show simple structures. To do what I had in mind, I had to search the internet. I wasted a lot of time.

Another setback is that many of CircleCI's (all?) integrations need permission from the company, which is annoying and can delay development.

But what was knockout for me on CircleCI was the difficulty of running a localhost on it. I searched on the internet and saw that this is really a problem. There were other solutions, but I found it very complicated for something that is straightforward in other CIs.

CircleCI Pros*

• Caching dependencies: If your project uses Yarn, Bundler, or Pip, for example, the dependencies downloaded during a job can be cached for later use rather than being re-downloaded on every build. The other CIs have it as well.
• Docker Layer Caching (paid): save Docker image layers created within your jobs rather than rebuilding the entire image every time, saving building time. Has no effect on Docker images used as build containers. Is only useful when creating your own Docker image
• Machines up to 128GB RAM (Windows), versus 7GB on GitHub. You can choose GPUs as well
• Orbs vc Actions: orbs are just packaged, reusable YAML. Actions are written for execution inside a Docker container or coded as individual steps using JavaScript. So anyone can code Orbs, not just JS devs.
• Can run jobs locally using the CircleCI Local CLI
• Rerun from start or from failed job
• A YAML validator, on CI and locally with CLI

Cons

• Need authorizations to connect with GitHub and to do some other tasks
• Need to config the virtual environment (this could be an advantage). GitHub Actions provides a "ready-to-go” operational system.
• Some issues running a localhost server. I considered it an eliminatory issue
• No built-in way to continue after failing tests, but display the test failed.

My opinion

• YAML a little complicated to configure.
• Lots of permissions to ask
• Doc examples too simple. Had to google to find more complex examples

*The pros and cons were based in features that were useful at that time

Azure DevOps

Azure DevOps was the first CI I actually used, I mean, it was the first one that I built a pipeline from scratch (after Jenkins on a Mac Mini). This was possible because it has excellent documentation, with several examples.

I was able to do everything I wanted on it, so it quickly became one of my favourites.

A big advantage of Azure over the others (I don't remember if I needed to do this in Gitlab), is clone other repositories within the same domain into your pipeline. It was just a git clone like you do on your computer. On GitHub, for example, you need a PAT (Personal Access Token) and use their action/checkout.

Azure native tool for test reporting, to me, is the prettiest of all these CIs.

Another very positive point for Azure DevOps is the price. Free for teams smaller than 5 people and $6 per person for larger teams. Currently, it is necessary to ask for an authorization to use the pipeline, which can take a few days to be approved.

Azure DevOps Pros

• Node JS: able to run --only=dev or --only=prod to speed up the dependencies install (don’t know in other package managers)
• Good documentation
• Easy to checkout other repositories from the same company
• Rerun failed jobs
• As GitHub, have a “whole” OS image, like ubuntu-latest not just tools like node:latest
• Allows caching
• Pricing

Cons

• Just one YAML per pipeline. You can have more than one, but need to specify manually which one to run.

Gitlab CI

Gitlab has always been my favourite tool. I was enchanted with it from the first time I used it. I don't remember if at that time other tools had the same functionalities as Gitlab, but my first use and the team's choice to use Gitlab was because it already had a built-in Wiki.

Other native Gitlab tools I used were Gitlab Pages and Gitlab Runner, meaning Gitlab always had everything I needed. Also, it saves the files generated by the pipelines for quite long time, so I didn't need to add a step in yaml to save the test results somewhere else to have a test history.

Not to mention that Gitlab also has excellent documentation, and since I already had experience with Azure DevOps, it was even easier to understand.

The only bad thing about Gitlab, for me, is that it doesn't have the separation of jobs like other CIs. The entire log is on a single terminal. If your pipeline is big, it becomes a little confusing to find a broken test, for example.

But what was a knockout for Gitlab was the site's build time. While we other CIs stayed in at most 20 seconds, in Gitlab it was a minute! And I have no idea why…

Gitlab CI Pros

• Static Application Security Testing (SAST - Free): Analyze your source code for known security vulnerabilities.
• Some built in tests, like SAST mentioned above
• Gitlab Pages (better than GitHub, to me)
• Good documentation
• Allow cache between pipelines and jobs. Specify a list of files and directories paths that are in the local working copy, e.g. node_modules

Cons

• Only "raw images" like CircleCI
• Too slow to build the website
• No visual division between jobs

GitHub Actions (GHA)

Finally, I'm going to talk about the tool we were already using.

One of the things I mhttps://github.com/actions/virtual-environmentsost liked about GHA is that it has full operational systems, with several tools already installed. This practically makes me program the pipeline as if I were on my own machine, with no additional configuration.

The Actions, most of them, are very well documented and simple to use, so there is no need to search the internet for examples, only in specific cases or if you want to see other examples of use.

An advantage of GitHub over other CIs is that it allows you to have many different workflow files. This is great for organizing workflows. For example, you can have a file just to run the tests, another one just to deploy in staging, another one to deploy in production etc. Triggers are set up on each file and fired automatically if the condition is met. A disavantage is to call a workflow inside another workflow. Is not straightfoward and a little hard to do.

Azure DevOps also allows you to have multiple yaml files, but you have to define only one for each pipeline. I didn't find this possibility in other CIs.

Another thing that makes GitHub stand out are its APIs, Rest and GraphQL, and its CLI (Command Line Interface), which allow you to do almost everything from the command line and, consequently, automate it all!

Summary

• CircleCI was the one that took me more time to start the job. I liked the separation of jobs and workflows, but generated too many errors before the first green pipeline.
• Azure DevOps (together with GitHub Actions) was the most straightforward to begin
• Gitlab I had to do some workarounds to deal with the lag of website build, because the pipeline broke due to the delay. One disadvantage is that the job stays just in one terminal. Could be confusing to follow with large jobs. Besides, it was the slowest one to build the website.

The decision

In the specific case of this POC, the best ones were GitHub Actions that we were already using, together with Azure DevOps. As we stopped using the process that was causing the limitation on GitHub, we continued using it.

If you want to take a look at the configuration ymls, go to my repository. Remember that it was just a POC and the ymls are very simple, but they can be a start point.

Do you disagree with any point in this text or have other experience with one of these tools? Comment here!