• Initial thoughts
• 1. Use a local docker image in early stages of job creation
• 2. Use the pipeline editor for minor YAML changes
• 3. Use gitlab-ci-local to run pipelines locally
  • Validate pipeline YAML
  • Run a full pipeline
  • Run just a job
  • Handle UI variables
• 4. Declare test branches to simulate long-lived branches and tags
• Wrapping up
• Further reading

Initial thoughts

As developers and CICD engineers, we are all too familiar with the time-consuming process of iterating on GitLab CI YAML modifications. It can be a frustrating cycle that involves multiple steps and setbacks:

• Make a change to the YAML file.
• Commit and push the changes.
• Verify that the CI YAML is valid and the graph is as expected.
• If there are any issues or errors, start the process all over again with a sense of despair.
• Wait for the entire pipeline to finish running on the current branch.
• If the pipeline fails or doesn't behave as desired, start over again with a heavy heart.
• Refactor commits to maintain a clean git graph and push with force.
• Merge multiple times to observe the behavior on long-lived branches and tags.
• If there are any issues with the merged code, start the process all over again with a sigh.

What we initially thought would only take a few minutes ends up consuming multiple days as we continuously go back and forth, all while trying to juggle other development tasks.

In this article, we aim to alleviate this pain by providing several tips that will make your life easier. Our goal is to transform the experience of tweaking your favorite pipelines from a tedious chore to an enjoyable task. So let's dive in and discover how you can streamline your GitLab CI YAML modifications!

1. Use a local docker image in early stages of job creation

When starting a new job with a new Docker image, it's natural to feel uncertain. Especially when using initially unknown public images, following the best practice Start CI with versioned public CI docker images. To accelerate the feedback loop and gain more confidence, we can try our commands directly in a local container.

Let's consider a NodeJS job as an example. The DockerHub image alpine/k8s seems to be the most suitable base image.

Before diving into the GitLab CI YAML, let's start an interactive container:

docker run -it --privileged=true node:20.5-alpine3.17 /bin/sh
Status: Downloaded newer image for node:20.5-alpine3.17
/#

You may wonder what shell GitLab is using in containers. In fact, it is a discovery process starting with bash, so best guess is to try /bin/bash and fallback to /bin/sh.

Now that you're inside the image, you can easily prepare and test your commands.

Although the base is alpine, let's assume we do not have that knowledge and want to use envsubst. If it's not available, we can try to install it with apt-get update && apt-get install envsubst:

/# envsubst --help
/bin/sh: envsubst: not found
/# apt-get update && apt-get install envsubst
/bin/sh: apt-get: not found

Since envsubst and apt-get are not available, it must be the alpine Linux package manager, apk:

/# apk --update add --no-cache envsubst
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/aarch64/APKINDEX.tar.gz
ERROR: unable to select packages:
  envsubst (no such package):
    required by: world[envsubst]

Ah! It seems that envsubst cannot be installed this way. A quick internet search reveals that the package is actually named gettext:

/apps# apk --update add --no-cache gettext
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/aarch64/APKINDEX.tar.gz
(1/9) Installing libgomp (12.2.1_git20220924-r4)
(8/9) Installing libxml2 (2.10.4-r0)
(9/9) Installing gettext (0.21.1-r1)
Executing busybox-1.35.0-r29.trigger
OK: 18 MiB in 26 packages
/#
/# envsubst --help
Usage: envsubst [OPTION] [SHELL-FORMAT]

Substitutes the values of environment variables.

By going through this discovery process within a few minutes, we saved ourselves from waiting for the pipeline to fail, which could have taken several hours for a job, especially positioned in last stages.

2. Use the pipeline editor for minor YAML changes

The GitLab CI/CD Pipeline Editor is a powerful web-based tool that provides a visual representation of your pipeline configuration. Accessible from the GitLab UI, the Pipeline Editor allows you to interactively modify your .gitlab-ci.yml file directly in the browser.

The Pipeline Editor offers several advantages. Before even commiting changes, it provides real-time feedback on the validity of your YAML syntax and helps prevent common errors. Additionally, it assists in exploring available keywords and includes inline documentation for GitLab CI/CD features. This visual approach can significantly streamline the process of making and validating changes to your pipeline configuration, reducing the need for multiple iterations.

The main limitation is that it can only edit the main .gitlab-ci.yml file.

3. Use gitlab-ci-local to run pipelines locally

Imagine being able to run your pipelines locally without the need to push, wait, or pollute the GitLab instance. Even better, what if you could achieve this without even interacting with the GitLab instance at all?

Initially, the gitlab-runner exec command was intended to fulfill this purpose. However, over time, the code has diverged too much from an actual pipeline running, rendering it capable of handling only 20% of the keywords. As a result, it is no longer usable. There is an official issue on this matter, but progress has been minimal.

Fortunately, the open-source community has made remarkable efforts to implement a local runner, resulting in several options:

sudo wget -O /etc/apt/sources.list.d/gitlab-ci-local.sources https://gitlab-ci-local-ppa.firecow.dk/gitlab-ci-local.sources
sudo apt-get update
sudo apt-get install gitlab-ci-local

curl -s "https://gitlab-ci-local-ppa.firecow.dk/pubkey.gpg" | sudo apt-key add -
echo "deb https://gitlab-ci-local-ppa.firecow.dk ./" | sudo tee /etc/apt/sources.list.d/gitlab-ci-local.list
# OR

# MUST be `.asc` at least for older apts (e.g.

yarn global add glci

glci

glci --only-jobs=install,test:e2e

RadianDevCore / Tools / gcil · GitLab

Launch .gitlab-ci.yml jobs locally - https://radiandevcore.gitlab.io/tools/gcil

gitlab.com

cunity / gitlab-emulator · GitLab

Run most gitlab jobs locally before pushing to your gitlab server. Ideal to validate your yaml changes or test simple builds out locally.

gitlab.com

Among these options, the one that has gained the most traction is gitlab-ci-local:

Here are some key features of gitlab-ci-local:

• The only prerequisite is having functional Docker commands.
• You can trigger the execution of the entire pipeline, a single job, or the fastest path to a job using the needs: chain.
• No need to push changes or establish a connection to the server. Committing is also not required.
• It supports 99% of the keywords used in GitLab CI pipelines and continuously closes the gap. This means that even complex real-life pipelines with artifacts and lightning-fast cache can be executed seamlessly.
• YAML from includes are seamlessly downloaded
• GitLab CI predefined variables are populated with valid values.
• Global, group, and project variables can be managed using files.
• It supports running unlimited jobs in parallel.

Validate pipeline YAML

To validate your pipeline, simple try to display the jobs list:

$ gitlab-ci-local --list

parsing and downloads finished in 178 ms
name                                   description  stage              when        allow_failure  needs
📦✅-webapp-package-and-test                        📦 package ✅ test  on_success  false
📦✅-backend-package-and-test                       📦 package ✅ test  on_success  false
🐳🧪-webapp-build-push-dev                          🐳 build-push      on_success  false
🐳🧪-backend-build-push-dev                         🐳 build-push      on_success  false
🗑✨-delete-namespace-mr                            ☸ deploy           manual      true
☸🧪-webapp-deploy-dev                               ☸ deploy           on_success  false
☸🧪-backend-deploy-dev                              ☸ deploy           on_success  false
✔-e2e-tests                                         ✔ e2e tests       on_success  false

It helps test your rules. For this, you can also force some predefined variables, to test alternative branching situations.

Run a full pipeline

By default gitlab-ci-local will run the full pipeline for the current branch, or for the current merge request, depending of the predefined variables you overrode. It will trigger and run jobs in parallel, as a GitLab runner would do:

$ gitlab-ci-local

Run just a job

We can also run a single job:

$ gitlab-ci-local 📦✅-webapp-package-and-test

Or a job and its dependencies before it (needs: chain and previous stages):

$ gitlab-ci-local 🐳🧪-webapp-build-push-dev --needs
📦✅-webapp-package-and-test           starting node:18.11-alpine (📦 package ✅ test)
📦✅-webapp-package-and-test           copied to docker volumes in 7.72 s
📦✅-webapp-package-and-test           mounting cache for path projects/webapp/node_modules
📦✅-webapp-package-and-test           mounting cache for path projects/webapp/.next/cache
📦✅-webapp-package-and-test           $ cd projects/$MODULE
📦✅-webapp-package-and-test           $ if [ ! -d node_modules/.bin ]; then yarn install; fi
[...]

Handle UI variables

Global, group and project variables can be handled in $HOME/.gitlab-ci-local/variables.yml, or locally in each project.

Official example:

$HOME/.gitlab-ci-local/variables.yml

project:
  gitlab.com/test-group/test-project.git:
    # Will be type Variable and only available if remote is exact match
    AUTHORIZATION_PASSWORD: djwqiod910321
  gitlab.com:project/test-group/test-project.git: # another syntax
    AUTHORIZATION_PASSWORD: djwqiod910321

group:
  gitlab.com/test-group/:
    # Will be type Variable and only available for remotes that include group named 'test-group'
    DOCKER_LOGIN_PASSWORD: dij3213n123n12in3

global:
  # Will be type File, because value is a file path
  KNOWN_HOSTS: "~/.ssh/known_hosts"
  DEPLOY_ENV_SPECIFIC:
    type: variable # Optional and defaults to variable
    values:
      "*production*": "Im production only value"
      "staging": "Im staging only value"
  FILE_CONTENT_IN_VALUES:
    type: file
    values:
      "*": |
        Im staging only value
        I'm great for certs n' stuff

4. Declare test branches to simulate long-lived branches and tags

Now that we have the tools to test our pipeline changes with confidence before pushing them, there is still one tricky aspect to consider: how will our pipeline behave on long-lived branches after the merge request is closed?

While we can use gitlab-ci-local locally to force certain variables, such as setting $CI_COMMIT_BRANCH to main in $HOME/.gitlab-ci-local/variables.yml, this approach is only applicable locally and does not serve as valid proof for people reviewing our merge request.

A satisfying solution is to always declare special branch/tag cases and include a corresponding test version. This means that our jobs will run on both main and main-test, on tags and tags-test branch, for example:

deploy-module-staging:
  extends:
    - .staging
    - .deploy-module
  rules:
    - if: $CI_COMMIT_TAG || $CI_COMMIT_BRANCH == "tag-test"

By doing this, we can test all branches conveniently from our current merge request without actually merging anything. We can create these test branches from our latest commit in the GitLab UI. Once we are satisfied with the pipeline execution (or just the composition of pipeline jobs for sensitive tasks), we can stop the associated pipeline and delete the test branch. If we are not satisfied, we can simply add more commits to our merge request and recreate (or rebase) our test branch(es). When everything seems fine in all scenarios, we can confidently proceed with merging our merge request, knowing that it won't cause any unexpected issues.

Wrapping up

By following these practices and leveraging the available tools, you can significantly reduce the time spent on iterating and testing GitLab CI pipelines. This will lead to a more efficient and enjoyable pipeline development process, ultimately improving your overall development workflow.

So go ahead and implement these tips in your GitLab CI pipeline development, and experience the benefits of faster iterations and more reliable pipelines. Happy coding!

Did this pieces of advice help you ? Do you have others on the same subject ? Please share in the comments below 🤓.

Illustrations generated locally by Automatic1111 using ZavyComics model with DieselPunkAI LoRA

Further reading

🔀 Efficient Git Workflow for Web Apps: Advancing Progressively from Scratch to Thriving

Benoit COUETIL 💫 for Zenika ・ Oct 10

#git #webdev #devops #cicd

🔀🦊 GitLab: Forget GitKraken, Here are the Only Git Commands You Need

Benoit COUETIL 💫 for Zenika ・ Aug 31

#gitlab #git #devops #gitflow

🦊 GitLab: A Python Script Displaying Latest Pipelines in a Group's Projects

Benoit COUETIL 💫 for Zenika ・ Jun 29

#gitlab #devops #python #productivity

🦊 GitLab: A Python Script Calculating DORA Metrics

Benoit COUETIL 💫 for Zenika ・ Apr 5

#gitlab #devops #accelerate #dora

🦊 GitLab CI: The Majestic Single Server Runner

Benoit COUETIL 💫 for Zenika ・ Jan 27

#gitlab #devops #pipeline #cicd

🦊 GitLab CI Optimization: 15+ Tips for Faster Pipelines

Benoit COUETIL 💫 for Zenika ・ Nov 6 '23

#gitlab #devops #pipeline #cicd

🦊 GitLab CI: 10+ Best Practices to Avoid Widespread Anti-Patterns

Benoit COUETIL 💫 for Zenika ・ Sep 25 '23

#gitlab #devops #pipeline #cicd

🦊 GitLab Pages per Branch: The No-Compromise Hack to Serve Preview Pages

Benoit COUETIL 💫 for Zenika ・ Aug 1 '23

#gitlab #hack #devops #cicd

🦊 ChatGPT, If You Please, Make Me a GitLab Jobs YAML Attributes Sorter

Benoit COUETIL 💫 for Zenika ・ Mar 30 '23

#chatgpt #gitlab #ai #python

🦊 GitLab Runners Topologies: Pros and Cons

Benoit COUETIL 💫 for Zenika ・ Feb 7 '23

#gitlab #devops #kubernetes #aws

This article was enhanced with the assistance of an AI language model to ensure clarity and accuracy in the content, as English is not my native language.