GitLab: Components, Architecture, Infrastructure, and Launching from the Helm Chart in Minikube

As GitLab recently changed the policy of providing free access, and now only 5 users will be available for the Free subscription, we decided to move to the self-hosted version.

In general, they have interesting terms with the license: the price depends on the number of users, you can buy it for at least one year, and after the purchase, you cannot reduce the number of users in the license (but you can increase it).

Our GitLab will live in Kubernetes, and there are many questions before launch, especially since I personally did not use GitLab very much before.

We will deploy GitLab through ArgoCD, will run it in AWS Elastic Kubernetes Service, and will use AWS S3 for the object store. But more on that later, first let’s see what GitLab is “from the inside” and how to deploy it in general.

GitLab Operator vs GitLab Helm chart

First — GitLab Operator or GitLab Helm chart?

I looked at the possibilities and didn’t see much difference between the Operator and the chart, moreover, the Operator uses the same Helm chart under the hood.

For backup and recovery capabilities, similar utilities are used and the process is the same, see General backup and restore guidance.

But with the version update process of the GitLab instance, the documentation of the Operator looks simpler. See Upgrade the GitLab chart vs. How the Operator handles GitLab upgrades.

And even Gitlab.com itself is deployed with their chart :

The largest known GitLab instance is on GitLab.com, which is deployed using our official GitLab Helm chart

But in the documentation for the Operator, the “Experimental” and “Beta” words are met too often, so for now probably let’s run without it.

In any case, everything is tied to the chart, so we will get acquainted mainly with it and its parameters.

GitLab architecture and components

For reference:

• GitLab Cloud architecture: Production Architecture
• GitLab components and their interaction: GitLab architecture overview

A simplified scheme from the documentation:

Here:

• NGINX: accepting incoming connections
• GitLab Workhorse: a reverse proxy for file uploads and downloads, Git push/pull, etc.
• Puma: A Ruby web server used by GitLab for its API and web pages
• Sidekiq: for creating and managing job queues
• uses Redis to store job information
• PostgreSQL: storage of information about users, access rights, metadata, etc.
• GitLab Shell: works with repositories over SSH and manages access keys
• calls Gitaly to process Git objects
• sends information to Redis to create jobs in Sidekiq
• Gitaly: Git RPC server, handles tasks in repositories received from GitLab Shell and GitLab web applications

Infrastructure

GitLab requires PostgreSQL, Redis, AWS S3 buckets, and mail service to receive and send emails.

Helm chart installs PostgreSQL and Redis by default, but for production, PostgreSQL is recommended to be installed separately, see Configure the GitLab chart with an external database and Configure PostgreSQL, although we will use the PostgreSQL Operator and run a PostgreSQL cluster in Kubernetes.

Similar requirements for Redis and Gitaly — it is also desirable to run them off the chart and not in a Kubernetes cluster. See Installing GitLab by using Helm. Instead of Redis, we will most likely use KeyDB, also through the Operator, also in Kubernetes.

The Gitaly deployment documentation says that for up to 500 users, a single virtual machine with Gitaly itself is sufficient. If 1000 or more users are planned, it is recommended to run Gitaly Cluster with Praefect. See Gitaly > High Availability. Considering the number of users we have, I don’t see the point of moving it to a separate EC2, so we will deploy it together with the chart, then we will look at its work and resources, and maybe will move it to a dedicated Kubernetes node.

Minio is installed in the chart to work with the object store, but again an external service such as AWS S3 is recommended for production. See Configure the GitLab chart with an external object storage.

Since we are on AWS and using AWS ALB Controller, it makes sense to disable NGINX Controller, see Customize the GitLab Ingress options.

There is a good example of infrastructure in the GitLab on AWS Partner Solution Deployment Guide, but this is for really big projects, and we will do it more simply. However, the scheme itself is interesting and useful for understanding the general concept of infrastructure planning:

In addition, there are Reference architectures, which describe options for running GitLab under different workloads. From them, we may be particularly interested in the Cloud native hybrid, which describes running in Kubernetes (hybrid — because it is recommended to run some services out of a Kubernetes cluster):

For LoadBalancer, the least outstanding requests are recommended instead of the standard round-robin — you must not forget.

It is useful to go through all the available options, see what and how you can configure, see GitLab Helm chart deployment options, we will deal with this in the next post.

GitLab monitoring is a separate topic, we will come to it later, for now, see Monitoring GitLab .

Running GitLab in Minikube

Generally used by GitLab’s developers who are working on features for Kubernetes, but we use it to familiarize ourselves with the GitLab chart. See Developing for Kubernetes with minikube.

Let’s start Minikube:

$ minikube start — cpus 4 — memory 10240

Enable the Ingress plugin:

$ minikube addons enable ingress

Clone the repository and install the dependencies — it is useful to look at them, although they are described in the documentation of the chart:

$ git clone https://gitlab.com/gitlab-org/charts/gitlab.git
$ cd gitlab
$ helm dependency update
…
Dependency gitlab did not declare a repository. Assuming it exists in the charts directory
Dependency certmanager-issuer did not declare a repository. Assuming it exists in the charts directory
Dependency minio did not declare a repository. Assuming it exists in the charts directory
Dependency registry did not declare a repository. Assuming it exists in the charts directory
Downloading cert-manager from repo https://charts.jetstack.io/
Downloading prometheus from repo https://prometheus-community.github.io/helm-charts
Downloading postgresql from repo https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
Downloading gitlab-runner from repo https://charts.gitlab.io/
Downloading grafana from repo https://grafana.github.io/helm-charts
Downloading redis from repo https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
Dependency nginx-ingress did not declare a repository. Assuming it exists in the charts directory
…

Check the Minicube IP:

$ minikube ip
192.168.49.2

And install with the values-minikube.yaml:

$ helm upgrade --install gitlab . --timeout 600s -f https://gitlab.com/gitlab-org/charts/gitlab/raw/master/examples/values-minikube.yaml --set global.hosts.domain=$(minikube ip).nip.io --set global.hosts.externalIP=$(minikube ip)

Let’s check the pods — there are a lot of things here:

$ kubectl get pod
NAME READY STATUS RESTARTS AGE
gitlab-gitaly-0 1/1 Running 0 11m
gitlab-gitlab-exporter-5c8dbdc954-hr7jj 1/1 Running 0 11m
gitlab-gitlab-runner-7c4488ff58-bg8f5 0/1 Running 3 (39s ago) 8m30s
gitlab-gitlab-shell-7f9f5bb9ff-qlpxr 1/1 Running 0 11m
gitlab-gitlab-shell-7f9f5bb9ff-wc9rx 1/1 Running 0 11m
gitlab-kas-84cb5c548b-jbp69 1/1 Running 0 11m
gitlab-kas-84cb5c548b-jxqqr 1/1 Running 0 11m
gitlab-migrations-2-vw5jd 0/1 Completed 0 8m30s
gitlab-minio-74467697bb-z8nms 1/1 Running 0 11m
gitlab-minio-create-buckets-2-b6zl6 0/1 Completed 0 8m30s
gitlab-postgresql-0 2/2 Running 0 11m
gitlab-prometheus-server-6bf4fffc55–6xpfm 2/2 Running 0 11m
gitlab-redis-master-0 2/2 Running 0 11m
gitlab-registry-cd64f65dc-frsmt 1/1 Running 0 8m30s
gitlab-registry-cd64f65dc-nvfv2 1/1 Running 0 8m10s
gitlab-sidekiq-all-in-1-v2–59ccd7d6b9–9mpmz 1/1 Running 0 8m30s
gitlab-toolbox-6586c478f5-ktj5x 1/1 Running 0 7m58s
gitlab-webservice-default-6787f4b5db-kfp9h 2/2 Running 0 6m55s
gitlab-webservice-default-6787f4b5db-q62f9 2/2 Running 0 8m30s

Services:

$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
gitlab-gitaly ClusterIP None <none> 8075/TCP,9236/TCP 11m
gitlab-gitlab-exporter ClusterIP 10.110.108.219 <none> 9168/TCP 11m
gitlab-gitlab-shell NodePort 10.110.231.31 <none> 32022:32022/TCP 11m
gitlab-kas ClusterIP 10.102.124.129 <none> 8150/TCP,8153/TCP,8154/TCP,8151/TCP 11m
gitlab-minio-svc ClusterIP 10.99.213.94 <none> 9000/TCP 11m
gitlab-postgresql ClusterIP 10.102.38.18 <none> 5432/TCP 11m
gitlab-postgresql-headless ClusterIP None <none> 5432/TCP 11m
gitlab-postgresql-metrics ClusterIP 10.107.123.35 <none> 9187/TCP 11m
gitlab-prometheus-server ClusterIP 10.104.133.44 <none> 80/TCP 11m
gitlab-redis-headless ClusterIP None <none> 6379/TCP 11m
gitlab-redis-master ClusterIP 10.96.133.252 <none> 6379/TCP 11m
gitlab-redis-metrics ClusterIP 10.105.193.58 <none> 9121/TCP 11m
gitlab-registry ClusterIP 10.109.96.193 <none> 5000/TCP 11m
gitlab-webservice-default ClusterIP 10.108.231.229 <none> 8080/TCP,8181/TCP,8083/TCP 11m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 23m

Go to the https://gitlab.192.168.49.2.nip.io URL:

Find the password:

$ kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath=’{.data.password}’ | base64 — decode ; echo
yNU***njU

And log in as the root user :

Let’s see what we have deployed here:

• gitaly  — already known
• gitlab-exporter  — GitLab metrics collection for its Prometheus
• gitlab-runner — workers for CI/CD
• gitlab-shell  — already known
• kas  — Kubernetes agent server для Gitlab Agent, “GitLab Agent for Kubernetes is an active in-cluster component for solving any GitLab<->Kubernetes integration tasks”
• migrations  — jobs for working with database migrations
• minio — High Performance Object Storage, API compatible with Amazon S3 — used when there is no S3
• postgresql  — already known
• prometheus-server  — GitLab monitoring
• redis-master  — Redis :-)
• registry — Container Registry for storing images
• sidekiq  — already known
• toolbox — backups and other utilities
• webservice-default  — GitLab Rails webserver

Let’s see what’s inside — go to the Admin:

Mmm… Delicious) And a lot. The administration of GitLab will have to be dealt with separately.

In the meantime, we can start investigating the Helm chart and try to deploy it in Kubernetes — the draft of the next post is ready, and will be published soon.

Originally published at RTFM: Linux, DevOps, and system administration.