Sunny Bhambhani for AWS Community Builders

Posted on Dec 26, 2023 • Edited on Mar 15 • Originally published at cncf.io

Installing multiple helm charts in one go [Approach 2 - using helmfile]

#linux #kubernetes #devops #helm

In this article we will be talking about Approach 2 i.e. how to get multiple helm charts installed using helmfile.

If you haven't read the previous article where I discussed about Approach 1, feel free to read it over.

Ref: https://dev.to/aws-builders/installing-multiple-helm-charts-in-one-go-approach-1-5d1p

Prerequisites:

A running Kubernetes cluster with proper permissions, here I have used minikube.
kubectl, helm, and helmfile installed on your machine to interact with the Kubernetes cluster.

Installation of helmfile is pretty straight forward, we just need to grab its binary and put that in /usr/local/bin and we are done (Ref: https://helmfile.readthedocs.io/en/latest/#installation for operating system other than linux).

Grab the latest one from the assets section from here: https://github.com/helmfile/helmfile/releases.

sudo wget https://github.com/helmfile/helmfile/releases/download/v0.159.0/helmfile_0.159.0_linux_amd64.tar.gz
sudo tar -xxf helmfile_0.159.0_linux_amd64.tar.gz
sudo rm helmfile_0.159.0_linux_amd64.tar.gz
sudo mv helmfile /usr/local/bin/

Once helmfile is installed, we need to initialize it so that it can download necessary helm plugins.

helmfile init

If you want to list it out what all plugins init installs, it can be found using below command:

$ helm plugin list
NAME            VERSION DESCRIPTION
diff            3.8.1   Preview helm upgrade changes as a diff
helm-git        0.12.0  Get non-packaged Charts directly from Git.
s3              0.14.0  Provides AWS S3 protocol support for charts and repos. https://github.com/hypnoglow/helm-s3
secrets         4.1.1   This plugin provides secrets values encryption for Helm charts secure storing

Note: A notice to upgrade helm may appear if the helm version is 3.10 or lower.

helm version is too low, the current version is 3.10.1+g9f88ccb, the required version is 3.12.3
use: 'https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3' [y/n]: y

Approach 2:

helmfile is pretty intresting and a very useful utility when it comes to get multiple helm charts installed. Using a single file helmfile.yaml we can manage N number of charts.

When it comes to installing mutiple helm charts, helmfile is a really helpful and interesting tool. The helmfile.yaml values file and helmfile binary is all that is needed to manage N number of charts. Internally helmfile invokes helm command to manage Kubernetes objects/charts. Also you can think of helmfile as a wrapper around helm together with some helm plugins.

Below is a quick example, wherein I have a 3-Tier application comprising of a frontend, backend, and database and for all of them I have 3 individual charts.

$ tree
.
├── backend
│   ├── Chart.yaml
│   ├── templates
│   │   ├── _helpers.tpl
│   │   ├── deployment.yaml
│   │   ├── hpa.yaml
│   │   ├── ingress.yaml
│   │   ├── service.yaml
│   │   └── serviceaccount.yaml
│   └── values.yaml
├── database
│   ├── Chart.yaml
│   ├── templates
│   │   ├── _helpers.tpl
│   │   ├── deployment.yaml
│   │   ├── hpa.yaml
│   │   ├── ingress.yaml
│   │   ├── service.yaml
│   │   └── serviceaccount.yaml
│   └── values.yaml
└── webapp
    ├── Chart.yaml
    ├── templates
    │   ├── _helpers.tpl
    │   ├── deployment.yaml
    │   ├── hpa.yaml
    │   ├── ingress.yaml
    │   ├── service.yaml
    │   └── serviceaccount.yaml
    └── values.yaml

6 directories, 24 files

Now, we will use helmfile (that too a single command to get these charts installed).

In the same directory where my charts are there, lets create a new file called helmfile.yaml. Just to add here we are referring local charts now, but we will refer charts from helm repositories in next examples.

Location: Local helm charts

$ cat helmfile.yaml
releases:
  - name: webapp
    namespace: default
    chart: ./webapp
    version: "0.1.0"
    wait: true
    installed: true
  - name: backend
    namespace: default
    chart: ./backend
    wait: true
  - name: database
    namespace: default
    chart: ./database
    wait: true

Lets break this file now:

There can be multiple sections in this YAML file, but majorly we will focus on .releases and .repositories sections.
For more details around what all things we can have in a helmfile.yaml, ref: https://helmfile.readthedocs.io/en/latest/#configuration.
.releases is an array which basically signifies what all helm releases we need to deploy in a Kubernetes cluster.
- Important KV pairs in each item are .releases[].name which refers to the name of the release.
- .releases[].namespace, it refers to the name of the namespace where we need to install the chart.
- .releases[].chart, it refers to the location of the chart from where helm needs to pull it. In this example its in my local machine and in the current directory.
- .releases[].version, it refers to the version of chart which needs to be installed.
- .releases[].installed, it is a boolean key which referes if we want to get specific chart installed or not (by default the value is true).
- .releases[].wait, it is same as --wait flag of helm, which means the operation will wait untill all the components are ready/healthy. Ref: https://helm.sh/docs/intro/using_helm/.

Now, lets see helmfile in action:

As of now nothing is installed.

$ helm list -A
NAME    NAMESPACE       REVISION        UPDATED STATUS  CHART   APP VERSION
$

We will first initialize helm to gather all the dependencies.

$ helmfile init
helmfile initialization completed!
$

Lets list out what all releases are defined in our release file?

$ helmfile list
NAME            NAMESPACE       ENABLED INSTALLED       LABELS  CHART           VERSION
webapp          default         true    true                    ./webapp        0.1.0
backend         default         true    true                    ./backend
database        default         true    true                    ./database
$

All, looks good, lets apply the state file using helmfile apply.

$ helmfile apply
Building dependency release=webapp, chart=webapp
Building dependency release=backend, chart=backend
Building dependency release=database, chart=database
Comparing release=webapp, chart=webapp
********************

        Release was not present in Helm.  Diff will show entire contents as new.

********************
default, webapp, Deployment (apps) has been added:
-
+ # Source: webapp/templates/deployment.yaml
+ apiVersion: apps/v1

#.........
#.........
# OUTPUT TRIMMED

UPDATED RELEASES:
NAME       CHART        VERSION   DURATION
webapp     ./webapp     0.1.0           3s
database   ./database   0.1.0           3s
backend    ./backend    0.1.0           3s

If you will see here, using a single command it instructed helm to build the dependency of all the charts specified in the state file then it used helm diff to list out actual k8s yaml files, then it installs all the releases.
If you will fire helm list or kubectl get pods now, you will see all the intended resources.

$ helm list -A
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
backend         default         1               2023-12-11 16:13:24.769966852 +0530 IST deployed        backend-0.1.0   1.16.0
database        default         1               2023-12-11 16:13:24.771955874 +0530 IST deployed        database-0.1.0  1.16.0
webapp          default         1               2023-12-11 16:13:24.771767015 +0530 IST deployed        webapp-0.1.0    1.16.0
$ k get pods
NAME                       READY   STATUS    RESTARTS   AGE
backend-7f458d4566-zwcz2   1/1     Running   0          3m47s
database-b4f679788-z6r84   1/1     Running   0          3m47s
webapp-7f6ffdc676-g5w7j    1/1     Running   0          3m47s
$

Get the status using helmfile status if required.

$ helmfile status
Getting status backend
Getting status webapp
Getting status database
NAME: backend
LAST DEPLOYED: Mon Dec 11 16:13:24 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

NAME: webapp
LAST DEPLOYED: Mon Dec 11 16:13:24 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

NAME: database
LAST DEPLOYED: Mon Dec 11 16:13:24 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

Now, lets assume you just need to update the database release and that too just the replicaCount,
- How to update that using values file or using set key?
- What helmfile will do?
- How to check what all changes helmfile will make?

By default, similar to helm, this will also use helm chart's default values.yaml file which is a part of helm chart itself. So either you can update that and fulfill your requirement (which idelly is not a good practice, since in real world scenario that is most unlikely to happen and ideally we should to override intended keys).

To cater this need we have something called values and set which we can specify in the state file and helmfile will consider those values and will override them.

Lets use the same example, but now I have added few additional line for backend as well as for the database item in our releases array.

releases:
  - name: webapp
    namespace: default
    chart: ./webapp
    version: "0.1.0"
    wait: true
    installed: true
  - name: backend
    namespace: default
    chart: ./backend
    wait: true
    set:
      - name: replicaCount
        value: 2
  - name: database
    namespace: default
    chart: ./database
    wait: true
    values:
      - "./db-values.yaml"

$ cat db-values.yaml
replicaCount: 2

Just to cut short, values is same as --values and set is same as --set in helm.
Here if you will see, for backend I have used set and I am overwriding relicaCount to 2 and for database I have used 'values' and there I have passed a path where my values file resides which contains again the replicaCount as 2 but for database release.
Now, comes the 2nd question, how to see what all differences will helmfile make, simply fire helmfile diff.

$ helmfile diff
Building dependency release=webapp, chart=webapp
Building dependency release=backend, chart=backend
Building dependency release=database, chart=database
Comparing release=webapp, chart=webapp
Comparing release=backend, chart=backend
default, backend, Deployment (apps) has changed:
  # Source: backend/templates/deployment.yaml
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: backend
# ......
# OUTPUT TRIMMED

-   replicas: 1
+   replicas: 2
# ......
# OUTPUT TRIMMED

Comparing release=database, chart=database
default, database, Deployment (apps) has changed:
  # Source: database/templates/deployment.yaml
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: database
# ......
# OUTPUT TRIMMED

-   replicas: 1
+   replicas: 2
# ......
# OUTPUT TRIMMED

Here, if you will observe, firstly it compared the release webapp (wherein it didn't found any differences then it compared backend where it found that the replicaCount was updated to 2) then it compared database where it again found that the replicaCount was updated to 2.
Using helmfile apply, we will apply the changes now.

Note: helmfile apply basically calls two operations helmfile diff and helmfilesync.

Again it will show you what all changes it will bring then will apply the changes.

$ helmfile apply
Building dependency release=webapp, chart=webapp
Building dependency release=backend, chart=backend
Building dependency release=database, chart=database
Comparing release=webapp, chart=webapp
default, backend, Deployment (apps) has changed:
# ..........
# OUTPUT TRIMMED

-   replicas: 1
+   replicas: 2

default, database, Deployment (apps) has changed:
# ..........
# OUTPUT TRIMMED

-   replicas: 1
+   replicas: 2

Upgrading release=database, chart=database
Upgrading release=backend, chart=backend
Release "backend" has been upgraded. Happy Helming!
NAME: backend
LAST DEPLOYED: Mon Dec 11 16:37:27 2023
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None

Listing releases matching ^backend$
Release "database" has been upgraded. Happy Helming!
NAME: database
LAST DEPLOYED: Mon Dec 11 16:37:27 2023
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None

UPDATED RELEASES:
NAME       CHART        VERSION   DURATION
backend    ./backend    0.1.0           3s
database   ./database   0.1.0           3s

$ helm list
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
backend         default         2               2023-12-11 16:37:27.366097438 +0530 IST deployed        backend-0.1.0   1.16.0
database        default         2               2023-12-11 16:37:27.366586959 +0530 IST deployed        database-0.1.0  1.16.0
webapp          default         1               2023-12-11 16:13:24.771767015 +0530 IST deployed        webapp-0.1.0    1.16.0

Revision got incremented by 1, and now I have 2 new pods running +1 for backend and +1 for database.

$ k get pods
NAME                       READY   STATUS    RESTARTS   AGE
backend-7f458d4566-6jgxk   1/1     Running   0          12s
backend-7f458d4566-zwcz2   1/1     Running   0          24m
database-b4f679788-9kxhl   1/1     Running   0          12s
database-b4f679788-z6r84   1/1     Running   0          24m
webapp-7f6ffdc676-g5w7j    1/1     Running   0          24m

In the above example we referred the helm charts which were placed locally, now lets see how to consume the helm charts which are a part of OCI or HTTPS repositories.

Location: helm charts from OCI repositories

Let's now install a helm chart from an opensource oci repository (ref: https://artifacthub.io/); Just for an example we will install nginx chart.
Again, I have a new helmfile.yaml and the only difference here is in the chart key.

releases:
  - name: nginx
    namespace: default
    chart: oci://registry-1.docker.io/bitnamicharts/nginx
    version: "15.4.4"
    wait: true
    set:
      - name: service.type
        value: ClusterIP

Here, if you will observe firstly helmfile is pulling the helmchart then will list out the difference i.e. what all things it will create and then will get the k8s resources created.

$ helmfile apply
Pulling registry-1.docker.io/bitnamicharts/nginx:15.4.4
Comparing release=nginx, chart=/tmp/helmfile976774944/default/nginx/nginx/15.4.4/nginx
********************

        Release was not present in Helm.  Diff will show entire contents as new.

********************
default, nginx, Deployment (apps) has been added:
# ......
# ......
# OUTPUT TRIMMED

Location: helm charts from HTTPS repositories

Let's now install a helm chart from an opensource HTTPS repository and here I have taken an example from the official helmfile documentation (ref: https://helmfile.readthedocs.io/en/latest/#getting-started)
Here is the new helmfile.yaml and now the difference here is .repositories and .releases[].chart.

repositories:
 - name: prometheus-community
   url: https://prometheus-community.github.io/helm-charts

releases:
- name: prom-norbac-ubuntu
  namespace: prometheus
  chart: prometheus-community/prometheus
  set:
  - name: rbac.create
    value: false

Here, if you will observe in this helmfile we have a new section called .repositories which again is an array and can contain multiple repositories with a human friendly name which can be referred in the .releases section. For instance here the repo is https://prometheus-community.github.io/helm-charts and it can be referred in the releases section as prometheus-community.
And in-turn .releases[].chart now is prometheus-community/prometheus where prometheus is the chart name. When installing many charts from the same source, this eliminates duplication, and if we only need to change the repository, the update would only be needed at one place and we are done.
Another point to note here is, adding .repositories is similar to running helm repo add prometheus-communit https://prometheus-community.github.io/helm-charts this command.

$ helmfile apply
Adding repo prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories

Comparing release=prom-norbac-ubuntu, chart=prometheus-community/prometheus
********************

        Release was not present in Helm.  Diff will show entire contents as new.

********************
prometheus, prom-norbac-ubuntu-alertmanager, ConfigMap (v1) has been added:
# ......
# ......
# OUTPUT TRIMMED

Repositories

We already touched upon this topic but just for completeness sake similar to above example we can have OCI repositories as well defined under .repositories section.
Here is an example for the same:

repositories:
  - name: ocirepo
    url: registry-1.docker.io/bitnamicharts
    oci: true
releases:
  - name: nginx
    namespace: default
    chart: ocirepo/nginx
    version: 15.4.4
    wait: true
    set:
      - name: service.type
        value: ClusterIP

If you are referring an oci repository make sure to use repositories[].oci as true.

$ helmfile apply
Pulling registry-1.docker.io/bitnamicharts/nginx:15.4.4
Comparing release=nginx, chart=/tmp/helmfile4237214626/default/nginx/nginx/15.4.4/nginx
********************

        Release was not present in Helm.  Diff will show entire contents as new.

********************
default, nginx, Deployment (apps) has been added:
# ......
# ......
# OUTPUT TRIMMED