Note: this article was originally published on Stack Labs blog.
We always need to customize our deployment with Kubernetes and, I don't know why but the main tool around for now is HELM which throws away all the logic we learn on docker and Kubernetes.
Here I will introduce to you an alternative called Kustomize ❤️
Kustomize isn't a new tool, it is under construction since 2017 and has been introduced as a native kubectl
sub-command in the version 1.14
. Yeah, you've heard correctly, this is now embedded directly inside the tool you use everyday... so you will be able to throw that helm
command away 😉.
Philosophy
Kustomize tries to follow the philosophy you are using in your everyday job when using Git as VCS, creating Docker images or declaring your resources inside Kubernetes.
So, first of all, Kustomize is like Kubernetes, it is totally declarative !
You say what you want and the system provides it to you. You don't have to follow the imperative way and describe how you want it to build the thing.
Secondly, it works like Docker. You have many layers and each of those is modifying the previous ones.
Thanks to that, you can constantly write things above others without adding complexity inside your configuration.
The result of the build will be the addition of the base and the different layers you applied over it.
Lastly, like Git, you can use a remote base as the start of your work and add some customization on it.
Installation
Of course, for 🍎 Mac users, you can use brew
to install it :
$ brew install kustomize
If you are on another operating system, you can directly download the binary from the release page and add it to your path.
For the others, you also can build it from source, why not 😅.
Your base
To start with Kustomize, you need to have your original yaml files describing any resources you want to deploy into your cluster.
Those files will be stored for this example in the folder ./k8s/base/
.
Those files will NEVER (EVER) be touched, we will just apply customization above them to create new resources definitions
Note: You can build base templates (e.g. for dev environment) at any point in time using the command kubectl apply -f ./k8s/base/.
In this example, we will work with a service
and a deployment
resources:
apiVersion: v1
kind: Service
metadata:
name: sl-demo-app
spec:
ports:
- name: http
port: 8080
selector:
app: sl-demo-app
apiVersion: apps/v1
kind: Deployment
metadata:
name: sl-demo-app
spec:
selector:
matchLabels:
app: sl-demo-app
template:
metadata:
labels:
app: sl-demo-app
spec:
containers:
- name: app
image: foo/bar:latest
ports:
- name: http
containerPort: 8080
protocol: TCP
We wil add a new file inside this folder, named kustomization.yaml
:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- service.yaml
- deployment.yaml
This file will be the central point of your base and it describes the resources you use.
Those resources are the path to the files relatively to the current file.
Note: This
kustomization.yaml
file could lead to errors when runningkubectl apply -f ./k8s/base/
, you can either run it with the parameter--validate=false
or simply not running the command against the whole folder
To apply your base template to your cluster, you just have to execute the following command:
$ kubectl apply -k k8s/base
To see what will be applied in your cluster, we will mainly use in this article the command kustomize build
instead of kubectl apply -k
.
The result of kustomize build k8s/base
command will be the following, which is for now only the two files previously seen, concatenated:
apiVersion: v1
kind: Service
metadata:
name: sl-demo-app
spec:
ports:
- name: http
port: 8080
selector:
app: sl-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: sl-demo-app
spec:
selector:
matchLabels:
app: sl-demo-app
template:
metadata:
labels:
app: sl-demo-app
spec:
containers:
- image: foo/bar:latest
name: app
ports:
- containerPort: 8080
name: http
protocol: TCP
Kustomization
Now, we want to kustomize our app for a specific case, for example, for our prod
environement.
In each step, we will see how to enhance our base with some modification.
The main goal of this article is not to cover the whole set of functionnalities of Kustomize but to be a standard example to show you the phiplosophy behind this tool.
First of all, we will create the folder k8s/overlays/prod
with a kustomization.yaml
inside it.
The k8s/overlays/prod/kustomization.yaml
has the following content:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
If we build it, we will see the same result as before when building the base.
$ kustomize build k8s/overlays/prod
This will output the following yaml
apiVersion: v1
kind: Service
metadata:
name: sl-demo-app
spec:
ports:
- name: http
port: 8080
selector:
app: sl-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: sl-demo-app
spec:
selector:
matchLabels:
app: sl-demo-app
template:
metadata:
labels:
app: sl-demo-app
spec:
containers:
- image: foo/bar:latest
name: app
ports:
- containerPort: 8080
name: http
protocol: TCP
We are now ready to apply kustomization for our prod
env
Define Env variables for our deployment
In our base
, we didn't define any env variable. We will now add those env
variables above our base.
To do so, it's very simple, we just have to create the chunk of yaml we would like to apply above our base and referece it inside the kustomization.yaml
.
This file custom-env.yaml
containing env variables will look like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: sl-demo-app
spec:
template:
spec:
containers:
- name: app # (1)
env:
- name: CUSTOM_ENV_VARIABLE
value: Value defined by Kustomize ❤️
Note: The
name
(1) key here is very important and allow Kustomize to find the right container which need to be modified.
You can see this yaml
file isn't valid by itself but it describes only the addition we would like to do on our previous base.
We just have to add this file to a specific entry in the k8s/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
patchesStrategicMerge:
- custom-env.yaml
If we build this one, we will have the following result:
$ kustomize build k8s/overlays/prod
apiVersion: v1
kind: Service
metadata:
name: sl-demo-app
spec:
ports:
- name: http
port: 8080
selector:
app: sl-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: sl-demo-app
spec:
selector:
matchLabels:
app: sl-demo-app
template:
metadata:
labels:
app: sl-demo-app
spec:
containers:
- env:
- name: CUSTOM_ENV_VARIABLE # (1)
value: Value defined by Kustomize ❤️
image: foo/bar:latest
name: app
ports:
- containerPort: 8080
name: http
protocol: TCP
You can see our env
block has been applied above our base and now the CUSTOM_ENV_VARIABLE
(1) will be defined inside our deployment.yaml.
Change the number of replica
Like in our previous example, we will extend our base to define variables not already defined
Note: You can also override some variables already present in your base files.
Here, we would like to add information about the number of replica. Like before, a chunk or yaml
with just the extra info needed for defining replica will be enought:
apiVersion: apps/v1
kind: Deployment
metadata:
name: sl-demo-app
spec:
replicas: 10
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
And like before, we add it to the list of patchesStrategicMerge
in the kustomization.yaml
:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
patchesStrategicMerge:
- custom-env.yaml
- replica-and-rollout-strategy.yaml
The result of the command kustomize build k8s/overlays/prod
give us the following result
apiVersion: v1
kind: Service
metadata:
name: sl-demo-app
spec:
ports:
- name: http
port: 8080
selector:
app: sl-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: sl-demo-app
spec:
replicas: 10
selector:
matchLabels:
app: sl-demo-app
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: sl-demo-app
spec:
containers:
- env:
- name: CUSTOM_ENV_VARIABLE
value: Value defined by Kustomize ❤️
image: foo/bar:latest
name: app
ports:
- containerPort: 8080
name: http
protocol: TCP
And you can see the replica number and rollingUpdate strategy have been applied above our base.
Use a secret define through command line
One of the things we often do is to set some variables as secret from command-line.
In our case, we are doing this directly from our Gitlab-CI on Gitlab.com.
But you can do this from anywhere else, the main purpose here is to define Kubernetes Secret without putting them inside Git 😱.
To do so, kustomize
has a sub-command to edit a kustomization.yaml
and create a secret for you. You just have to use it in your deployment like if it already exists.
$ cd k8s/overlays/prod
$ kustomize edit add secret sl-demo-app --from-literal=db-password=12345
These commands will modify your kustomization.yaml
and add a SecretGenerator
inside it.
Note: You can also use secret comming from properties file (with
--from-file=file/path
) or from env file (with--from-env-file=env/path.env
)
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
patchesStrategicMerge:
- custom-env.yaml
- replica-and-rollout-strategy.yaml
secretGenerator:
- literals:
- db-password=12345
name: sl-demo-app
type: Opaque
If you run the kustomize build k8s/overlays/prod
from the root folder of the example project, you will have the following output
apiVersion: v1
data:
db-password: MTIzNDU=
kind: Secret
metadata:
name: sl-demo-app-6ft88t2625
type: Opaque
---
apiVersion: v1
kind: Service
metadata:
name: sl-demo-app
spec:
ports:
- name: http
port: 8080
selector:
app: sl-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: sl-demo-app
spec:
replicas: 10
selector:
matchLabels:
app: sl-demo-app
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: sl-demo-app
spec:
containers:
- env:
- name: CUSTOM_ENV_VARIABLE
value: Value defined by Kustomize ❤️
image: foo/bar:latest
name: app
ports:
- containerPort: 8080
name: http
protocol: TCP
Note: The secret name is
sl-demo-app-6ft88t2625
instead ofsl-demo-app
, it's normal and this is made to trigger a rolling update of the deployment if secrets content is changed.
If we want to use this secret from our deployment, we just have, like before, to add a new layer definition which uses the secret.
For example, this file will mount the db-password
value as environement variables
apiVersion: apps/v1
kind: Deployment
metadata:
name: sl-demo-app
spec:
template:
spec:
containers:
- name: app
env:
- name: "DB_PASSWORD"
valueFrom:
secretKeyRef:
name: sl-demo-app
key: db.password
And, like before, we add this to the k8s/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
patchesStrategicMerge:
- custom-env.yaml
- replica-and-rollout-strategy.yaml
- database-secret.yaml
secretGenerator:
- literals:
- db-password=12345
name: sl-demo-app
type: Opaque
If we build the whole prod
files, we now have
apiVersion: v1
data:
db-password: MTIzNDU=
kind: Secret
metadata:
name: sl-demo-app-6ft88t2625
type: Opaque
---
apiVersion: v1
kind: Service
metadata:
name: sl-demo-app
spec:
ports:
- name: http
port: 8080
selector:
app: sl-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: sl-demo-app
spec:
replicas: 10
selector:
matchLabels:
app: sl-demo-app
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: sl-demo-app
spec:
containers:
- env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
key: db.password
name: sl-demo-app-6ft88t2625 # (1)
- name: CUSTOM_ENV_VARIABLE
value: Value defined by Kustomize ❤️
image: foo/bar:latest
name: app
ports:
- containerPort: 8080
name: http
protocol: TCP
You can see the secretKeyRef.name
used is automatically modified to follow the name defined by Kustomize (1)
Note: Don't forget, the command to put the secret inside the
kustomization.yaml
file should be made only from safe env and should not be commited.
The same logic exists with ConfigMap with hash at the end to allow redeployement of your app if ConfigMap changes.
Change the image of a deployment
Like for secret, there is a custom directive to allow changing of image or tag directly from the command line.
This is very useful if you need to deploy the image previously tagged by your continuous build system.
To do that, you can use the following command:
$ cd k8s/overlays/prod
$ TAG_VERSION=3.4.5 # (1)
$ kustomize edit add secret sl-demo-app --from-literal=db-password=12345 # To create my required secret
$ kustomize edit set image foo/bar=foo/bar:$TAG_VERSION
Note: the
TAG_VERSION
here is usualy defined by your CI/CD system
The k8s/overlays/prod/kustomization.yaml
will be modified with those values:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
patchesStrategicMerge:
- custom-env.yaml
- replica-and-rollout-strategy.yaml
- database-secret.yaml
secretGenerator:
- literals:
- db-password=12345
name: sl-demo-app
type: Opaque
images:
- name: foo/bar
newName: foo/bar
newTag: 3.4.5
And if we build it, with the kustomize build k8s/overlays/prod/
we have the following result:
apiVersion: v1
data:
db-password: MTIzNDU=
kind: Secret
metadata:
name: sl-demo-app-6ft88t2625
type: Opaque
---
apiVersion: v1
kind: Service
metadata:
name: sl-demo-app
spec:
ports:
- name: http
port: 8080
selector:
app: sl-demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: sl-demo-app
spec:
replicas: 10
selector:
matchLabels:
app: sl-demo-app
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: sl-demo-app
spec:
containers:
- env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
key: db.password
name: sl-demo-app-6ft88t2625
- name: CUSTOM_ENV_VARIABLE
value: Value defined by Kustomize ❤️
image: foo/bar:3.4.5 # (1)
name: app
ports:
- containerPort: 8080
name: http
protocol: TCP
You see the first container.image
of the deployment have been modified to be run with the version 3.4.5
(1).
Conclusion
We see in these examples how we can leverage the power of Kustomize to define your Kubernetes files without even using a templating system.
All the modification files you made will be applied above the original files without altering it with curly braces and imperative modification.
There is a lot of advanced topic in Kustomize, like the mixins and inheritance logic or other directive allowing to define a name, label or namespace to every created object...
You can follow the official Kustomize github repository to see advanced examples and documentation.
Note: You can find all code from this article in this Gitlab project.
Top comments (0)