Introduction
As a Kubernetes user, I chose to use the default tooling available, named Kustomize
, to manage my manifests and all modifications I need to do over them for each environment I want to deploy. For that, Kustomize
is a powerful solution based on concepts of inheritance (resources
) and composition (components
).
But one thing I miss the most is the ability to distribute manifests to other people, letting them inherit or compose them as they want. This is one subject where helm
is ahead of Kustomize
now.
However, Kustomize
provides a mechanism to use remote
elements, for both resources
and components
using git
under the hood. This solution is perfect, because we can use any git
repository as a yaml
registry for our reousrces
and components
🔥.
But, the downside is FluxCD
does not supports so well, for a lot of good reasons (especially when your repositories are private)… In this article, we will see how to bypass this limitation and use remote resources
and components
, the FluxCD
way!
Cluster & FluxCD configuration
In this article, I won't detail so much the cluster & flux configuration… because there is already plenty of documentation and blogposts about that!
For this article, I have set up a cluster using Google Kubernetes Engine (aka GKE) using this documentation and installed FluxCD
connected to a GitLab repository with the following command (extracted from the official documentation).
$ flux bootstrap gitlab \
--owner=davinkevin.fr/articles/flux-and-kustomize-remote-base \
--repository=clusters \
--branch=gke \
--path=fluxcd
NOTE: I don't like to use default
branch, so I chose to create a branch for this cluster, so if I have another cluster, I just need another branch 😉. This is some kind of GitLab Flow for environment.
With this, I have a complete Kubernetes Cluster ready for GitOps, using clusters
repository as source-of-truth. Every modification made in this repository will be automatically deployed by FluxCD
🤩.
bases
repository
I have set up another git
repository dedicated to resources
and components
. It hosts what we usually call base
, because it contains the common core of an application and also many components
to enable some extra features (at infrastructure or application level).
This repository is available in GitLab and is agnostic of my deployment environment. Again, I only host here the bare definition of my applications, we can compare this to an helm
registry. Because it is managed in git
, I can use branch & tags for management 🚀.
NOTE: Usually, this kind of yaml
repository is only populated by CI from other projects automatically.
PodInfo base publication
I have chosen the well known podinfo
application as an example, and I have developed the base
and some components
associated to it.
.
├── README.md
└── podinfo
└── base
├── components
│ ├── hpa
│ │ ├── hpa.yaml
│ │ └── kustomization.yaml
│ ├── ingress
│ │ ├── ingress.yaml
│ │ └── kustomization.yaml
│ └── redis
│ ├── kustomization.yaml
│ ├── redis.conf
│ └── redis.yaml
├── kustomization.yaml
└── podinfo.yaml
Core PodInfo
This part is the common part of all potential deployment. It contains:
- The
podinfo
Kubernetes Service - The
podinfo
Kubernetes Deployment
And nothing else… if you need to deploy any other part of the app, you have to use components
👇.
Components
In this components
, we can find all optional part of our application. As an end-user, I need to declare those I want to use, because by default, they are not applied at all in the common core.
Here, we have:
-
hpa
with a default preconfiguredHorizontal-Pod-Autoscaler
if you want the application to scale up and down automatically -
ingress
definition to expose the application to the outside world (see official documentation) -
redis
component to enable cache on the PodInfo application with default configuration.
It's interesting to note components can be of any sort, hpa
and ingress
are focused on infrastructure when redis
is a feature of the application. Without this redis
component, the application is able to work, just without any caching system. This redis
component is here to deploy a redis
server (of course 😅) but also to enable caching at podinfo
level too.
NOTE: If you want a better description of this component
feature, I advise you to discover this article from the official documentation
So, we have a complete, environment agnostic resource
and components
committed to our base
repository. Now, It is time to use it!
Publish PodInfo to dev
Back to our cluster, I want to deploy this podinfo
application, but of course I want to apply some extra customizations on it, like:
- Define
components
I want to enable - Define a domain name for the
ingress
definition - Define the required
tls
configuration to theingress
- Define an alternative registry for the image
So, to do that, I need to have one (or multiple) kustomization.yaml
to define this:
# <clusters-repo>/podinfo-dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: podinfo-dev
images:
- name: podinfo
newName: ghcr.io/stefanprodan/podinfo
resources:
- ../base
- ingress
components:
- ../base/components/redis
NOTE: The ingress
modification is in its own file to keep this one simple enough.
But, what is this ../base
folder? There is no base
at all in cluster
repository. And you are right… we will use FluxCD
feature available in flux to include
the base
repository at the right location during reconciliation
. To do that, we need to define multiple flux resources.
podinfo-base
flux repository
First, we need a representation of our flux.GitRepository
for bases
, which is materialized in flux with:
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
name: podinfo-base-dev
namespace: flux-system
spec:
interval: 5m
ref:
branch: podinfo
url: ssh://git@gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases
secretRef:
name: flux-system
NOTE: Here, I have chosen to define spec.ref.branch: podinfo
to follow every commit made to this branch. It is, for development, a continuous deployment system 🚀.
podinfo
flux repository
Now, we need to create a flux.GitRepository
for manifests in clusters
repository and manifests from the bases
repository created in the step before. To do that, we are going to use the include
feature from FluxCD:
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
name: podinfo-dev
namespace: flux-system
spec:
interval: 5m
include:
- repository:
name: podinfo-base-dev
fromPath: podinfo/base
toPath: base
ref:
branch: gke
url: ssh://git@gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/clusters
secretRef:
name: flux-system
The important part is the following:
spec:
include:
- repository:
name: podinfo-base-dev # <1>
fromPath: podinfo/base # <2>
toPath: base # <3>
This means the folder podinfo/base
(2) previously defined in the podinfo-base-dev
flux.GitRepository
(1) will be injected in the path base
(3) of the current podinfo-dev
flux.GitRepository
during FluxCD reconciliation… making all resources
and components
available.
podinfo-dev
flux Kustomization
This one is the standard flux.Kustomization
you would use in any case. This one is just using the composite flux.GitRepository
from the previous step as source (spec.sourceRef
):
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
name: podinfo-dev
namespace: flux-system
spec:
suspend: false
interval: 10m0s
path: podinfo-dev # file path in the <clusters> repository
prune: true
sourceRef:
kind: GitRepository
namespace: flux-system
name: podinfo-dev
validation: client
If we deploy all those files (you can put them in the same one), you will see your application deployed by FluxCD with your provided custom configuration.
❯ kubectl get all -n podinfo-dev
NAME READY STATUS RESTARTS AGE
pod/redis-869ff7c78b-jjbbk 1/1 Running 0 103m
pod/podinfo-6dbf56d6d7-ctxxr 1/1 Running 0 103m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/podinfo ClusterIP 10.43.115.207 <none> 9898/TCP,9999/TCP 103m
service/redis ClusterIP 10.43.192.84 <none> 6379/TCP 103m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/redis 1/1 1 1 103m
deployment.apps/podinfo 1/1 1 1 103m
NAME DESIRED CURRENT READY AGE
replicaset.apps/redis-869ff7c78b 1 1 1 103m
replicaset.apps/podinfo-6dbf56d6d7 1 1 1 103m
Publish PodInfo to prod
Now, we want to use again the same base
this time for a production environment.
Because we want something relatively stable, we chose to define spec.ref.tag
to a specific tag in the bases
repository (instead of the podinfo
branch).
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
name: podinfo-base-prod
namespace: flux-system
spec:
interval: 5m
ref:
tag: podinfo-1.0
url: ssh://git@gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases
secretRef:
name: flux-system
All others flux.Kustomization
and flux.GitRepository
are the same.
Because we are in production, we also want to enable different modules, here the HorizontalPodAutoscaler
. So we have a different kustomization.yaml
. Thanks to the full control we have over the kustomization.yaml
per environment, we can modify this only in production 😍.
Once everything is published and FluxCD reconciliation is OK, we have:
❯ kubectl get all -n podinfo-prod
NAME READY STATUS RESTARTS AGE
pod/redis-869ff7c78b-lnll5 1/1 Running 0 113m
pod/podinfo-6dbf56d6d7-k8rjl 1/1 Running 0 113m
pod/podinfo-6dbf56d6d7-qldr8 1/1 Running 0 113m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/podinfo ClusterIP 10.43.109.213 <none> 9898/TCP,9999/TCP 113m
service/redis ClusterIP 10.43.252.54 <none> 6379/TCP 113m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/redis 1/1 1 1 113m
deployment.apps/podinfo 2/2 2 2 113m
NAME DESIRED CURRENT READY AGE
replicaset.apps/redis-869ff7c78b 1 1 1 113m
replicaset.apps/podinfo-6dbf56d6d7 2 2 2 113m
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/podinfo Deployment/podinfo 4%/99% 2 4 2 113m
Conclusion
It was tough, with many yaml
but we have achieved our original goal. Being able to publish bare manifests in a central place and allow other team / people to consume them instantaneously with all features available in Kustomize.
We are then able to enjoy all the feature of a GitOps model with a dedicated (yaml
) registry, like helm
already have 😇.
There is some follow-up improvements to come, like OCI registries (announced in FluxCD 0.32), to replace flux.GitRepository
. We still need this issue to be solved to be able to use it with all features provided by Kustomize.
In a more distant future, some other solution like porch could be a good candidate to simplify again our deployment strategy… but it is currently in alpha
stage.
I hope you liked this article, you can find all resources used in GitLab, and you can reproduce it in your own cluster! If you have any questions, don't hesitate to comment or ping me on twitter @davinkevin
Top comments (1)
Can you please also show how HelmRepository works and how can HelmRepo be integrated via FluxCD