FluxCD is a powerful tool for managing deployments in Kubernetes using GitOps principles. While it offers a wide range of features, this post will explore scenarios where a simpler approach might be preferable, aligning with the #SimplerIsBetter philosophy.
NOTE: This article shares insights and perspectives gained through experience in various contexts and companies. Feel free to disagree, but with respect! 😉
HelmRelease
, what is this?
HelmRelease
is a custom resource provided by FluxCD, which provides users
a way to automatically install helm
charts using FluxCD and its declarative system.
For example, if you want to install the PodInfo app, you have to declare
the following manifest:
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: podinfo
spec:
chart:
spec:
chart: podinfo
version: '6.5.*'
sourceRef:
kind: HelmRepository
name: podinfo
interval: 5m
releaseName: podinfo
values: # part dedicated to the all `values` the chart accept
replicaCount: 2
NOTE: For the sake of simplicity, I kept only relevant attributes, but I can say FluxCD offers a rich API to cover most of use cases.
The simplified workflow can be described like this:
- The
HelmOperator
verifies and retrieves the Helm chart from a remote source like GitHub, GitLab, Artifactory or an OCI registry (recommended) - It then renders the chart using configuration from the
HelmRelease
object, which can include values provided in different ways (inlined, using ConfigMap or Secret) - Finally, the
HelmOperator
applies the generatedYAML
manifests to the Kubernetes API to install or upgrade the application.
This approach can work for simple deployments, but as an operator, I've encountered several design drawbacks that I'd like to discuss. Let's see these issues!
NOTE Like every GitOps solution, FluxCD requires a connection to a GitRepository
too. To keep the diagram simple, I've not materialized this item.
GitOps, aka "only source of truth"
The GitOps philosophy is built on top of 3 key principles:
- Declarative System: You define the desired state of your system (what you want) using declarative language (4GL). This approach focuses on the "what" instead of the "how," making your configuration easier to understand and maintain.
- System State Captured in a Git Repository: The desired state of your system is stored in a Git repository. This provides a central location for managing your infrastructure configurations, enabling version control, collaboration, and easy rollbacks if needed.
- Automatic Deployment system: Any changes pushed to the Git repository trigger an automated deployment process. This automates the process of translating your desired state into actual changes within your system, reducing manual intervention and the risk of errors.
The HelmRelease
is a Declarative approach, where automation is managed by the controller. However, "desired state of your system is stored in a Git repository" and all it implies are not respected.
External Chart Dependencies: A Potential Weak Point
A core principle of building resilient systems is ensuring the availability of all their components. When using HelmRelease
, we introduce an external dependency, the location where the Helm chart resides.
If communication with this external location fails (due to server downtime, network issues, etc.), you might not be able to install or update your application. This introduces a potential single point of failure (SPOF) in your deployment process. Additionally, the desired state of your cluster is not solely defined by your git
repository; it also depends on all the charts your system downloads.
Limited Visibility into Helm Chart Content
Another purpose of this System State Captured in a Git Repository is its auditability. If you capture the complete state of a system in git
, you can review it before an installation or upgrade. However, using HelmRelease
introduces a layer of opacity.
While you declare the specific chart you want to use, you might not have a complete picture of what resources the chart will actually install in your cluster. It could potentially create various resources like ClusterRoles
, NetworkPolicies
, or DaemonSets
. You can't say without… deploying it, running it locally or worth, reading the chart's source. 😞
Lack of Immutability in Helm Charts!
The problem we already have with container images applied the same way to helm
charts. A chart produced and published at a specific date might be un-published or re-published with a different content. In those case, you expose your system to the two previous points again…
NOTE We can now store charts in OCI registries, with immutability feature. To leverage it, you have to complexify your system with digest
pinning, because in security, we can't blindly trust a 3rd party 😇.
Chart customisation, another nightmare 👻
While Helm charts offer a convenient way to package deployments, maintaining them can be challenging due to their complexity. Kubernetes itself provides a wide range of configuration options, which can further complicate matters.
This complexity can lead to situations where the desired configuration isn't readily available within a chart. For example, you might want to add an annotation to a workload or modify taints and tolerations, but the chart may not offer built-in ways to do so.
To address this challenge, FluxCD introduced the concept of postRenderers
(see documentation within HelmRelease
resources. This feature leverages the Kustomize API to customize deployments after the initial Helm chart rendering.
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: podinfo
spec:
releaseName: podinfo
chart: { … }
values: { … }
postRenderers:
- kustomize:
patches:
- target:
version: v1
kind: Deployment
name: metrics-server
patch: |
- op: add
path: /metadata/labels/environment
value: production
images:
- name: docker.io/bitnami/metrics-server
newName: docker.io/bitnami/metrics-server
newTag: 0.4.1-debian-10-r54
The configuration for postRenderers
is separate from the Helm chart result. This can make it difficult to understand the complete picture of how the final deployment will be configured, potentially leading to hidden errors. 🤯
FluxCD doesn't provide robust mechanisms to analyze the resources created after the combined rendering and post-rendering steps. This can make troubleshooting issues arising from these modifications cumbersome. 😞
Solution is simplicity 🚀!
We've discussed the challenges associated with relying solely on Helm charts within FluxCD deployments. These challenges can compromise the visibility, maintainability, and overall health of your GitOps workflow.
So, how can we achieve the ideal balance: a declarative system, automatic deployments, and a clear picture of your system state captured entirely within your Git repository?
Render the chart and store it into git
for better auditability
The answer lies in a simple yet powerful sub-command – helm template
(or helmfile template
if you use Helmfile). This command allows you to locally render helm charts along with your desired values, generating the final deployment manifest files.
$ helm repo add podinfo https://stefanprodan.github.io/podinfo
"podinfo" has been added to your repositories
$ helm repo update podinfo
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "podinfo" chart repository
Update Complete. ⎈Happy Helming!⎈
$ echo "replicaCount: 2" > values.yaml
$ helm template podinfo/podinfo -f values.yaml --version 6.6.3 > podinfo.yaml
And that's it! It was not so complicated 😇. We have a file called podinfo.yaml
, located in /k8s/podinfo
of our git
repository.
This file is now yours, it can be read, analyzed and pushed to kubernetes
. After this generation, your system is free from the external chart registry where the chart is located.
NOTE I recommend to never modify a file "generated" by another tool, because you will loose this modification during the next rendering. tldr; treat them as "read-only".
How to deploy generated files with FluxCD?
Instead of using HelmRelease
, we're going to use Kustomization
resources provided by FluxCD. It is way simpler than HelmRelease
, because it just deploys manifests located in a specific location.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: podinfo
spec:
sourceRef:
kind: GitRepository
name: our-gitops-repository
path: "/k8s/podinfo" # our location in our GitOps repo
prune: true
timeout: 1m
Because we use the GitRepository
, called our-gitops-repository
and eventually used by FluxCD
itself, there is no extra dependency in our system.
Direct Customization with Kustomize
While HelmRelease
offers postRenderers
for some customizations, the Kustomization
resource provides full access to the powerful Kustomize capabilities.
This allows for more granular and flexible control over your manifests. Here's how to achieve a similar customization as the previous postRenderer
example using a kustomization.yaml
file:
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- podinfo.yaml # your generated file from the previous step
patches:
- target:
version: v1
kind: Deployment
name: metrics-server
patch: |
- op: add
path: /metadata/labels/environment
value: production
images:
- name: docker.io/bitnami/metrics-server
newName: docker.io/bitnami/metrics-server
newTag: 0.4.1-debian-10-r54
kustomize offers a wider range of functionalities compared to postRenderers
. You can manipulate resources using features like namePrefix
, labels
, replacements
, components
… The list is too long to be detailed here 😇.
As a bonus point, you can run kustomize build /k8s/podinfo/
and see the complete result of the generation before any interaction with FluxCD.
Enjoy reviews and audit with rich diff!
One of the significant advantages of managing your full Kubernetes state with GitOps is the ability to leverage Git's powerful version control capabilities for reviewing and auditing deployments.
From an operator perspective, there is nothing better than a clear and detailed diff views during tool upgrade:
Obviously, your IDE will be your best friend to understand what happened, with clear context and details of changes:
NOTE Upgrade can be automated using tools like renovate
or dependabot
Conclusion
While Helm charts offer a convenient way to package deployments, maintaining them in a FluxCD workflow can introduce challenges related to transparency, maintainability, and control.
This article explored the limitations of HelmReleases
and presented helm template …
as a more powerful and flexible alternative, leveraging FluxCD Kustomize
resource. Using Kustomize directly within your git
repository, you gain greater control, visibility, and the benefits of Git version control for reviewing and auditing changes.
Ultimately, by adopting a Kustomize-based approach within your FluxCD workflows, you can achieve a more declarative, transparent, and auditable approach to managing your Kubernetes deployments.
Top comments (1)
Good presentation about templating helm charts.
We’ve been using flux and helmrelease quite a lot in past few years, and did not feel any of the inconveniences described here were a problem for us, and we do upgrade helm charts quite frequently.
This is an interesting idea but here are some of the potential limitations with this templates approach…
The resulting rendered files can be quite large and complex and diffs across consecutive version of helm chart can be difficult to apprehend.
Helm chart values often provide a much needed simplified and higher level view of the configuration of the helm chart, and complex deployment use configuration variables (flux substitution variables) to control the behavior of a helm chart through the values section of a helmrelease. This can be extremely difficult to achieve with pre-templates files. Simple example is an enable field in values that includes conditionally certain manifest (eg a configmap or a service …) can no longer provide that flexibility after templating (at this time it is either enabled or disabled)