Introduction
Go through any GitOps tutorial (including some written by myself) and you will learn how to set up a single environment that works perfectly with you GitOps tool of choice. However, in reality you will most likely need more than a single environment. You might have a development environment, a staging environment, and a production environment. How to promote a release from development, to staging, and finally to production, is something that most tutorials gloss over (including mine).
In this post I will sketch out a possible solution on how to solve environment promotions. This post takes a lot of inspiration from a great Codefresh article, but tries to fill in details of how exactly to do this using GitHub Actions and Helm-charts instead of plain Kubernetes manifests with some Kustomize overlays.
To keep this post light and avoid all of the production-scenario complications I will restrict myself to the following:
- I will not include an actual application with source code and a continuous integration pipeline.
- I will include the configuration repository (config repo) which contains Kubernetes manifests in the shape of a simple Helm chart.
- I will restrict myself to using three environments: development, staging, production.
- I will not discuss the need for hot-fixes directly into production, I will assume all the releases go through development, to staging, and finally to production.
- I will start off with a working Kubernetes cluster with Argo CD installed in the
argocd
namespace. Tutorials of how to create a Kubernetes cluster and install Argo CD is available online (including some written by myself)1. - I will use the same Kubernetes cluster for all my environments, but I will separate them into their own namespaces.
- I will use the same Argo CD project (the one named
default
) for all my environments. I will not include any additional Argo CD concepts such as RBAC. - My Argo CD applications will all just care about the
main
branch in my repository. There is no need to separate the different environments into different branches if you follow the structure outlined in this post.
Configuration repository
My application consists of a simple Helm chart with a single Kubernetes deployment. The application could theoretically be however complicated it must be, but I don't want to spend too much time explaining what the application consists of. The structure and content of the repository looks like this:
.
├── .github
│ └── workflows
│ ├── promote-to-production.yaml
│ └── promote-to-staging.yaml
├── app
│ ├── Chart.yaml
│ ├── environments
│ │ ├── development
│ │ │ ├── values.yaml
│ │ │ └── version.yaml
│ │ ├── production
│ │ │ ├── values.yaml
│ │ │ └── version.yaml
│ │ └── staging
│ │ ├── values.yaml
│ │ └── version.yaml
│ └── templates
│ └── deployment.yaml
└── gitops
├── development.yaml
├── production.yaml
└── staging.yaml
In this section I will go through the details of what is in the app
and gitops
directory. In the next section I will go through what is in the .github
directory.
I will not bother go through the Chart.yaml
file for Helm, it is not of particular interest here. The manifest for my deployment is shown in the following code snippet:
# app/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-deployment
spec:
replicas: {{ .Values.deployment.replicas }}
selector:
matchLabels:
app: my-application
template:
metadata:
name: my-pod
labels:
app: my-application
spec:
containers:
- name: webserver
image: {{ .Values.deployment.image.name }}:{{ .Values.deployment.image.tag }}
There are three things I will vary between my environments:
- The number of
replicas
in the deployment. This value is expected to be different for each environment, but should not be updated frequently. - The container image
name
andtag
. Thetag
value is expected to change frequently.
For each environment I will use two Helm value files, they are located in the corresponding environment directory app/environments/development
, app/environments/staging
, or app/environments/production
:
-
values.yaml
contains differences between the environments that are not expected to be promoted from one environment to the next. In this case it just contains the number of replicas in the deployment. For the development environment this files looks like this:
# app/environments/development/values.yaml
deployment:
replicas: 1
-
version.yaml
contains changes that should be promoted from one environment to the next. In my example it contains the name and tag of the container image. To start off, the file looks the same for all environments:
# app/environments/development/version.yaml
# app/environments/staging/version.yaml
# app/environments/production/version.yaml
deployment:
image:
name: nginx
tag: 1.22.0
The last set of files in my repository is located in the gitops
directory. These are the Argo CD applications for each environment. I would not necessarily include them in the same repository, but in this case I chose to do that. The Argo CD application for the development environment looks like the following:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: development-application
namespace: argocd
spec:
project: default
source:
repoURL: <my github repository url>
path: app
targetRevision: main
helm:
valueFiles:
- environments/development/values.yaml
- environments/development/version.yaml
destination:
server: "https://kubernetes.default.svc"
namespace: development
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
The applications for the staging and production environments look similar.
Automate environment promotion with GitHub Actions
So now we have three Argo CD applications, one for each of our environments. To introduce a change in our development environment we would update app/environments/development/version.yaml
with a new value for the container name or tag. Most realistically we would update the tag value. To promote this change from development to staging we would simply copy this change from the development environment and push the change:
$ cp app/environments/development/version.yaml app/environments/staging/version.yaml
$ git commit -am "Promote change to staging" && git push
Likewise, to promote the change from staging to production we do another copy operation and push the change:
$ cp app/environments/staging/version.yaml app/environments/production/version.yaml
$ git commit -am "Promote change to production" && git push
This seems so simple that we should be able to automate it! Exactly how you want to automate it will depend a bit on your circumstances, what kinds of tests and checks you want to make before you promote the change from one environment to the next.
I will show you how to automate it in the following way using GitHub Actions:
- Any change to the development environment is continuously deployed to the development environment without restrictions. A pull-request to promote the change to the staging environment is automatically created.
- Changes to the staging environment requires an approved pull-request, once approved the change is deployed to the staging environment. A pull-request to promote the change to the production environment is automatically created.
- Changes to the production environment requires an approved pull-request, once approved the change is deployed to the production environment.
The workflow to promote a change to the staging environment looks like this:
# .github/workflows/promote-to-staging.yaml
name: Promote to staging
on:
push:
branches:
- main
paths:
- app/environments/development/version.yaml
permissions:
contents: write
pull-requests: write
jobs:
promote:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- run: |
# configure git client
git config --global user.email "<email address>"
git config --global user.name "<name>"
# create a new branch
git switch -c staging/${{ github.sha }}
# promote the change
cp app/environments/development/version.yaml app/environments/staging/version.yaml
# push the change to the new branch
git add app/environments/staging/version.yaml
git commit -m "Promote development to staging"
git push -u origin staging/${{ github.sha }}
- run: |
gh pr create \
-B main \
-H staging/${{ github.sha }} \
--title "Promote development to staging" \
--body "Automatically created by GitHub Actions"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
The workflow is triggered whenever there is a change to app/environments/development/version.yaml
on the main
branch. The workflow itself consists of three steps:
- Check-out the source code (since we need to update it!)
- Configure git, create a new git branch, perform the copy operation, push the change to the new branch
- Use the GitHub CLI to create a pull request for the new change
We could run additional automation to perform tests in our development environment before we go on to approve the promotion to the staging environment.
Once the change is approved a new workflow is triggered to initiate the promotion to the production environment:
# .github/workflows/promote-to-production.yaml
name: Promote to production
on:
pull_request:
types:
- closed
paths:
- app/environments/staging/version.yaml
permissions:
contents: write
pull-requests: write
jobs:
promote:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- run: |
# configure git client
git config --global user.email "<email address>"
git config --global user.name "<name>"
# create a new branch
git switch -c production/${{ github.sha }}
# promote the change
cp app/environments/staging/version.yaml app/environments/production/version.yaml
# push the change to the new branch
git add app/environments/production/version.yaml
git commit -m "Promote staging to production"
git push -u origin production/${{ github.sha }}
- run: |
gh pr create \
-B main \
-H production/${{ github.sha }} \
--title "Promote staging to production" \
--body "Automatically created by GHA"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
The difference in this workflow is just the trigger2. This workflow is triggered when a pull-request has been closed where there were changes to app/environments/staging/version.yaml
.
Before we approve the merge request to promote the change to production we can run additional automation against our staging environment. When we are ready we can approve the pull-request and the change will be promoted to production.
Note that there are some additional work required to lock down how changes can be promoted. There should be some rules in place that restricts changes directly to the staging and production environments. I might continue my work on this example in future posts, because there are a lot more that could be said. For now I will leave this example!
Top comments (0)