Using GitOps for Infrastructure and Applications With Crossplane and Argo CD

If you have been following the Codefresh blog for a while, you might have noticed a common pattern in all the articles that talk about Kubernetes deployments. Almost all of them start with a Kubernetes cluster that is already there, and then the article explains how to deploy an application on top.

The reason for this simplification comes mainly from brevity and simplicity. We want to focus on the deployment part of the application and not its infrastructure just to make the article easier to follow. This is the obvious reason.

The hidden reason is that until recently infrastructure deployments were handled in a different manner than applications deployments. Especially in large enterprise companies, the skillset of people that deal with infrastructure and application can vary a lot as the tools of the trade are completely different.

For example, a lot of people that deal with infrastructure prefer to use Terraform templates, but employ Kustomize/Helm or other similar tools for application development. While this is a very valid solution, it doesn’t have to be this way.

Now with GitOps you can have a uniform way of dealing with infrastructure and applications.

GitOps and Terraform

If you are not familiar with GitOps, head over to https://opengitops.dev/ which is the official page of the GitOps working group. The principles of GitOps are the following:

1. The system is described in a declarative manner. (In practice, this means Kubernetes manifests.)
2. The definition of the system is versioned and audited. (In practice, it is stored in Git.)
3. A software agent automatically pulls the Git state and matches the platform state. (In practice, this means Flux/ArgoCD.)
4. The state is continuously reconciled. This means that any changes happening in Git should also be reflected in the system, as well as the opposite scenario.

If you have already worked with Terraform, you should already understand why it is difficult to apply GitOps to the Terraform CLI. While Terraform only has the first requirement (declarative format), it doesn’t satisfy the other three requirements around Git storage, automatic reconciliation, and two-way sync. As soon as Terraform finishes its job, it doesn’t interface with the system in any way. If you manually delete a Virtual Machine that was created by Terraform, vanilla Terraform doesn’t know about it. And regarding state, Terraform stores its own state which is completely different from the definition files in Git.

So at first glance, getting GitOps to work with infrastructure is a complicated task. You realize that you need to add something on top of vanilla Terraform in order to satisfy the GitOps requirements.

But there is a new kid on the block, and that is Crossplane!

GitOps and Crossplane

Crossplane has similar capabilities to Terraform (creating infrastructure) but with the following major differences:

1. Crossplane itself is a Kubernetes application. (The infrastructure it creates can be anything.)
2. Crossplane definitions are Kubernetes manifests.
3. You can either use manifests that describe resources in the common cloud providers or create your own resources.

As a simple example here is an EC2 instance describe in Crossplane:

apiVersion: ec2.aws.crossplane.io/v1alpha1
kind: Instance
metadata:
  name: sample-instance
spec:
  forProvider:
    region: us-east-1
    imageId: ami-0dc2d3e4c0f9ebd18
    securityGroupRefs:
      - name: sample-cluster-sg
    subnetIdRef:
      name: sample-subnet1  
  providerConfigRef:
    name: example

You can find more examples for Amazon, Google, and Azure. Crossplane supports several other providers, and of course you can add your own.

The important point here is that the file above is a standard Kubernetes manifest. You can:

• Apply it with kubectl.
• Verify it with manifest verification tools (kubeval or kube-linter).
• Template it with Helm/Kustomize.
• Use any of the tools in the Kubernetes ecosystem to read/manage/store it.

And since it is a standard manifest, you can of course store it in Git and manage it with ArgoCD for a Full GitOps workflow. The importance of this capability cannot be overstated.

If you combine ArgoCD and Crossplane, you have a full solution for following the GitOps principles with infrastructure and not just applications. Imagine if your ArgoCD dashboard contained this:

Isn’t that cool?

How ArgoCD and Crossplane work together

Crossplane offers an easy way to model your infrastructure as Kubernetes manifests. This is great on its own, but if you put ArgoCD in the mix, you essentially gain all the advantages of GitOps for your infrastructure.

• You know exactly what infrastructure you have simply by looking at the Git repository.
• You know exactly what was changed and when by looking at Git history.
• The infrastructure state IS the same as the Git state. Terraform has its own state that is used a the single source of truth
• You don’t need any external credentials any more.
• You can easily rollback to a previous version of your infra with a git reset/git-revert. You completely avoid the dreaded configuration drift.

The last point is very critical. Terraform only knows what is in your infrastructure during deployment. If you make any manual changes afterwards (e.g. delete some infra), terraform knows absolutely nothing. You need to rerun terraform and pray that the correct action is taken. There are many terraform horror stories about incomplete/invalid states. Here is one of my favorite tfstate stories from Spotify.

ArgoCD will instantly detect any manual changes in your infrastructure, present you a diff, and even allow you to completely discard them if you want.

Essentially ArgoCD is agnostic as to what exactly is described by the Kubernetes manifests it manages. They can be plain Kubernetes applications, Virtual machines, Container registries, Load balancers, object storage, firewall rules, etc.

Create infrastructure and deploy to it using GitOps

Now that we saw that you can use ArgoCD and Crossplane together for managing infrastructure, we are now ready to treat applications and the platform they need in the same way.

This means that we can do the following:

1. Start from nothing.
2. Use Crossplane to create a Kubernetes cluster.
3. Commit the crossplane manifest and manage it with ArgoCD.
4. Use a standard manifest (e.g. deployment) to deploy an application to the cluster that was just created.
5. Commit that manifest in Git too. Like the application one, ArgoCD will manage it like any other Kubernetes manifest (even though it represents infrastructure).

Here is the whole workflow:

The Kubernetes cluster that Crossplane is running on is only used to bootstrap crossplane. It doesn’t get any production workloads on its own. For simple demos, this can be any cluster (even a local one running on your workstation).

The end result is that now you have a unified way to handle both infrastructure and applications right from Argo CD.

The process of changing either of them is exactly the same.

• If you want to change infrastructure, you commit a change and ArgoCD takes care of it (with the help of Crossplane).
• If you want to change your application, you commit a change and ArgoCD takes care of it.

They also both gain all the benefits of GitOps. If for example somebody changes the number of replicas in the deployment or tampers with the cluster nodes, ArgoCD will detect the manual changes automatically and give you the ability to discard them completely.

Why GitOps is the way forward

The whole point of DevOps is to make everything self-service and promote collaboration between developers and operators. Adopting Crossplane in a GitOps setting means that now both parties have a common language that they can communicate with.

There is no need anymore for separate workflows that are confusing to either of them. Adopting a common workflow for both infrastructure and applications is the embodiment of the DevOps spirit.

If you want to learn more about GitOps and how Codefresh has embraced it, check out the Codefresh DevOps platform for Argo. For more information about Crossplane, see the official site and the hosted solution by Upbound.