Deploy a Kubernetes App & AWS Resources using Crossplane on Kubernetes: Part 2

To properly enjoy this article

This tutorial assumes you already followed the steps in part 1: Deploy AWS Resources using Crossplane on Kubernetes.

Also, this is the Github Repository we’ll be using:
https://github.com/MeteorOps/crossplane-aws-provider-bootstrap

What’s this article about?

In this article, we’ll cover a use-case that can benefit from Crossplane: full environment deployment.

This is a step-by-step guide with an example and a Git repository, so by the end of it, you should be able to deploy a sample env.

You can technically walkthrough the entire thing by "copy-paste" and everything should work. But, diving into the explanations with an extra 5–10 minutes will leave you with longer-term value.

Hope you enjoy!

What to expect from this article?

By the end of it, you’ll understand:

1. How Crossplane can be used for full environment deployment
2. How to deploy a sample app with AWS resources

What not to expect?

This article guides you through a simple application deployment, and not a full set of apps.

It also doesn’t go into using Crossplane in conjunction with Helm, but does cover important principles regarding it.

Why Crossplane for the Full Environment Use-Case?

When you want to deploy a full environment, it usually involves 3 layers:

1. Infrastructure: Resources the application needs to run well
2. Application: The programs built by the company to serve users
3. Data: The data the application uses

But you already know that.

The thing is a tradition developed, and Crossplane sort of broke this tradition.

The tradition was this process: Build infrastructure, Deploy application on top.
How did Crossplane break this tradition?
The application deployment can now provision infrastructure required by the application.

Pull-Request Environments are also easier

By creating a namespace with all of the apps and the AWS resources required with Crossplane, the use-case of creating a full environment per Pull-Request as part of the CI becomes much easier.

That's a nice benefit of such setup for companies utilizing the feature-branch or Gitflow approaches.

A Traditional Full Env Example

To provision and deploy a full environment in the past, the process would generally look something like this:

1. Provision VPC+EKS+... using Terraform
2. Use Terraform to bootstrap the cluster with a CD tool (e.g., ArgoCD)
3. ArgoCD looks at a repo that deploys all apps from there
4. An application needs a new S3 Bucket, so the developer writes Terraform code for it
5. The application gets removed after a while (but the bucket stays)
6. Someone needs to remember that bucket was owned by that app and remove it from Terraform

A Crossplane Full Env Example

To provision and deploy a full environment with Crossplane the process is similar (we still need a Kubernetes Cluster to start with for the initial environment):

1. Provision VPC+EKS+... using Terraform
2. Deploy Crossplane’s prerequisites to the cluster with Terraform
3. Add Crossplane resources to application Helm Charts (so they get their required infra upon deployment)
4. Create a Crossplane manifest to deploy the Helm Charts + Some shared infra required by all apps
5. When an application is removed, its AWS resources are gone with it
6. When an entire environment is terminated, its AWS resources are gone with it

Crossplane in Helm vs. Helm in Crossplane

When using Crossplane alongside Helm, the question arises:

Should Helm apply the Crossplane code? Or, should Crossplane apply the Helm Charts?
I'm glad you asked it - the answer is both, depends when.

Reasons for Crossplane in Helm:

1. Create or modify app-specific resources when that app is deployed
2. Delete app-specific resources when that app is deleted

Reasons for Helm in Crossplane:

1. Manage dependencies between resources and applications using Crossplane
2. Create shared resources that are not owned by a single application

The Step-by-Step Guide

Deploy the simple application alongside a S3 bucket using a Crossplane Composite Application.

Before proceeding

Make sure you follow the steps in the 1st article (takes 3-minutes to just copy-paste the code snippets into your terminal and run the entire thing).

Deploy the Crossplane Kubernetes Provider

Prepare the AWS Credentials for the Application to be able to use AWS
Run the following oneliner to create the Secret containing the AWS credentials in the right format as required by the Application (the application will simply run aws s3 ls to show the bucket):

kubectl create secret generic aws-creds \
--from-literal=aws_access_key_id=$(grep -i aws_access_key_id creds | awk -F' = ' '{print $2}') \
--from-literal=aws_secret_access_key=$(grep -i aws_secret_access_key creds | awk -F' = ' '{print $2}')

Make sure it was created as expected by fetching the secret:

Deploy the Crossplane Kubernetes Provider resources using the k8s-provider-bootstrap.yaml file

kubectl apply -f k8s-provider-bootstrap.yaml

Make sure the provider was created and is ready before proceeding to the next steps:

kubectl get providers provider-kubernetes

You should see something like this:

Deploy the Crossplane Kubernetes Provider Configuration using the k8s-provider-conf.yaml file:

kubectl apply -f k8s-provider-conf.yaml

This is done separately as it needs to happen after the Provider resources were created.

This is where we tell the Crossplane Kubernetes Provider in which Kubernetes cluster it should operate when it’s creating resources.

Create a deployable unit of an App & AWS Resources using Crossplane

Here we do 3 things with 3 files:

1. The composite-app-xrd file:
   Contains the CompositeResourceDefinition (XRD) for the K8sApplication by using the Composition of a K8s Deployment and S3 Bucket (described below)

2. The composite-app-composition file:
   Contains the Composition definition which creates both the Kubernetes Deployment and the S3 Bucket

3. The composite-app-example file:
   Calls the CompositeResource defined by composite-app-xrd file

Crossplane Resources Files Breakdown & Creation

composite-app-xrd.yaml

~ K8sApplication CompositeResourceDefinition
This defines a composite resource for a Kubernetes application, with bucketName and bucketRegion fields in the spec. Users can claim this resource as K8sApplication.

The K8sApplication CompositeResource (XRD) accepts the bucketName & bucketRegion fields and uses them to create an S3 Bucket, and to create a K8s Deployment of a mock “service” that simply runs aws s3 ls to see the bucket.

~ Deploy the CompositeResourceDefinition (XRD)

kubectl apply -f composite-app-xrd.yaml

composite-app-composition.yaml

Defines a Composition of resources that can be created by a CompositeResource.

This is where we define the Composition that creates a combo of a Kubernetes Deployment with the mock “service” that runs aws s3 ls as well as the S3 bucket — The CompositeResource simply calls this resource.

~ Deploy the Composition

kubectl apply -f composite-app-composition.yaml

composite-app-example.yaml

Deploys the actual K8sApplication CompositeResource, and passes the details of the region in which the bucket should be created, and the name of the bucket (both are also passed to the Kubernetes Deployment as environment variables that helps it access the same bucket).

As mentioned above, the CompositeResource calls the Composition which creates the resources using the Crossplane providers.

Deploy the app by running the following command:

kubectl apply -f composite-app-example.yaml

Look at your pretty Application

Fetch the K8sApplication resource you’ve just created by running the below command obsessively until it’s marked as Healthy:

kubectl get K8sApplication

You should get something like this:

Print the logs of the application and see it fetching the AWS S3 Bucket:

kubectl logs -l app=awscli
# 2024-10-17 16:00:31 my-app-bucket-nqzhx-xzjcq
# 2024-10-17 16:00:50 my-app-bucket-nqzhx-xzjcq

Cleanup

kubectl delete -f composite-app-example.yaml

Recap

To briefly recap what you did here:

1. Prepared Crossplane for deploying a mix of Kubernetes and AWS resources
2. Defined the manifests required to deploy an app built of a Deployment and a S3 Bucket
3. Sharpened your grasp on some Crossplane concepts
4. Discussed some use-cases for which it’s useful

Hope you enjoyed this article, and if you are interested in another article about something related (or unrelated), please convince Michael it’s a good idea at michael@meteorops.com.

_{Disclaimer: In actual environments or production, it’s essential to fine-tune the permissions in the different manifests. Instead of using access keys and secret keys directly, consider implementing IAM Roles for Service Accounts (IRSA) to manage permissions more securely.}