Dynamic Volume Provisioning in Kubernetes with AWS and Terraform

Introduction

In this article, we will be dealing with understanding Persistent Volumes in Kubernetes, how they are provisioned, how they are managed, and how they are configured in AWS. We will do a comparative analysis between Static Volume provisioning and Dynamic volume provisioning, how they overlap, and which provisioning mechanism you should employ. Finally, we will provision a Kubernetes cluster in AWS using Terraform and then setup dynamic provisioning with the cluster, we will test the validity of our setup by attempting to install a helm chart that will require a reasonable amount of storage, at first we will fail due to a lack of sufficient storage, and then after sufficient setup, we will succeed. This article will require a basic understanding of Kubernetes, helm, and AWS. You can find all code examples here.

Dynamic vs Static provisioning

Understanding when to use dynamic or static volume provisioning is very vital depending on the type of application you are trying to build. If you want to pre-populate data in a volume, then you choose static provisioning however, if you want to create volumes on demand and then delete or retain after a successful run, then you should make use of Dynamic provisioning. Another difference is that in the Static provisioning workflow, you have to explicitly define and provision a PersistentVolume while in Dynamic provisioning, Kubernetes handles the process of creating a PersistentVolume and binding that Persistent volume to the necessary components.

Again, The major difference is having to explicitly create a PersistentVolume, as against allowing Kubernetes to create a PersistentVolume whenever it needs it. Additionally, dynamic provisioning abstracts the implementation details from your workloads, this means you can switch storage providers depending on the workload type; whether you are running on-premise, on the public cloud, or hybrid. Another important detail to note in Dynamic provisioning is that when the PersistentVolumeClaim is deleted, the volume is also deleted. However, this setting can be adjusted even after the Persistent Volume is created.

Dynamic Provisioning in AWS

In this article, I will assume you have a properly setup Kubernetes cluster however, if you do not have that running I have provided a Terraform project to help you quickly set up a Kubernetes cluster in AWS here, all you have to do is set up the right variables and you are good to go. Finally, ensure that your IAM user has the right permissions to set up a Kubernetes cluster as that is very key.

In order to demonstrate volume provisioning, we will attempt to install the Drupal chart on the bitnami helm repo. I am also assuming that you have helm installed, you don't have it installed, Install Helm is a good place to start.

At first, we are going to attempt to install the Drupal chart by running helm install mysite bitnami/drupal , this command should run successfully. However, when we start debugging the pods we will discover something interesting.

The helm installation is successful as helm only verifies that the yaml files provided are accepted by Kubernetes, to check the status of the deployments, we run kubectl get pods

We realize that our pods are stuck in a Pending state, so we take the next step in debugging, which is to run kubectl describe for each of the created pods

The actual reason that our pods are not coming up is found when we review the helm installation that we are trying to run. If you check the dependencies in the GitHub repository (https://github.com/bitnami/charts/blob/main/bitnami/drupal/values.yaml) you find out that persistent storage is enabled by default and set to 8Gi. Also, the helm package uses MariaDB and the database size is specified to a default of 8Gi, thus setting the minimum storage for this installation to be 16Gi.

The obvious reason for the failure of Kubernetes is that our pods do not have the exclusive rights to provision storage on demand, which is Dynamic provisioning, Let's look at a couple of steps that will give our pods this ability and will enable the deployment to be successful.

The first step here is to create an IAM OpenID connect issuer (OIDC) provider and attach it to your cluster, you can find further documentation on OIDC issuers here. To do this, we run the following command,

eksctl utils associate-iam-oidc-provider --cluster my-cluster --approve

Here my-cluster is your cluster name, My cluster name is article-helm 😂, so don't freak out if you see it in the commands.

The next step is to create a service account and attach the EBSDriverPolicy to it. We use the eksutil command to create the IAM role, and attach the IAM policy to it. We will do this with the following command.

eksctl create iamserviceaccount \
  --name ebs-csi-controller-sa \
  --namespace kube-system \
  --cluster my-cluster \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve \
  --role-only \
  --role-name AmazonEKS_EBS_CSI_DriverRole

This should create a cloud formation stack to create the service account and attach the necessary policies. You can find more documentation on this process here.

Finally, we will add the Amazon EBS CSI driver as an add-on to our cluster, we can do this by running the following command,

eksctl create addon --name aws-ebs-csi-driver --cluster my-cluster --service-account-role-arn arn:aws:iam::111122223333:role/AmazonEKS_EBS_CSI_DriverRole --force

Replace my-cluster with the name of your cluster, 111122223333 with your account ID, and AmazonEKS_EBS_CSI_DriverRole with the name of the IAM role created earlier. Find additional documentation on this here

Our cluster is fully set up with the right policies to request additional storage from AWS, whenever Kubernetes needs it.

The next step is to create a storage class and then ask Helm to refer to our storage class when it is asking for permission to set up additional storage for Kubernetes. I have included the details for the storage class in the official article repo, you can find it in yaml/storage-class.yaml.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: helm-storage
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
reclaimPolicy: Delete
allowVolumeExpansion: true 

We set up this storage class declaratively by running kubectl apply -f storage-class.yaml.

When we run kubectl get sc to retrieve all the storage classes, we realise that we have two default storage class.

To rectify this, we patch the default storage class and remove the default flag on it, we do this by running the following command

kubectl patch storageclass gp2 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

Now, we have just one default storage class, and we are ready to go!

If you have followed the instructions to the teeth, your storage class should be all set, and dynamic provisioning should be enabled on your cluster.

Running our helm installation successfully

Now that we are done with the necessary setup we can go on to install our helm chart successfully. I have provisioned another values.yaml file for our installation, this is to overrun some of the defaults that the bitnami/drupal installation covers. You can check the values.yaml file at /yaml/values.yaml in the article repository. I set all the storage class names to point to helm-storage and then I reduced the total volume required to 5Gi, so as not to blow up cost.

We set up our new installation by running the following command,

helm install mysite bitnami/drupal --values values.yaml 

Now, we check that our pods are running,

We also verify that Kubernetes has used our cluster to provision the necessary PersistentVolumes without us explicitly creating the volumes.

Finally, to get our load balancer service link and test the connection in a web browser, we run the following command

kubectl get svc --namespace default -w mysite-drupal

Then we visit the load balancer link in the browser to see that our service is set up and running efficiently.

Voila! We have now enabled Dynamic provisioning for our Kubernetes cluster.

Conclusion

In this article, we worked on the concepts of Dynamic and Static volume provisioning in the Kubernetes cluster, we also looked at scenarios where you should use dynamic volume provisioning or static volume provisioning. Finally we set up a cluster with dynamic provisioning and then we deployed a helm chart with high storage requirements. Thank you for reading this article, Please leave a comment and star the repository here. Best regards.