DEV Community

Darek Dwornikowski ☁
Darek Dwornikowski ☁

Posted on • Updated on

Resizing persistent volumes in EKS/AKS/GKE

#kubernetes #aws #eks #gke

Persistent volumes (PVs) in Kubernetes is the way to abstract out block volumes used by pods in the cluster. The way pods request PVs is by creating Persistent Volume Claims (PVCs), which work like resource demand documents. Storage Class is responsible for actual implementation of how the PVs are actually handled, it can be provided by ceph volumes, or in the case of cloud providers, by EBS volumes in AWS, Managed Disks in Azure and Disks in GCP.

This post is a note made to myself, when I was checking how does the PV resize works in managed kuberentes services. Feel free to replicate the commands and see what happens by yourself.

Amazon Elastic Kubernetes Service

Let's first create the cluster fast with eksctl. Mind it takes a significant time.

eksctl create cluster -n darek -r eu-central-1

After a while you should see some nodes running.

➜  ~ kubectl get nodes 
NAME                                              STATUS   ROLES    AGE   VERSION
ip-192-168-26-190.eu-central-1.compute.internal   Ready    <none>   67s   v1.13.7-eks-c57ff8
ip-192-168-55-249.eu-central-1.compute.internal   Ready    <none>   67s   v1.13.7-eks-c57ff8
Enter fullscreen mode Exit fullscreen mode

Let's see the storage class.

➜  ~ kubectl get sc gp2 -o yaml 
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"gp2"},"parameters":{"fsType":"ext4","type":"gp2"},"provisioner":"kubernetes.io/aws-ebs"}
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2019-08-24T09:23:35Z"
  name: gp2
  resourceVersion: "281"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2
  uid: d903b89e-c650-11e9-968c-0273bdd9611a
parameters:
  fsType: ext4
  type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
volumeBindingMode: Immediate
Enter fullscreen mode Exit fullscreen mode

To resize volumes, storage class needs to have the allowVolumeExpansion set to true. Fortunatelly, it can be patched.

k patch sc gp2 -p '{"allowVolumeExpansion": true}'

Now, create a basic PVC of size 10Gi and storageclass set to gp2. We will us the same yaml for AKS and GKE later too, save it as pvc.yml.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-volume-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: gp2
Enter fullscreen mode Exit fullscreen mode

Create the PVC.

kubectl apply -f pvc.yml

The PVC should eventually create a PV of size 10, which is reflected in a real EBS volume created in the AWS. The name of the EBS is encoded in Source/VolumeID. You can get it by doing describe operation.

➜  kubectl describe pv/pvc-0262b546-c65a-11e9-8f74-06e2726fa54c
Name:              pvc-0262b546-c65a-11e9-8f74-06e2726fa54c
Labels:            failure-domain.beta.kubernetes.io/region=eu-central-1
                   failure-domain.beta.kubernetes.io/zone=eu-central-1b
Annotations:       kubernetes.io/createdby: aws-ebs-dynamic-provisioner
                   pv.kubernetes.io/bound-by-controller: yes
                   pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      gp2
Status:            Bound
Claim:             default/test-volume-pvc
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          10Gi
Node Affinity:
  Required Terms:
    Term 0:        failure-domain.beta.kubernetes.io/zone in [eu-central-1b]
                   failure-domain.beta.kubernetes.io/region in [eu-central-1]
Message:
Source:
    Type:       AWSElasticBlockStore (a Persistent Disk resource in AWS)
    VolumeID:   aws://eu-central-1b/vol-056a0bc0115292add
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
Events:         <none>
Enter fullscreen mode Exit fullscreen mode

The above command shows the EBS volume name to be vol-056a0bc0115292add.
Now let's resize the PVC. Change the storage size in pvc.yml to 12Gi and do kubectl apply again.

➜  kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-0262b546-c65a-11e9-8f74-06e2726fa54c   12Gi       RWO            Delete           Bound    default/test-volume-pvc   gp2                     19h
Enter fullscreen mode Exit fullscreen mode

Ok so the PV shows 12Gi. Let's check the size of the actual EBS volume with aws cli.

aws ec2 describe-volumes --volume-ids vol-056a0bc0115292add --query 'Volumes[0].Size'

It should now show 12.

Have in mind that in Amazon you can do resize operation on EBS only once every 6h. So next resize won't work, unless you wait for it.

You can delete the clutser by doing eksctl delete cluster -n darek.

Azure AKS

Let's now check AKS. Create a cluster in AKS. The below snippet creates a resource group darek and a cluster called darekEKS.

az group create --name darek --location westeurope
az aks create --resource-group darek --name darekEKS --node-count 2 --enable-addons monitoring --generate-ssh-keys
az aks get-credentials --resource-group darek --name darekEKS
Enter fullscreen mode Exit fullscreen mode

Storage class in AKS is called default, it will provision azure managed disk.

➜  kubectl get sc
NAME                PROVISIONER                AGE
default (default)   kubernetes.io/azure-disk   13m
managed-premium     kubernetes.io/azure-disk   13m
Enter fullscreen mode Exit fullscreen mode

Let's see the whole class, similarly as in EKS the resize is not enbled.

➜  kubectl get storageclasses.storage.k8s.io default -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1beta1","kind":"StorageClass","metadata":{"annotations":{"storageclass.beta.kubernetes.io/is-default-class":"true"},"labels":{"kubernetes.io/cluster-service":"true"},"name":"default","namespace":""},"parameters":{"cachingmode":"ReadOnly","kind":"Managed","storageaccounttype":"Standard_LRS"},"provisioner":"kubernetes.io/azure-disk"}
    storageclass.beta.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2019-08-24T09:31:20Z"
  labels:
    kubernetes.io/cluster-service: "true"
  name: default
  resourceVersion: "459"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/default
  uid: ee21d5ae-c651-11e9-9f0b-aac03b918570
parameters:
  cachingmode: ReadOnly
  kind: Managed
  storageaccounttype: Standard_LRS
provisioner: kubernetes.io/azure-disk
reclaimPolicy: Delete
volumeBindingMode: Immediate
Enter fullscreen mode Exit fullscreen mode

Let's patch it: kubectl patch sc default -p '{"allowVolumeExpansion": true}'
And create the PVC, remember to change the storageClassName to default.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-volume-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: default
Enter fullscreen mode Exit fullscreen mode

Is PVC and PV created?

➜  kubectl get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
test-volume-pvc   Bound    pvc-eb332309-c654-11e9-a0fd-36761452f1dd   10Gi       RWO            default        26s

➜  kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-eb332309-c654-11e9-a0fd-36761452f1dd   10Gi       RWO            Delete           Bound    default/test-volume-pvc   default                 41s
Enter fullscreen mode Exit fullscreen mode

The name of the Azure disk will be seen in Source/DiskURI.

➜ ~ kubectl describe pv
Name:            pvc-eb332309-c654-11e9-a0fd-36761452f1dd
Labels:          <none>
Annotations:     pv.kubernetes.io/bound-by-controller: yes
                 pv.kubernetes.io/provisioned-by: kubernetes.io/azure-disk
                 volumehelper.VolumeDynamicallyCreatedByKey: azure-disk-dynamic-provisioner
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    default
Status:          Bound
Claim:           default/test-volume-pvc
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        12Gi
Node Affinity:   <none>
Message:
Source:
    Type:         AzureDisk (an Azure Data Disk mount on the host and bind mount to the pod)
    DiskName:     kubernetes-dynamic-pvc-eb332309-c654-11e9-a0fd-36761452f1dd
    DiskURI:      /subscriptions/6690b014-bdbd-4496-98ee-f2f255699f70/resourceGroups/MC_darek_darekEKS_westeurope/providers/Microsoft.Compute/disks/kubernetes-dynamic-pvc-eb332309-c654-11e9-a0fd-36761452f1dd
    Kind:         Managed
    FSType:
    CachingMode:  ReadOnly
    ReadOnly:     false
Events:           <none>
Enter fullscreen mode Exit fullscreen mode

Now change the disk size to 12 and see. We can check the disk size with the following command.

> az disk  show --ids /subscriptions/6690b014-bdbd-4496-98ee-f2f255699f70/resourceGroups/MC_darek_darekEKS_westeurope/providers/Microsoft.Compute/disks/kubernetes-dynamic-pvc-eb332309-c654-11e9-a0fd-36761452f1dd --query 'diskSizeGb'
12
Enter fullscreen mode Exit fullscreen mode

Ok works in AKS too. What is quite weird, when you do kubectl get pv it shows the new size but the PVC still shows 10Gi.

Clean resources you have created: az group delete -n darek -y

Google Kubernetes Engine

I created a zonal cluster using the CLI in my project called turnkey-cooler-31343.

gcloud config set project turnkey-cooler-31343
gcloud container clusters create darek --zone europe-north1-a --cluster-version "1.13.7-gke.19" --machine-type n1-standard-1 --num-nodes 2
Enter fullscreen mode Exit fullscreen mode

And generate a kubeconfig:

gcloud container clusters get-credentials darek --zone europe-north1-a --project turnkey-cooler-31343
Enter fullscreen mode Exit fullscreen mode

In GKE the default storageclass is called standard.

NAME                 PROVISIONER            AGE
standard (default)   kubernetes.io/gce-pd   3m44s
Enter fullscreen mode Exit fullscreen mode

Just as in AKS and EKS the auto expand was not enabled, let's patch the storage class.

kubectl patch sc standard -p '{"allowVolumeExpansion": true}'

And create a pvc using our PVC yaml. Remember to change storageClassName to standard.

kubectl apply -f pvc.yml

And check the disk:

kubectl get pv 
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-f79ca9c5-c750-11e9-911b-42010aa601a8   10Gi       RWO            Delete           Bound    default/test-volume-pvc   standard                8m30s
Enter fullscreen mode Exit fullscreen mode

Now check the name of the disk in GCP:

kubectl get pv/pvc-f79ca9c5-c750-11e9-911b-42010aa601a8 -o jsonpath='{.spec.gcePersistentDisk.pdName}'
gke-darek-a2e51c42-dyn-pvc-f79ca9c5-c750-11e9-911b-42010aa601a8
Enter fullscreen mode Exit fullscreen mode

And the size of the disk.

➜  gcloud compute disks describe gke-darek-a2e51c42-dyn-pvc-f79ca9c5-c750-11e9-911b-42010aa601a8 --zone europe-north1-a | grep sizeGb
sizeGb: '10'
Enter fullscreen mode Exit fullscreen mode

Let's now resize the disk, change the pvc.yml or patch the PVC.

➜  gcloud compute disks describe gke-darek-a2e51c42-dyn-pvc-f79ca9c5-c750-11e9-911b-42010aa601a8 --zone europe-north1-a | grep sizeGb
sizeGb: '12'
Enter fullscreen mode Exit fullscreen mode

Ok all fine but as in AKS and EKS the PVC still shows 10Gi.

Clean the infra: gcloud container clusters delete darek --zone europe-north1-a.

Top comments (0)

Read next

hulk-pham profile image

AWS Global Infrastructure

Hulk Pham -

akhil_mittal profile image

Security Posture for Production Grade K8 Cluster

akhil mittal -

hoangguruu profile image

DevOps Beginner | Day 108 | Use EC2 Ubuntu Setup Jenkins With User Data

Hoang Guruu -

uendi_hoxha profile image

Terraform vs. AWS CloudFormation: A Detailed Comparison

Uendi Hoxha -