If you have a kubernetes cluster running in AWS, you will most likely need to update the EKS control plane and data plane regularly, according to the AWS end of support schedule for Kubernetes versions. Updating an EKS Cluster might not be the most straight forward thing ever, and sometimes it can get even a little bit convoluted. I had to do it a clouple of times last year, and thats why my team and I came with the following the steps, which I’m sharing here in case it might help anyone plannig the update and avoiding most of the potential stress in the aftermath.
Remember: if you don't do the update by yourself, AWS will do it by force at anytime after the official end of support. As stated by their official documentation:
Amazon EKS can’t provide specific time frames [for the forced update]. Automatic updates can happen at any time after the end of support date. You won’t receive any notification before the update. We recommend that you proactively update your control plane without relying on the Amazon EKS automatic update process. For more information, see Updating an Amazon EKS cluster Kubernetes version.
1 - Plan it beforehand
Planning ahead is the only way to avoid unnecessary extra work. AWS provides extensive documentation on the end of support for every EKS version. You can check it out here.
Based on my experience, I would say you must try to have it updated at least three months before the official Amazon end of support deadline. This way your team will have plenty of time to test the new K8S version on a controlled environment before going live on production.
2 - List your cluster tools and update them accordingly
Nobody wants incompatiblility surprises after an update, right? So the best approach is to list every support software you run on your cluster, like a service mesh or observability softwares, and check if the new K8S version you are about to install is compatible.
For example, imagine you have Istio 1.17 (service mesh) and Kiali 1.63.2 (observability tool) in your cluster. Istio 1.17 is compatible with Kubernetes up to 1.26. However, it doesn’t support Kubernetes 1.27, so you have to update Istio to 1.18 before going to K8S 1.27, and also update Kiali to 1.67.0.
Map everything and update accordingly.
3 - Get a picture of your current Kubernetes Events and Nodes
In order to know what to expect after the update, you have to know how is your system working before it happens. To do this, you should first check the status of the events in your main K8S namespaces, and the health of you pods. Use the kubectl commands bellow in your terminal to get the info and take note of everything:
kubectl get events -n (example: kubectl get events -n kube-system)
kubectl get pods -A
It’s a simple thing, but it can help you a long way after the update is done.
4 - Check deprecated/removed Kubernetes APIs
Before you update your cluster, check out the list of Kubernetes APIs that are going to need an update or are going to be completely removed in the next K8S version.
But how do I check which API needs to be updated?
The traditional way consists of reading the Kubernetes official documentation (here) and going through your manifests one by one, updating when needed.
A quicker way of checking which API needs to be updated in your cluster is by using the kubent tool (a.k.a. Kube no Trouble). This tool searches your cluster and shows every API that need attention prior to the EKS update. Here is an example of how it runs straight from the Kube No Trouble website:
$./kubent
6:25PM INF >>> Kube No Trouble `kubent` <<<
6:25PM INF Initializing collectors and retrieving data
6:25PM INF Retrieved 103 resources from collector name=Cluster
6:25PM INF Retrieved 0 resources from collector name="Helm v3"
6:25PM INF Loaded ruleset name=deprecated-1-16.rego
6:25PM INF Loaded ruleset name=deprecated-1-20.rego
__________________________________________________________________________________________
>>> 1.16 Deprecated APIs <<<
------------------------------------------------------------------------------------------
KIND NAMESPACE NAME API_VERSION
Deployment default nginx-deployment-old apps/v1beta1
Deployment kube-system event-exporter-v0.2.5 apps/v1beta1
Deployment kube-system k8s-snapshots extensions/v1beta1
Deployment kube-system kube-dns extensions/v1beta1
__________________________________________________________________________________________
>>> 1.20 Deprecated APIs <<<
------------------------------------------------------------------------------------------
KIND NAMESPACE NAME API_VERSION
Ingress default test-ingress extensions/v1beta1
Pretty cool! With this info you will be wasting less time updating your K8S API’s. The download of kubent can be found on their official Github page.
It’s recommended to do this step just before you update your controlplane, so the problems with removed or deprecated API’s in the “previous” K8S version are minimized.
5 - Check your cluster Add-Ons
We need to take notes before updating the controlplane, so we can adjust the add-ons accordingly after updating the controlplane.
Usually, the most common Add-Ons are the VPC CNI, CoreDNS, Kube-proxy and recently the Amazon EBS CSI driver, but there are a couple of others. Check the AWS documentation and write down the respective Add-On versions compatible with your upcoming K8S update.
6 - Choose an AMI and create your launch template
Since we are still not taking advantage of the advent of Serverless AWS, either by choice or by cost, we will need to choose a new AMI for our EC2 instances that can work with the new K8S version with full compatibility. If your company doesn’t provide an official image, you can search for one in the Community AMI’s, buy one in the AMI’s Marketplace, or even create/update your own using Amazon EC2.
With the AMI at hand, the next step is to create your launch template. This is done by going into the Launch Template option on the AWS console, choosing your current Launch Template, and choosing the option to Create/Modify the LT. Always provide a description for your new template and select the AMI before checking if the other parameters are OK, and then create the template.
7 - Finaly, update your Cluster
It’s time! Now we will get the notes we took and update the Cluster. I’m going to assume that by now you have already updated the Cluster tools, like Istio for example (but if you haven’t, go back to Step 2 and do it). So follow the process:
a) Use the notes taken on Step 4 to update the Kubernetes APIs;
b) Access your EKS Cluster, choose “Update Now” and select the new Kubernetes version, and wait until the Cluster is available;
c) With the cluster in the Available state, go to the Add-Ons tab, update your add-ons using the notes you took on Step 5 and wait for them to be available (in case they change to the Tainted state, read the logs provided and treat accordingly before retrying);
d) Go to the Compute tab and change the Launch Template of your node to the version created on Step 6 and wait for the completion.
If your cluster is taking to long to update: do a “kubectl get nodes” in your terminal, write down which nodes are still pending, go to the EC2 page and terminate all the instances related to the pending nodes;
e) After everything is updated, do the same thing you did in Step 3 and compare the “After” picture with the “Before” one, and troubleshoot accordingly if needed.
I hope you find this little guide useful. I know there is vast documentation on the internet about this topic, and I linked some of it in this article, but I wanted to make something that could look all the steps from above instead of specifically aproaching each one. Please feel free to point out anything that might need adjustments. Good luck on your journey!
Top comments (0)