Considering cloud market share held by Amazon, it is very likely that the cloud of your choice is AWS. If that is the case and you have already decided that Kubernetes is the way to schedule your workloads, you are probably asking yourself what cluster hosting approach should you choose. While choosing AWS EKS - the AWS managed Kubernetes offering could be the first thing that comes to your mind - it is worth to know the alternatives and make an educated decision.

The most popular Kubernetes cluster hosting methods for AWS

There are a myriad of different ways to run your Kubernetes cluster in AWS. Let's discuss some of the most popular ones.

Currently the most popular cluster hosting methods for AWS are as follows:

• Kubernetes The Hard Way
• Kubeadm
• KubeSpray
• Kops
• EKS

Kubernetes The Hard Way

Hosting Kubernetes cluster "the hard way" means that the cluster has been bootstrapped by following steps from the famous Kubernetes The Hard Way guide authored by Kelsey Hightower. Kubernetes The Hard Way describes how to manually configure and install each Kubernetes component (like Kubernetes scheduler, Kubernetes API server, Containerd, CoreDNS, CNI and Etcd).

In practice The Hard Way approach requires you to start some EC2 Linux virtual machines (that will serve as your master and worker nodes) and installing each Kubernetes component (Etcd, Containerd, Kubernetes scheduler and so forth) into these Linux servers.

While Kubernetes The Hard Way is a great way learn a ton about Kubernetes internals, it is probably not the best choice for your production Kubernetes cluster. The main reason for that is because with The Hard Way approach all the cluster administration burden is on you and your organization. Creating and configuring all the infrastructure required by Kubernetes The Hard Way cluster is an extremely expensive activity, and I don't recommend following this approach if you don't plan to hire devOps engineers / SREs (Site Reliability Engineers) dedicated for this task.

The good case for The Hard Way approach would be if you would like to run customized version of Kubernetes cluster where you have full control over an every component of your installation. And if you are not afraid to cover the cost associated with this approach.

Kubeadm

Kubeadm is a cluster provisioning tool implementing the best-practices for creating Kubernetes clusters (following Kubernetes The Hard Way steps). Kubeadm performs the actions necessary to get a minimum viable and secure cluster. It follows Kubernetes operator architectural approach for cluster creation and management. What is also important, Kubeadm is an official Kubernetes SIG Lifecycle project - this is important because official Kubernetes SIG projects usually tent to have better community support and be more mature than their non-official counterparts.

Kubeadm is a great choice if you are ready to deal with the same con as in case of The Hard Way approach (i.e. provisioning EC2 servers and handling a cluster administration burden by yourself), but you would like to make provisioning of the cluster itself easier and automated out-of-the-box. Kubeadm makes it also easier to perform common cluster maintenance practices, such as a cluster version upgrade process or adding additional worker node to a cluster.

Kubespray

Kubespray is a set of Ansible resources (playbooks, inventories, etc) focused on Kubernetes provisioning, configuration and management. Kubespray can deploy to almost any kind of target environment you might want (including AWS, GCE, Azure, OpenStack, vSphere, Packet, Oracle Cloud Infrastructure) or even to bare metal (with an array of supported Linux distributions).

Kubespray is alsoan official Kubernetes SIG Lifecycle project - as in case of Kubeadm, this is important because official Kubernetes SIG projects have great community support. What is very interesting, starting from version 2.3
Kubespray supports kubeadm-based deployments. Moreover, starting since version 2.8 Kubespray deprecated non-Kubeadm deployments. It means that currently Kubespray basically relies on Kubeadm to perform cluster creation under the hood.

Kubespray is a strong candidate if you don't want to use EKS for some reasons (for example you need more customized Kubernetes installation over regular EC2 instances), but at the same time you need some higher level opinionated approach for a cluster provisioning process and configuration. Kubespray is also a great choice if you need a cloud-agnostic way of cluster provisioning and management - Kubespray allows you to provision your AWS cluster in a bare metal mode over pre-provisioned EC2 instances accessible via SSH. Another important factor for Kubespray would be familiarity with Ansible, if your operation folks already know Ansible - they will feel at home.

Kops

Kops is described as "kubectl for clusters" . Kops is a toolkit which brings you
very opinionated way of managing clusters on supported cloud providers (including AWS and GCP). Kops is not based on Ansible like Kubespray, instead it relies on AWS API calls directly. Kops provisions not only the cluster itself, but also an EC2-based infrastructure for your cluster. Kops leverages cloud specific features of each cloud provider to brings you the best experience - for example in AWS it will create auto scaling and self healing EC2 autoscaling group to host your cluster nodes. Kubespray is also an official Kubernetes SIG Lifecycle project.

When to use Kops? If you don't want to use EKS for some reasons (like fine-tuning cluster configuration), but you would prefer to rely on higher level and very opinionated tool providing rock solid and battle tested cluster setup - Kops is a way to go. This is also a good option if you don't care so much about being cloud-agnostic.

EKS

AWS EKS is a managed Kubernetes offering from AWS. It gives you highly available managed Kubernetes control plane, and a couple of ways to connect node workers to it (including on demand EC2 instances, Spot instances or even serverless compute engine Fargate). Cluster itself costs about 72 USD per month. In general EKS is the easiest way of provisioning and operating Kubernetes clusters in AWS. I highly recommend you choosing EKS if there is really no good reason for doing otherwise.

What could be reason for not using EKS? One scenario would be customization of your Kubernetes installation. EKS provides
you opinionated setup where you cannot tune every possible piece of your installation. Or even if certain customization is possible, you might lose AWS support for it. For example Amazon only officially supports the Amazon VPC CNI plugin.

Summary

AWS EKS is in general a reasonable default for running Kubernetes clusters in the AWS environment. If for some reason you don't want or can't use EKS, you can try to use less opinionated and more configurable alternatives. Keep in mind however that the more configurable and less opinionated approach is, the more expertise and operational resources is required in-house for your organization to successfully operate it.