Creating a k3s Cluster with k3sup & Multipass 💻☸️

This started off due to frustration with my current Single Board Computers setup and limitations using Docker Desktop meaning I couldn't fully utilise all the features of Kubernetes.

I'd previously setup up a k3s cluster using k3sup on my Raspberry Pi's and ASUS Tinkerboard running Ubuntu 20.04. Due to using older models – Pi 2 & 3 -, with limited resources, it wasn't a great experience for testing demanding applications or load testing.

Docker Desktop Kubernetes is great for starting out but the limitations kick in when you want to test out multiple nodes and installing additional features like Ingress Controllers. Plus, running it on a MacBook Pro 2019 I found the battery life draining quickly – running just Docker Desktop is bad enough at times.

So here comes in Multipass. I'd seen it a few times and was very curious. I'd previously used Vagrant combined with VirtualBox with various levels of success and agony. So the idea of being able to run an Ubuntu VM and Kubernetes Cluster without additional programs to manage and debug got me hooked. I like things to be as native as possible and Multipass can run on HyperKit and Hyper-V.

Installing Multipass is pretty slick and having a CLI – like Vagrant – meant there was a way to wrap it all up in a script for repeatability. And install k3sup is even easier.

Bonus feature of Multipass is being able to pass in cloud-init files to do additional configuration setup, which adds many benefits beyond this example.

The plan is to create a 3 node – 1 master, 2 workers – k3s cluster, which is disposable and portable.

Tools used:

• k3sup
• Multipass
• cloud-init

So let's begin!

k3sup uses SSH to install and by default Multipass uses a predefined User – ubuntu – and SSH Key-Pair, which can be found after bit of digging - see Issue #913. This is where cloud-init comes in.

Using cloud-init, we can easily create a user, assigned it to groups, and pass in a Public SSH Key of our choice. And this is the minimal to do so:

users:
- name: tom
  groups: sudo
  sudo: ALL=(ALL) NOPASSWD:ALL
  ssh_authorized_keys: 
  - ssh-rsa...

Next is to spin up the nodes, which can all be done with the Multipass CLI. We'll leave the VMs with the default values for CPU, RAM, Disk Space and OS - latest LTS - but include our cloud-init configuration:

multipass launch -n master --cloud-init - <<EOF
users:
- name: tom
  groups: sudo
  sudo: ALL=(ALL) NOPASSWD:ALL
  ssh_authorized_keys: 
  - ssh-rsa...
EOF

multipass launch -n node1 --cloud-init - <<EOF
users:
- name: tom
  groups: sudo
  sudo: ALL=(ALL) NOPASSWD:ALL
  ssh_authorized_keys: 
  - ssh-rsa...
EOF

multipass launch -n node2 --cloud-init - <<EOF
users:
- name: tom
  groups: sudo
  sudo: ALL=(ALL) NOPASSWD:ALL
  ssh_authorized_keys: 
  - ssh-rsa...
EOF

Using the Multipass CLI, we can then view the IP Addresses of our new VMs:

multipass list

Name                    State             IPv4             Image
master                  Running           192.168.64.8     Ubuntu 20.04 LTS
node1                   Running           192.168.64.9     Ubuntu 20.04 LTS
node2                   Running           192.168.64.10    Ubuntu 20.04 LTS

Now to get our k3s cluster built, again using the default values, first by setting up our master node:

k3sup install --ip 192.168.64.8 --context k3s-cluster --user tom --ssh-key ./demo-key

This will install k3s, setup the instance as the master node and return the kubeconfig to the current directory.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ...
    server: https://192.168.64.8:6443
  name: k3s-cluster
contexts:
- context:
    cluster: k3s-cluster
    user: k3s-cluster
  name: k3s-cluster
current-context: k3s-cluster
kind: Config
preferences: {}
users:
- name: k3s-cluster
  user:
    client-certificate-data: ...
    client-key-data: ...

Next, to set up our worker nodes to join the cluster:

k3sup join --server-ip 192.168.64.8 --ip 192.168.64.9 --user tom --ssh-key demo-key

k3sup join --server-ip 192.168.64.8 --ip 192.168.64.10 --user tom --ssh-key demo-key

Finally, we can now check our k3s cluster is up and running:

KUBECONFIG=kubeconfig kubectl get nodes

NAME     STATUS   ROLES    AGE     VERSION
master   Ready    master   7m17s   v1.19.7+k3s1
node1    Ready    <none>   6m47s   v1.19.7+k3s1
node2    Ready    <none>   6m24s   v1.19.7+k3s1

And now time to tear it all down now:

multipass delete --all && multipass purge

The Next Level

I mentioned that I wanted this setup to be disposable and portable. So let's wrap this all up in a good ol' Shell script.

So with a bit of refactoring and Command Line tooling - Piping & Parsing - we can end up with a script like so:

#!/bin/sh

master="master"
nodes=("node1" "node2")
context="k3s-cluster"

createInstance () {
    multipass launch -n "$1" --cloud-init - <<EOF
users:
- name: ${USER}
  groups: sudo
  sudo: ALL=(ALL) NOPASSWD:ALL
  ssh_authorized_keys: 
  - $(cat "$PUBLIC_SSH_KEY_PATH")
EOF
}

getNodeIP() {
    echo $(multipass list | grep $1 | awk '{print $3}')
}

installK3sMasterNode() {
    MASTER_IP=$(getNodeIP $1)
    k3sup install --ip "$MASTER_IP" --context "$context" --user "$USER" --ssh-key  "${PRIVATE_SSH_KEY_PATH}"
}

installK3sWorkerNode() {
    NODE_IP=$(getNodeIP $1)
    k3sup join --server-ip "$MASTER_IP" --ip "$NODE_IP" --user "$USER" --ssh-key "${PRIVATE_SSH_KEY_PATH}"
}

createInstance $master

for node in "${nodes[@]}"
do
    createInstance "$node"
done

installK3sMasterNode $master

for node in "${nodes[@]}"
do
    installK3sWorkerNode "$node"
done

And with only needing two Environment Variables, we can now bring up a disposable cluster within minutes - I clocked just under 2 minutes!

PUBLIC_SSH_KEY_PATH=./demo-key.pub PRIVATE_SSH_KEY_PATH=./demo-key ./minimal-k3s-multipass-bootstrap.sh

Making a script as well allows for easy use on multiple systems without needing to install more than the necessary.

Multipass offers extra configuration options for the VMs and cloud-init can do more internally on the OS. Plus k3sup has even more options, but I'll keep it simple for now!

Enjoy!