Setting Up 3-node Kubernetes with Kind, Prometheus, and Ingress on Windows

Kind can make it really easy to setup a test multi-node Kubernetes cluster even on a Windows host. However, setting up ingress and making it work with Prometheus turned out to be harder than it needs to be due to typos in Kind documentation and the fact that rewrite-target doesn't work with Prometheus out of the box. In this comprehensive guide, I'll walk you through fixing these typos and automating the entire setup process. This will also work if you want to setup ingress with other services on Kind cluster. Although, the instructions here are for Windows hosts, with slight modifications, they can be applied to other operating systems as well.

TL,DR

In this tutorial, we will create a local Kubernetes cluster via Kind, then install Prometheus and an ingress for easy access.

Quick setup:

1. install the dependencies

2. get the code on github.com/taro-ball/tutorial-kind-prometheus-ingress

   • check out the latest commit for ingress setup
   • or #5acfaef for simple (non-ingress) setup

3. run ./create_cluster.sh

4. perform the manual step (the last step in each part)

Part I. Simple Prometheus setup (no ingress)

Step 1. Install kind and dependencies

Pre-requisites:
kind, helm, kubectl, docker, git-bash

I'm doing this on windows, so using choko. For other platforms check the installation instructions. Once kind, helm and kubectl are installed the instructions will be the same for all platforms.

# install git-bash
choco install git.install

# install kubectl
choco install kubernetes-cli

# install helm
choco install kubernetes-helm

# install kind (also installs docker if it's missing)
choco install kind

Choco will automatically install all kind dependencies including Docker if you don't have it installed yet.

Note:
If you don't have choco, follow https://chocolatey.org/install, remember you need to run powershell as administrator.

Step 1. Create a cluster

Note: all commands and scripts should be run in bash (git-bash on windows)

First create a 3node.yaml (or checkout this commit in git, mind this is not the final commit):

# three node (two workers) cluster config
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker

Then create cluster via the following command (or run the .\create-cluster.sh from the repo):

kind create cluster --config 3node.yaml

If you are using the .\create-cluster.sh go straight to Step 3

Step 2. Install Prometheus

Install Prometheus via helm:

helm install prometheus prometheus-community/prometheus --create-namespace --namespace prom

console output:

$ helm install prometheus prometheus-community/prometheus --create-namespace --namespace prom

NAME: prometheus
LAST DEPLOYED: Thu Dec  7 09:41:18 2023
NAMESPACE: prom
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.prom.svc.cluster.local


Get the Prometheus server URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace prom -l "app.kubernetes.io/name=prometheus,app.kubernetes.io/instance=prometheus" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace prom port-forward $POD_NAME 9090

Step 3. Port-forward the Prometheus server pod.

Run the kubectl port-forward command from the helm install output, look for line starting with Get the Prometheus server URL (see example in the console output block above) and you will be able to access prometheus console on http://localhost:9090

Go back to the search page by clicking on prometheus logo at the top and select the Graph tab. You can now try to search for a metric, e.g. container_memory_usage_bytes:

If you just wanted to play around with Prometheus on ur local cluster that's actually enough.

However, if you want to set up an ingress (for example if you don't want to port-forward every single service manually every time) follow the rest of the tutorial.

Part II. Advanced setup

Step 1. The ingress.

Let's install the NGINX ingress controller as per kind documentation :

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml

Note1:

As in part 1 you can simply clone the repo and run the create-cluster.sh, you will still need to perform Step 4 (update the hosts file). You will see a couple of "failed calling webhook" errors, this is normal as it takes some time for Prometheus service to start up.

Note2:

If you want to test the ingress at this point remember that you need to label the master node as ingress-ready. Also, mind that the example given in the kind docs (as of December 2023) doesn't work out of the box so make sure you change pathType: to ImplementationSpecific in the manifest if you want to test the code (see issue and k8s docs). With this adjustment the example works for the simple echo service. However, the path-based (rewrite-target) approach given in the example does not work straight away for a more advanced app such as Prometheus because Prometheus is not aware of the difference in the external path. There seem to be some good leads here, but this approach didn't seem to work for me. I've put the relevant Prometheus helm chart values in the appendix. Let me know if you could make it work with this setup!

Step 2. Prometheus ingress

First install Prometheus (see step 2 in the Part 1). After Prometheus is running we can create an Ingress and point it to prometheus-server service.

Create and apply the following manifest YAML:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-prom
  namespace: prom
spec:
  rules:
  - host: prom.local # remember to add to ur localhost: 127.0.0.1 prom.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: prometheus-server
            port:
              number: 80
  ingressClassName: nginx

Note:

I tried creating Prometheus ingress via helm's values.yaml (see Appendix) but it didn't seem to work. Let me know in the comments if you find the values that work!

Step 3. Docker port-forward

Now, you could port-forward ingress service manually like we did in part 1, but there a way to automate this.
We can define port-forwarding in the kind cluster yaml config:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 9080
    protocol: TCP
  - containerPort: 443
    hostPort: 9043
    protocol: TCP
- role: worker
- role: worker

Note:

First time you port-forward docker on Windows you will see an admin prompt asking you for local network permissions, this is normal, answer "Yes".

Step 4. Hosts file

With port-forwarding you will be able to reach the ingress endpoint, however to access the app via a nice URL like "app.com" you will need to create the following entry in the hosts file:

127.0.0.1 prom.local

Also, rewrite-target doesn't seem to work with Prometheus so using host is our best bet (see the Note2 in Step 1).

Step 5. Enjoy Prometheus!

Now you can simply go to http://prom.local:9080 to open Prometheus! If something doesn't work, refer to the code in https://github.com/taro-ball/tutorial-kind-prometheus-ingress/blob/master/create-cluster.sh . If I find any typos in code, I will first fix them there.

Appendix

see helm show values prometheus-community/prometheus to check all values

Prometheus helm values related to ingress

server:
  ## The URL prefix at which the container can be accessed. Useful in the case the '-web.external-url' includes a slug
  ## so that the various internal URLs are still able to access as they are in the default case.
  ## (Optional)
  prefixURL: "/"

  ## External URL which can access prometheus
  ## Maybe same with Ingress host name
  baseURL: "http://localhost:9080/prom"

  ingress:
    ## If true, Prometheus server Ingress will be created
    ##
    enabled: true

    ## Prometheus server Ingress hostnames
    ## Must be provided if Ingress is enabled
    ##
    hosts:
      - prom.local