Setting up Jenkins on MicroK8s

I recently bought a mini-pc (MSI Cubi 3 Silent) for home that I use as a server for various purposes and something that I wanted was to set up Jenkins for automatically building and deploying my side projects to my local MicroK8s cluster (Ubuntu Server 18.04).

It turns out you can install Jenkins on Kubernetes and thanks to kubernetes-plugin Jenkins agents will run in ephimeral pods created just for building your task.

If you know what you have to do, it's actually pretty easy to configure but that's very often the case in hindsight, isn't it? I battled it out for a whole day and here are my findings, should probably take 30 mins to set up.

Prerequisites

• MicroK8s cluster (should work with other Kubernetes clusters too), with storage enabled.
• Docker service running on the host (outside Kubernetes)
• Helm installed and deployed in Kubernetes

Installing Jenkins on Kubernetes

# Download Jenkins chart values for customization. I'm linking the version I used, but the newest should work.
$ wget -O jenkins-chart.yml https://github.com/helm/charts/blob/b22774e2f2019c0e5d8182aab6111a84e95fa8ca/stable/jenkins/values.yaml

# Edit chart options to use NodePort instead of LoadBalancer, as MicroK8s doesn't
# ship with any: https://github.com/ubuntu/microk8s/issues/200#issuecomment-441180273
# 
# ServiceType: NodePort
# NodePort: 32000 (optional, if you want to have a fixed one for exposing it externally)
#
# Alternatively, if you have Ingress enabled on MicroK8s you could 
# use ServiceType: ClusterIP and add a rule in your Ingress configuration to
# direct traffic to Jenkins.
$ vim jenkins-chart.yml

$ helm install --name jenkins -f jenkins-chart.yaml stable/jenkins

# Follow the instructions to retrieve the generated password.
# If you need to uninstall Jenkins, then
$ helm del --purge jenkins

At this point, Jenkins should be accessible at http://your-server:32000. I then installed the Blue Ocean plugin as I find it more aesthetically pleasant plus it's designed for the Jenkins Pipeline. Installing it is as easy as going to Manage Plugins and checking the Blue Ocean box. You don't need to check the other ones, they'll be added automatically as dependencies.

Now we're ready to create our first pipeline!

Creating a Github pipeline

Let's assume we want to create a pipeline for a private repository we've got at Github.

1. Follow the instructions here but skip the final pipeline creation wizard, as we're going to use a Jenkinsfile with the declarative pipeline syntax instead.
2. Create a webhook in your Github repo pointing to http://your-server:32000/github-webhook/ (customize with your domain/port/https, and keep the trailing slash) and verify that it's working (the secret doesn't matter). Theoretically this should be done automatically by Blue Ocean in step 1, but it didn't work for me. Note that if you're using a firewall or you're in a private network, you'll have to open the port and/or forward it to your server appropriately.
3. Create a file named Jenkinsfile at the root of the repository. I'm going to assume you've got a dockerized app and you've got the Kubernetes deployment files well configured.

// Jenkinsfile

pipeline {
  agent {
    kubernetes {
      // this label will be the prefix of the generated pod's name
      label 'jenkins-agent-my-app'
      yaml """
apiVersion: v1
kind: Pod
metadata:
  labels:
    component: ci
spec:
  containers:
    - name: docker
      image: docker
      command:
        - cat
      tty: true
      volumeMounts:
        - mountPath: /var/run/docker.sock
          name: docker-sock
    - name: kubectl
      image: lachlanevenson/k8s-kubectl:v1.14.0 # use a version that matches your K8s version
      command:
        - cat
      tty: true
  volumes:
    - name: docker-sock
      hostPath:
        path: /var/run/docker.sock
"""
    }
  }

  stages {
    stage('Build image') {
      steps {
        container('docker') {
          sh "docker build -t your-registry/my-app:latest ."
          sh "docker push your-registry/my-app:latest"
        }
      }
    }

    stage('Deploy') {
      steps {
        container('kubectl') {
          sh "kubectl delete -f ./kubernetes/deployment.yaml"
          sh "kubectl apply -f ./kubernetes/deployment.yaml"
          sh "kubectl apply -f ./kubernetes/service.yaml"
        }
      }
    }
  }
}

Commit and push it to your repository. Jenkins should be notified and a build should be starting!

A few things to note from the Jenkinsfile

• The docker and kubectl containers declared are so that we can run these commands in the pipeline without having them installed in the host, which is typically what you'd do with Jenkins outside Kubernetes.
• We're using the host's Docker daemon, which is why it was a prerequisite and the reason we defined docker-sock with path: /var/run/docker.sock. If you don't want to need to have Docker in your host, you could try Docker in Docker.
• We're using lachlanevenson/k8s-kubectl, which is a very simple image with just kubectl. You could use any other image with kubectl, but make sure the image doesn't run with another user id or you'll have permission issues with Jenkins. I know it from the experience of trying to use bitnami/kubectl.
• You might wonder why we've got command: cat and tty: true. They avoid containers exiting early. To be honest, I don't know exactly how it works.
• There's no specific step for git. If you use webhooks, the git step is automagically added when a push event is received.

Other things of interest

• If you have submodules, make sure you check Recursively update submodules in your Jenkins repo build configuration, under Advanced sub-modules behaviours.
• You might notice 'zombie' processes (see top command) as builds are executed. I don't know why this happens but it could be a bug in Jenkins or Docker. It doesn't happen consistently though.

Final words

Configuring Jenkins might seem a very daunting task and it's certainly not obvious. With Kubernetes and Helm I believe it's much simpler but still it's difficult to find easily reproducible instructions.

If you're setting a Jenkins environment without an existing Kubernetes cluster completely from scratch, you might want to have a look at Jenkins X. Otherwise, I hope this guide works for you.

See you next time!