- Jenkins Preparation
- Kubernetes Setup
- Jenkins Kubernetes Setup
- Agent Setup
- Jenkins Agent Configuration
- Jenkins Pipeline Test
- Conclusion
Jenkins is a Java based CI/CD system that can be self hosted. In order to initiate builds, Jenkins utilizes a component called an agent to execute build commands. These agents can be a constantly running service or part of an on demand service such as a cloud provider or Docker containers. Kubernetes is one such solution to deploying build agents on demand. This article will look at how to setup Kubernetes as a provider for Jenkins build agents.
Jenkins Preparation
Long lived credentials can be problematic to deal with when setting up authentication. In some cases there isn't even an expiration for the credentials at all. To work around this we're going to setup Jenkins to be an OpenID Connect (OIDC) provider.
Plugin Setup
Jenkins doesn't provide this functionality out of the box. A plugin will be required to enable it. We'll go ahead and install this plugin, along with the Kubernetes plugin that will be used later:
- Go to Manage Jenkins at the instance root
- Click "Plugins" under "System Configuration"
- Click "Available plugins" on the left
- Search for "kubernetes"
- Select the plugin with the description "This plugin integrates Jenkins with Kubernetes"
- Clear the search box and enter "oidc"
- Select the "OpenID Connect Provider" plugin
- Click "Install" in the upper right
OIDC Claim Templates
Before doing the actual setup I'd like to detail on how Jenkins provides OIDC authentication:
- Go to Manage Jenkins in the instance root
- Click on "Security" under the "Security" category
- While we're here, scroll down to "Agents"
- Click on "Random" for TCP port for inbound agents as this will be required for the JNLP agent to work in our Kubernetes setup
- Now scroll down to "OpenID Connect"
- Expand "Claim templates"
So here there are a number of variables with per build and global scope. For this use case I simply want basic access to handle pods as build agents, so I won't be doing anything build scoped. Instead I'll be working off the subject or sub
which is set to the URL of the Jenkins instance. This can be any variable that Jenkins is aware of, including ones introduced by plugins. Basic global ones can be found in the Jenkins documentation. Go ahead and click on "Save" at the bottom when you're finished inspecting the claim template to ensure the agent port is applied.
Creating an OIDC token
An OICD token acts as the entity that Kubernetes will utilize to call back and confirm the token. It works much like a standard credential but with a few other attributes:
- Go to Manage Jenkins at the instance root
- Select "Credentials" under the "Security" category
- Click "(global)" under the "Stores scoped to Jenkins" section at the bottom
- Click "Add Credentials" at the upper right
- Fill out the following:
- Kind: OpenID Connect id token
- Scope: System
- Issuer: (leave blank)
- Audience: [unique value of some kind]
- ID:
jenkins-kubernetes-integration
- Description: OpenID Connect Credentials For Kubernetes
So the important part here is the Audience value. This will be used as a way for Kubernetes to identify itself to the system. Make sure it's unique for you and note it down somewhere for later use. As we've completed as much that can be on the Jenkins side, it's time to configure Kubernetes.
Kubernetes Setup
The OIDC setup will actually offload a decent amount of authentication setup that we'd normally have to do. You don't even need a service account! However, before we can do anything with OIDC we'll need to set it up so the kube-apiserver understands it.
OIDC Setup
Note: There's actually a Structured Authentication Config established via KEP-3331. It's in v1.28 as a feature flag gated option and removes the limitation of only having one OIDC provider. I may look into doing an article on it, but for now I'll deal with the issue in a manner that should work even with a bit older versions versions of Kubernetes.
In order for OIDC to work the apiserver needs to know it exists. This can be achieved via command line options. So go ahead and open:
$ sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml
This will bring up YAML configuration for the API server. The command
section has a decent amount of arguments which we'll be adding to:
- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
- --oidc-issuer-url=https://jenkins:8080/oidc
- --oidc-client-id=[unique value of some kind]
- --oidc-username-claim=sub
- --oidc-username-prefix="oidc:"
So first off is the OIDC issuer URL, which is the Jenkins instance root + /oidc
. oidc-client-id
is the unique value that was setup as Audience
during the OpenID Connect token creation (yes, it's not the credential ID). Remember the OIDC claim template we were looking at? That's what this is about. --oidc-username-claim
indicates the value that will be used as User
when dealing with any authorization components. In this case it will be https://jenkins:8080/
as that's what my Jenkins instance URL is. The prefix exists to avoid conflicting with an actual in-system user's name (service account for example). Unfortunately, because it's YAML we have to enclose it in double quotes to avoid parsing issues (this will lead to an interesting issue later). Once this is done simply save and exit. If the YAML file doesn't have any issues the process list should have it with the new options.
Namespace Creation
Now it's time to work with setting up some permissions. To help have fine grained permissions we'll use the namespace feature for scope purposes. Namespaces allow for a way to isolate Kubernetes resources. Here is an example of namespaces for a cluster I operate:
$ kubectl get namespaces
NAME STATUS AGE
default Active 35h
kube-node-lease Active 35h
kube-public Active 35h
kube-system Active 35h
default
is where any actions which require a namespace will go into if one is not explicitly defined in a default setup (tools such as kubens can alter this behavior). In the context of Jenkins, namespaces are a useful way to allow isolation of individual Jenkins instances that want to utilize the same Kubernetes cluster. Creation of a namespace is a simple option to kubectl
:
$ kubectl create namespace jenkins
namespace/jenkins created
$ kubectl get namespaces
NAME STATUS AGE
default Active 36h
jenkins Active 3s
kube-node-lease Active 36h
kube-public Active 36h
kube-system Active 36h
Now it's time to establish some actual permissions which will be bound to this namespace.
Role Creation
The permissions here will be enough to maintain and monitor agent pods. We'll also use a RoleBinding to attach it to our OIDC entity:
jenkins-role.yaml
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: jenkins-oidc
namespace: jenkins
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: jenkins-oidc-binding
namespace: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: jenkins-oidc
subjects:
- kind: User
name: "\"oidc:\"https://jenkins:8080/"
namespace: jenkins
So here the Role, RoleBinding, and subject of the binding are all namespace bound to the jenkins namespace. Due to oidc:
being enclosed in "'s the quotes themselves are actually part of the literal value. This makes up for an odd case where we have to escape quotes so they appear properly. After that is the Jenkins URL which maps to the sub
value that we're getting our username from via --oidc-username-claim=sub
. All that's left is to apply the YAML and create the resources:
$ kubectl apply -f jenkins-role.yaml
Jenkins Kubernetes Setup
Now it's time to setup our Kubernetes cluster as an agent cloud. Before beginning make sure to grab the value of the Kubernetes server certificate via:
$ kubectl config view --raw -o go-template='{{index ((index (index .clusters 0) "cluster")) "certificate-authority-data"|base64decode}}'
This is used so the Jenkins system knows the API server's certificate is trusted and won't complain about SSL verification errors. Now for the cluster setup:
- Go to Manage Jenkins at the instance root
- Click "Clouds" under "System Configuration"
- Click "New Cloud"
- Enter "Kubernetes Cluster" as the Cloud name
- Select "Kubernetes" as the Type
- In the new page, expand "Kubernetes Cloud details"
- Enter in the following:
- Enter the API server's URL,
https://k8-control-plane:6443/
as an example - Enter the Kubernetes CA cert that was obtained earlier
- Enter "jenkins" under "Kubernetes Namespace"
- For "Credentials", select the OpenID Connect credentials we created
Click Test Connection
to confirm everything works. If it does, go ahead and click Save
. If something isn't working make sure the Kubernetes API server is up and has the OIDC arguments. Also validate your Role and RoleBinding in case permissions are listed as the issue.
Agent Setup
This is actually an optional step and mostly tied to my particular setup. My Jenkins instance is resolved via /etc/hosts
and has an SSL certificate signed by a self-managed Root CA. This leads to two issues:
- The Jenkins agent works off the Java certificate trust store which doesn't know about my root CA cert out of the box
- The Kubernetes cluster works off it's own DNS, and even when DNS is customized it expects a domain name (which
jenkins
doesn't have).
So time to address these two issues with their own resolutions.
Agent Customization
To handle the certificate issue the root certificate for my CA will need to be trusted by the agent's Java CA store. While probably not necessary I'll also added it to the system's trust store. In this case I'll be using a certificate bundle which is a combination of the Root CA and Intermediate CA's certificate. While sometimes the root CA is good enough I tend to find that some software will complain if the intermediate isn't also present. So I'll go ahead and make a docker file with the certificate chain in the same folder:
FROM jenkins/inbound-agent:3148.v532a_7e715ee3-1-jdk11
COPY ca-chain.crt /usr/local/share/ca-certificates/
USER root
RUN apt-get update
RUN apt-get upgrade -y
RUN update-ca-certificates
RUN ${JAVA_HOME}/bin/keytool -import -trustcacerts -cacerts -noprompt -storepass changeit -alias jenkins-ca -file /usr/local/share/ca-certificates/ca-chain.crt
Here I'm using a specific version tag of jenkins/inbound-agent
, which is the container image used for Jenkins agents that are part of the Kubernetes plugin workflow. This means it will be spun up regardless of what pipeline declarations you have. So some important things to note here:
-
/usr/local/share/ca-certificates
is the locations where certs need to be forupdate-ca-certificates
to make them trusted root certs - The reallly long
keytool
declaration is what pulls the certificate into the Java cacerts trust store so the Jenkins java agent recognizes my Jenkins SSL certificate while in the Kubernetes cluster - System packages are updated to ensure software isn't too outdated (though to be fair Jenkins agents are put out fairly frequently)
Agent Build
Now since Kubernetes works off of containerd I'll be taking a different approach on handling container builds by using nerdctl and the buildkit that comes bundled with it. I'll do this on the amd64 control plane node since it's beefier than my Raspberry Pi workers for handling builds and build related services. Go ahead and download and unpack the latest nerdctl release as of writing (make sure to check the release page in case there's a new one):
$ wget https://github.com/containerd/nerdctl/releases/download/v1.5.0/nerdctl-full-1.5.0-linux-amd64.tar.gz
$ sudo tar Cxzvf /usr/local nerdctl-full-1.5.0-linux-amd64.tar.gz
Now we can start the buildkitd daemon to prepare for building the container:
$ sudo systemctl enable buildkit --now
buildkit
is recognized as a service due to /usr/local/lib/systemd/system
being in the unit path:
$ sudo systemctl --no-pager --property=UnitPath show | tr ' ' '\n'
UnitPath=/etc/systemd/system.control
/run/systemd/system.control
/run/systemd/transient
/run/systemd/generator.early
/etc/systemd/system
/etc/systemd/system.attached
/run/systemd/system
/run/systemd/system.attached
/run/systemd/generator
/usr/local/lib/systemd/system
/lib/systemd/system
/usr/lib/systemd/system
/run/systemd/generator.late
Next up is that containerd has the concept of namespaces much like Kubernetes. You can see an example here:
# sudo ctr namespaces list
NAME LABELS
buildkit
default
k8s.io
moby
moby
is usually the docker applied namespace, while k8s.io
is what's used by Kubernetes. This means if we want Kubernetes to recognize our built agent image, we need to make sure it's in the k8s.io
namespace. Also I'll build for both amd64
and arm64
platforms as my workers are Raspberry Pi, and just in case I decide to support more agents by allowing pods on my control plane. The end result looks something like this:
$ sudo nerdctl build --namespace k8s.io --platform=amd64,arm64 -t k8s.io/test:v1.0 .
$ sudo ctr --namespace k8s.io images list | grep k8s.io/test
k8s.io/test:v1.0
Agent Deployment
Now there's a slight problem in that right now only my control plane has a copy of this image. The way Kubernetes works is that each node has its own containerd
instance. This means that a worker node can't see the containerd
images of the control plane/other worker nodes, so I'll need to make sure all nodes know about the image. One solution would be to run a local image registry. However, given that it's a two worker cluster I'll take a somewhat more simplistic route:
worker_image_deploy.sh
#!/bin/bash
TARGET="k8s.io/${1}"
echo "Building image ${TARGET}"
sudo nerdctl --namespace k8s.io build --platform=amd64,arm64 -t "${TARGET}" .
echo "Exporting image ${TARGET}"
sudo ctr --namespace=k8s.io image export --platform linux/arm64 image.tar "${TARGET}"
for host in rpi rpi2
do
echo "Deploying image to ${host}"
scp image.tar ${host}:~/
echo "Importing image ${TARGET}"
ssh ${host} sudo ctr --namespace=k8s.io images import --base-name "${TARGET}" image.tar
ssh ${host} sudo rm image.tar
done
sudo rm image.tar
So what this does is build the image in the k8s namespace given a certain target name. Then it will export the image, copy it to each worker, and import it to the worker's containerd
. It also cleans up after the images so they don't duplicate space. rpi
and rpi2
are special hosts I have setup so you'll want to replace them appropriately with whatever worker hosts/IPs/SSH connection string you have. Now to build out the Jenkins agent:
$ ./worker_image_deploy.sh jenkins-agent:v1.0
<build output spam snip>
Exporting image k8s.io/jenkins-agent:v1.0
Deploying image to rpi
image.tar 100% 163MB 44.7MB/s 00:03
Importing image k8s.io/jenkins-agent:v1.0
unpacking k8s.io/jenkins-agent:v1.0 (sha256:eae87bca0014a6f6f6bc24bd5c9e4a93a454909b82e7f73cfedfa60db2c5260c)...done
Deploying image to rpi2
image.tar 100% 163MB 74.9MB/s 00:02
Importing image k8s.io/jenkins-agent:v1.0
unpacking k8s.io/jenkins-agent:v1.0 (sha256:eae87bca0014a6f6f6bc24bd5c9e4a93a454909b82e7f73cfedfa60db2c5260c)...done
If I check both my workers:
# rpi
$ sudo ctr --namespace k8s.io images list | grep k8s.io/jenkins-agent:v1.0
k8s.io/jenkins-agent:v1.0
# rpi2
$ sudo ctr --namespace k8s.io images list | grep k8s.io/jenkins-agent:v1.0
k8s.io/jenkins-agent:v1.0
Both of them now recognize the agent image and can interact with it for pod purposes.
Jenkins Agent Configuration
Now that our custom agent is built we've taken care of the certificate issue. It's time to make our new image the designated JNLP agent, and then solve the host issue:
- Go to Manage Jenkins at the instance root
- Select "Clouds" under "System Configuration"
- Select "Kubernetes Cluster" (or the name you chose for it)
- Select "Configure" on the left
- Expand "Kubernetes Cloud details"
- Expand "Advanced"
- Enter "jnlp" as "Defaults Provider Template Name"
- Expand "Pod Templates"
- Click on "Add Pod Template"
- Name it "jnlp"
- Expand "Pod Template details"
- Enter
k8s.io/jenkins-agent:v1.0
for "Docker image" - Ensure "Always pull image" is not selected, so it doesn't try to pull our local image from Docker Hub
- Go down to "Raw YAML for the Pod"
- Enter the following and make sure "Merge" is set for "Yaml merge strategy":
spec:
hostAliases:
- ip: [ip address of jenkins host]
hostnames:
- "jenkins"
Be sure to replace [ip address of jenkins host] with what your Jenkins instance would resolve to from a worker DNS point of view. Now go ahead and click "Save" at the bottom.
Jenkins Pipeline Test
It's finally time to test that Jenkins pipelines are able to work off the Kubernetes cluster. Go to the Jenkins instance root:
- Select "New Item"
- Enter "Kubernetes Pipeline Test" for the name
- Select "Pipeline" for the type
- Select "OK" at the bottom
- Scroll down to "Pipeline" on the next screen and enter the following for the pipeline script:
podTemplate(containers: [
containerTemplate(name: 'busybox', image: 'busybox:latest', command: 'sleep', args: '99d')
]) {
node(POD_LABEL) {
stage('Test') {
container('busybox') {
sh 'echo "test"'
}
}
}
}
Finally click "Save" at the bottom. Now we'll start a build with "Build Now" on the left. If everything went well you'll see something similar to the following output:
So what happened is our new JNLP agent orchestrated the build process with proper certificate validation and host resolution. Then the pipeline initiated a busybox container that simply echo
'ed "test". The node(POD_LABEL)
is left as is and will be dynamically replaced appropriately. Pod definitions are declared through podTemplate
which has two main forms:
- An object like form with a list of
containerTemplates
- Straight YAML declaration
We've already seen the object like form, so here's an example of the YAML form:
podTemplate(yaml: '''
apiVersion: v1
kind: Pod
metadata:
labels:
some-label: some-label-value
spec:
containers:
- name: busybox
image: busybox
command:
- sleep
args:
- 99d
''') {
node(POD_LABEL) {
container('busybox') {
sh 'echo "test"'
}
}
}
The Jenkins documentation has a list of supported properties. There's also the yamlMergeStrategy
present if you want to combine methods. I'll take a quick look at what option might suit you. Though in the end work with what's best for you and/or your team.
Object Like Pattern
This is generally what I recommend going with if possible. The main pain point is the lack of pod properties you'd expect for a more involved Kubernetes pod definition. On the other hand, being object like means they can be programmatically generated with ease. This makes them very ideal for situations with global pipeline libraries. It's also much more terse code wise than a full YAML pod definition.
YAML Pod Definition
This is a straight Kubernetes Pod definition. The big takeaway is that they can be very verbose, which means more lines of code to scroll though. Putting them in a file to be loaded is an option, though that becomes a "yet another place to look for code" issue. If you're coming from more of a Kubernetes background though it might suit you better. It will be necessary if your Pod definition has strict requirements that the containerTemplate
method can't meet.
Conclusion
Much like the previous Kubernetes installment this one was equally as involved. The fun included things like:
- Trying to get the
oidc:
prefix to work because colons in YAML is annoying - Realizing CoreDNS resolution wasn't going to work out because my Jenkins instance isn't in a Kubernetes cluster ( which is more painful to setup than what I have now )
- Figuring out the JNLP container was a thing and that I'd have to custom build it
- Trying to make containerd image builds work without the help of Docker
- Packages for buildkit? It's complicated...
- Really keytool? Why don't you just use my system trust store? What on earth is this 10 mile long command just to import a root CA cert?
- Figuring out how to make host resolution container properties work
- Not realizing Kubernetes Roles needed a namespace. Thought it was just the binding
- Trying to make cert auth with kubeconfig before I knew OIDC was a thing (painful)
- Trying to scroll through pages of lifecycle bot of doom and additional comments required to unmark stale issues when checking anything Kubernetes related
Well I know more now I suppose! Hope you enjoyed this article and I'll be off to think on what I want to write on next (probably should get back to my Python series since the Kubernetes learning urge has calmed down a bit).
Top comments (0)