If you are developing multiple applications in the same time for Kubernetes, you will realise that running and debugging these during the daily work is very sophisticated and time consuming, since we have to repeat the following steps:
- Building the application
- Building the docker image
- Pushing the container image to a docker registry
- Creating/Refreshing the Kubernetes' objects
It's enough to have a database and an MQ connection in addition, and you are in a serious trouble. I'm not saying that It's impossible to test everything together, but I'm absolutely sure that if you need to do the steps above by hand, it will break your productive flow.
Fortunately Google recently announced Jib 1.0.0 which combined with Skaffold able to solve this problem. In this article I'm going to show you a workflow with these tools alongside Spring Boot and Helm.
A Spring Boot application
First we need to have a basic Spring Boot application. I suggest to use Spring Initiallizr which can help you bootstrap a Spring Boot application in a few seconds. In this article I'm using Spring Boot 2.1.3 with gradle, however you will find the maven related files in the repository too.
Just download the generated project and let's create a RestController
@RestController
public class TestController {
@GetMapping(value = "/echo/{text}")
public ResponseEntity test(@NotNull @PathVariable String text) {
return ResponseEntity.ok(text);
}
@GetMapping
public ResponseEntity hello() {
return ResponseEntity.ok("HELLO");
}
}
During the article I want to demonstrate Jib's customizability, so let's change the default port number to 55000
. Create an application.properties
and put this line into the file
server.port=55000
Open your favorite shell and run ./gradlew bootRun
. If both http://localhost:55000
and http://localhost:55000/echo/test
seems to work, let's continue with the containerisation phase.
Using Jib for docker image creation
Jib is implemented in Java and runs as part of your Maven or Gradle build. You do not need to maintain a Dockerfile, run a Docker daemon. You just need to open your build.gradle
file, append the plugins section with id 'com.google.cloud.tools.jib' version '1.0.0'
and add the following piece of code to the end
jib {
to {
image = 'com.github.pozo/spring-boot-jib'
}
container {
ports = ['55000']
}
}
It should be noted that the configuration above is not mandatory. However, without these lines Jib would produce an image named spring-boot-jib:0.0.1-SNAPSHOT
where the first part is the value of the rootProject.name
from settings.gradle
and the second one is provided by the version
variable from build.gradle
. In order to build a more advanced image, I suggest to look over the available options.
The above configuration’ equal would look like this with Dockerfile
FROM gcr.io/distroless/java:latest
COPY dependencyJars /app/libs
COPY snapshotDependencyJars /app/libs
COPY resources /app/resources
COPY classFiles /app/classes
COPY src/main/jib /
ENTRYPOINT ["java", jib.container.jvmFlags, "-cp", "/app/resources:/app/classes:/app/libs/*", jib.container.mainClass]
CMD [jib.container.args]
Jib uses distroless java as the default base image, which seems to use container related JVM flags by default. The multiple copy statements are used to break the app into layers, allowing for faster rebuilds after small changes.
If you want to specify a different base image, just add a from
section
jib {
from {
image = 'java:8-jre-alpine'
}
to {
image = 'com.github.pozo/spring-boot-jib'
}
container {
ports = ['55000']
}
}
If we want to build the container image we have two options. Using jib
task which pushes a container image for your application to a container registry, or jibDockerBuild
which uses the docker command line tool and requires that you have docker available on your PATH or you can set the executable location via the dockerClient
object.
Open a terminal and execute
./gradlew jibDockerBuild
I'm going to explain why we need to use this task instead ofjib
in the next paragraph.
Check the output of docker images
command. If you wonder why your image created by ~49 years ago, It's for reproducibility purposes, Jib sets the creation time of the container images to 0. In order to use the current time just add useCurrentTimestamp = true
inside of the jib.container
. For more advanced questions check out the Jib's FAQ.
After the build phase we can run our image with
docker run -p 8080:55000 com.github.pozo/spring-boot-jib
If both http://localhost:8080
and http://localhost:8080/echo/test
seems to work, we can continue with our Kubernetes objects.
Running the application image inside of Kubernetes
If you are not familiar with Kubernetes I recommend to start with the official documentation.
First of all we need an up and running Kubernetes cluster. Fortunately we have several options nowadays. For instance we can use Minikube which runs a single-node Kubernetes cluster inside a VM on your laptop.
An another option is Docker for Desktop Kubernetes. It runs a single-node cluster locally within your Docker instance. I believe this one is the most comfortable way to hack around with Kubernetes, so I suggest to use this one.
We also need kubectl which is a command line interface for running commands against Kubernetes clusters.
If you decided to use Minikube then start the cluster with
minikube start
This command creates and configures a Virtual Machine that runs a single-node cluster. This command also configures your kubectl installation to communicate with this cluster. You also need to run
eval $(minikube docker-env)
The command minikube docker-env
returns a set of Bash environment variable exports to configure your local environment to re-use the Docker daemon inside the Minikube instance. (source)
This means you don't have to build on your host machine and push the image into a docker registry, you can just build inside the same docker daemon as Minikube. So after
./gradlew jibDockerBuild
Minikube will be able to reach out the image com.github.pozo/spring-boot-jib
. Don't forget to run eval $(minikube docker-env)
every time when you prompt a new terminal.
To enable the Kubernetes cluster in case of Docker for Desktop just follow the platform specific instructions.
To run our image in the cluster we need to define a Deployment first. Deployment represent a set of multiple, identical Pods with no unique identities. A Deployment runs multiple replicas of your application and automatically replaces any instances that fail or become unresponsive. In addition we need an externally exposed Service whereby we can reach our application from outside of the cluster. Here is a diagram about what we want to achieve
Let's Create a kubernetes
directory under the project's root and create a file spring-boot-jib.yaml
with the following content under the freshly created directory
apiVersion: apps/v1
kind: Deployment
metadata:
name: spring-deployment
spec:
replicas: 1
selector:
matchLabels:
app: spring-boot-jib
template:
metadata:
labels:
app: spring-boot-jib
spec:
containers:
- name: spring-boot-jib-pod
image: com.github.pozo/spring-boot-jib
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 55000
---
apiVersion: v1
kind: Service
metadata:
name: spring-boot-jib-service
spec:
type: LoadBalancer
ports:
- port: 8080
targetPort: 55000
protocol: TCP
name: http
selector:
app: spring-boot-jib
A few important things to mention here.
- The Deployment's
spec.selector.matchLabels.app
value must be the same as thespec.template.metadata.labels.app
value - Service's
spec.selector.app
value must be the same as the Deployment'sspec.selector.matchLabels.app
value. So the service can find our Pod, and hand over every request to it. - The Service's
spec.ports.port
value should be what we want to expose to the outside world, in our case8080
- The Service's
spec.ports.targetPort
must be the same as the Deployment'sspec.template.spec.containers.ports.containerPort
so the Service will redirect everything to the container's port number55000
. - Finally the Pod's
spec.containers.imagePullPolicy
must beIfNotPresent
orNever
. The default valueAlways
would produce an error, since there is no suchcom.github.pozo
repository exist.
As we previously used jibDockerBuild
we have our image locally, and because of Docker for Desktop's cluster uses our host's docker instance, it will able reach the image by default. In case of Minikube, due to eval $(minikube docker-env)
the image built by Minikube's docker daemon, and it will able reach the image too.
Open a terminal and run
kubectl create -f kubernetes/
The create command iterates over the kubernetes
directory and creates Kubernetes resources from it's content.
Open a browser and try to reach http://localhost:8080
and http://localhost:8080/echo/test
again. In case of a Minikube execute minikube service list
to find out the deployed service address. If we are getting status code 200
then we did a great job, and we have a running application in our cluster.
Put everything together with Skaffold
We almost crossed the finish line! Currently we have a containerised application and we can deploy it anytime into the cluster by hand. At this point we must repeat the building and deploying steps after every changes.
Skaffold going to help us to eliminate this handwork. Go to their website and follow the installation instructions. If everything set, create a skaffold.yaml
into the project's root directory with the following content
apiVersion: skaffold/v1beta4
kind: Config
build:
local:
push: false
artifacts:
- image: com.github.pozo/spring-boot-jib
jibGradle: {}
deploy:
kubectl:
manifests:
- kubernetes/*.yaml
This is our brand new Skaffold pipeline file.
- The
build.local.push: false
enforces Skaffold to usejibDockerBuild
. - The
build.artifacts
is a list of the actual images we're going to be build. - The
build.artifacts.jibGradle
configures Skaffold to use Jib during the image building phase. - The
deploy.kubectl.manifests
value set the folder name where we already have our kubernetes manifest files. If we skip this the default directory name would bek8s
.
If you are looking for more advanced pipeline configuration I recommend to check out this well annotated example.
Open a terminal and run
skaffold dev --trigger notify
This command runs our pipeline file in development mode, which means every change in our codebase will trigger Skaffold to call Jib to build the image, and kubectl to deploy it. Sounds cool right ?
Change the return value of TestController
's hello
method to "GOODBYE"
and see what happens in the terminal. Refresh the browser after a few seconds, and you will see "GOODBYE"
instead of "HELLO"
.
Using helm
If you are not familiar with helm I suggest to take a look at the official quick starter guide first.
I must admit It's absolutely not mandatory to use Helm for development, and someone suggest you to think twice before using it, however according to my experience helm making the application deployment easy, standardised and reusable, especially when you have to work with several applications.
Create a helm chart for our application with
helm create spring-boot-jib
The helm's create command will generate a directory structure with several files. In order to make it cleaner what do we have in this folder, rename it to helm
. The most important files are in the templates directory and the values.yaml
itself.
Change the generated service.yaml
's spec.ports.targetPort
to
targetPort: {{ .Values.service.containerPort }}
The deployment.yaml
's spec.template.spec.containers.image
value to
image: "{{ .Values.image.repository }}{{ if .Values.image.tag }}:{{ .Values.image.tag }}{{ end }}"
And the deployment.yaml
's spec.template.spec.containers.ports
value to
ports:
{{- toYaml .Values.container.ports | nindent 12 }}
Everything between {{ }}
came from the values.yaml
or _helper.tpl
files. In fact we are using Go templating. And change the values.yaml
file like this
replicaCount: 1
image:
repository: com.github.pozo/spring-boot-jib
tag: latest
pullPolicy: IfNotPresent
nameOverride: ""
fullnameOverride: "spring-boot-jib"
service:
type: LoadBalancer
port: 8080
containerPort: 55000
container:
ports:
- name: http
containerPort: 55000
protocol: TCP
As we want to use Helm instead of kubectl, we need to adjust the Skaffold pipeline accordingly
apiVersion: skaffold/v1beta4
kind: Config
build:
local:
push: false
artifacts:
- image: com.github.pozo/spring-boot-jib
jibGradle: {}
deploy:
helm:
releases:
- name: spring-boot-jib
chartPath: helm
values:
image.repository: com.github.pozo/spring-boot-jib
setValues:
image.tag: ""
We must configure the chartPath
, the image.repository
, and we must set image.tag
value to empty string for Helm, so Skaffold will be able to manage custom tags on the deployment.
If everything set, run
skaffold dev --trigger notify
If we are using Minikube we don't need to execute eval $(minikube docker-env)
anymore, since Skaffold will take care of it. If you want to see what happens under the hood, just add the -v debug
switch.
skaffold dev --trigger notify -v debug
Live debugging
During the development It's a natural demand to setup a breakpoint and check the application's state while It's running. Locally It's a very easy process but what if we have everything in the cluster ? Actually It's easier as you might think, we need to adjust just a few things.
- Add
5005
to the list of port under the jib configuration in thebuild.gradle
- Add the
agentlib
related configuration inside ofjvmFlags
property
jib {
to {
image = 'com.github.pozo/spring-boot-jib'
}
container {
useCurrentTimestamp = true
ports = ['55000', '5005']
jvmFlags = [ '-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005' ]
}
}
- Extend
values.yaml
'scontainer.ports
with
- name: debug
containerPort: 5005
protocol: TCP
- Create a remote configuration for the debugger
Then run the pipeline. Skaffold will automatically forward every listed ports, however be careful, after a change Skaffold might pick a random port if the requested one isn't available. Set a breakpoint, and run the previously created Remote configuration, call the corresponding endpoint and voilà.
I hope you enjoyed reading the article as much as I enjoyed writing it. I'm not a proficient writer, so if you have a comment, remark feel free to share it in the comments section. The source code of this article is available on GitHub.
I want to thank Gergő Szabó and Dániel Szabó for all the help.
If you are interested in this topic I suggest to look over these articles as well
- Dockerize Spring Boot Application With Jib
- Dockerizing Java Apps using Jib
- Continuous Development for Kubernetes (Slides)
- Introducing Jib — build Java Docker images better
- awesome-kubernetes - A curated list for awesome kubernetes sources
- Kubernetes Helm: Why It Matters
- Draft vs Gitkube vs Helm vs Ksonnet vs Metaparticle vs Skaffold
Top comments (3)
We are building an open-source dev tool for Kubernetes to make the dev cycle even faster:
github.com/okteto/okteto
We have a java sample in case you want to give it a try:
okteto.com/blog/how-to-develop-jav...
A very useful article. Skaffold is a great tool, but the documentation is limited and it can be tricky getting all the different parts working together correctly. Especially for local dev.
Thanks Zoltan,
Really great.
Cheers,
Adrian