We expose notes on Kubernetes manifests with the focus on deploying a full stack Rails app on a local Kubernetes cluster, powered by Minikube, with the help of Tilt.
no Helm charts, just plain Kubernetes yaml manifests.
To orchestrate the containers with Kubernetes, you firstly build and push images and then send manually manifests with kubectl apply
to the Kubernetes engine. These manifests represent the desired state of your containers. The first part can be automated with a Github action. However, the deployment can be tedious to manage in dev mode.
We will use Tilt to orchestrate the (local) deployment: it just needs the Dockerfiles, the Kubernetes manifests, and a Tiltfile to conduct the deployment. It also automatically builds the images, no need to push them. You can also use a dev mode where changes in code are synchronised with the containers with the live_update
function: a cool tool. You can also add Starlark code.
Testing and setting up a Minikube cluster is only part of the job. In real life, you may need to secure your databases, manage permissions, DNS, service discovery and more. In a following post, we will use another tool GruCloud. It is designed to go live in the cloud with the main providers, AWS EKS or GCC and Azure.
1. The Demo app
We want to run on Kubernetes a simple app to visualise the load-balancing of the pods as scaled. It uses the classic full stack tooling: PostgreSQL, Redis, background job framework Sidekiq, web sockets with ActionCable, and a web server Nginx in front of Rails. It is SSR with the lightweight Preact. It connects to the Kubernetes API to broadcast the existing pods-ID: on every new connection (or page refresh), the counters of the existing pods are refreshed and broadcasted. It also broadcasts the number of page hits and a click counter to any other connected browser. If you change the number of replicas, you can see the load balancing on page refresh or new connected client.
This is done async by using workers, broadcasting the data and rendering with state mutation client side.
2. Vocabulary
Best practices
Follow the guide!
REST objects
Most objects in Kubernetes are REST objects, that can be created, deleted, patched. This is done by sending requests with the appropriate verb to the API: kubectl create
or kubectl delete
or kubectl apply
for example.
All Kubernetes objects can be written in json or yaml form . You can use the yaml processor yq.
Configs
All environment variables and configs will be set by Kubernetes, not coded in the app (cf 12factor). The data is passed to the pods with volumes such as ConfigMap and Secret objects.
kind: ConfigMap
apiVersion: v1
metadata:
name: config
data:
POSTGRES_HOST: "pg-svc"
BUNDLE_PATH: "vendor/bundle"
...
For the credentials, the text string values have to be base64 encoded. For example, with the string "postgres", you get:
echo -n 'postgres' | base64
> cG9zdGdyZXM=
and you add this to a Secret
object:
kind: Secret
apiVersion: v1
metadata:
name: secrets
type: Opaque
data:
POSTGRES_USER: cG9zdGdyZXM= # postgres
...
Then you can pass the data to the containers with:
envFrom:
- configMapRef:
name: config
- secretRef:
name: secrets
Deployment
We use a "Deployment" object for stateless processes. This wraps the "ReplicaSet" controller which guarantees the number of living pods and enables rolling updates or rolling back. The pods englobe the containers. This Russian doll nesting explains the way manifests are built with matching labels.
You understand the ReplicaSet action when a pod is deployed with a Deployment and you try to delete it: a new one will immediately be created because of the ReplicaSet. It will aim to recreate one unless you set the count to zero. You should delete the Deployment instead.
StatefulSet
Stateful processes, typically databases, should be deployed as a StatefulSet. We will use local storage here, but in real life, we would use network attached storage or cloud volumes or managed database services. Indeed, databases should be clustered, for high availability and disaster recovery.
This object is used because there is a master/slave relation to sync the pods, with a unique indexing. A StatefulSet comes with a PersistentVolume and a PersistentVolumeClaim.
Services
A pod needs a service to be able to communicate with the cluster's network. Because pods can be created and deleted (think of a version update for example), the IP will change. The Service
will take care of the follow-up.
The Service
object links ports with any pod matching the defined metadata.name
label::
kind: Service
apiVersion: v1
metadata:
name: pgsql
spec:
type: ClusterIP
ports:
port: 5432
targetPort: 5432
Every service is a load balancer, but unless described with spec.type: NodePort
, all are internal services, of default type ClusterIP
, meaning no access from outside the cluster.
The entry point of the app is via Nginx, thus we want the Nginx service to be external. We can use either
type: LoadBalancer
ortype: NodePort
to expose the app outside the cluster. However, in the case ofLoadBalancer
, you have to pay for each service. ForNodePort
, theports.nodePort
value needs to be above 30.000. There exists an alternative, theIngress
. To shorten, we do not use it here with Minikube.
Mapping external services Youtube Google
You may want to use remote hosted databases instead of deploying databases into the cluster.
In your playground, you may use free tiers such as ElephantSQL or Redislabs. They provide an URL.
You can put the IP address directly into an env variable, so into a ConfigMap
and pass this data to the pods.
Kubernetes can abstract the URLs and handle CNAME redirections and Endpoints
objects.
Take for example an ElephantSQL service whose database-url
is:
postgres://usr:pwd@batyr.db.elephantsql.com/mydb
We use a Service
object of type: ExternalName
. This will perform a CNAME redirection. There is no port involved here, and no reference to a pod.
kind: Service
apiVersion: v1
metadata:
name: pqsql
spec:
type: ExternalName
externalName: batyr.db.elephantsql.com
We can now use:
postgres://usr:pwd@pgsql/db
thus the ConfigMap
is:
kind: ConfigMap
apiVersion: v1
metadata:
name: env
data:
POSTGRES_HOST: "pgsql"
...
If you change the provider, just change the spec.externalName
.
If we want to address the port, we need more. Take for example:
redis://user:pwd@redis-13444.c256.us-east-1-4.ec2.cloud.redislabs.com:13444
First we can discover the IP from the URI with nslookup
or ping
against the URI (server name):
nslookup redis-13444.c256.[..].redislabs.com
-> 54.164.246.16
Then we create a Service
object but without any pod selector:
kind: Service
apiVersion: v1
metadata:
name: redisdb
namespace: stage-v1
spec:
type: ClusterIP
ports:
- port: 6379
targetPort: 13444
Then create Endpoints
object and insert manually the IP address (but no loopback
) and use the same name. It doesn't use selectors but a DNS. It will receive the traffic from the service with matching name.
kind: Endpoints
apiVersion: v1
metadata:
name: redisdb #<- Service name
namespace: stage-v1
subsets:
- addresses:
- ip: 54.164.246.16 #<- manual discovery
ports:
- port: 13444
You now can access the database without any IP address anywhere in the code.
redis://user:pwd@redisdb
and once base64 encoded, we put this value in a Secret
object for the credentials.
kind: Secret
apiVersion: v1
metadata:
name: env
data:
REDIS_DB: "cmVkaXM6Ly91c2VyOnB3ZEByZWRpc2Ri"
# redis://user:pwd@redisdb
# redis://user:pwd@redis-13444.c256.us-east-1-4.ec2.cloud.redislabs.com:13444
...
Ephemeral volumes with pods
We can illustrate the host based ephemeral volume emptyDir
with the ambassador pattern. In other words, Nginx is proxying Rails within the same pod. They also share the same network.
spec.volumes:
- name: shared-data
emptyDir: {}
spec.containers:
- name: nginx
image: nginx:1.21.1-alpine
volumeMounts:
- name: shared-data
mountPath: /usr/share/nginx/html
- name: rails
image: usr/rails-base
volumeMounts:
- name: shared-data
mountPath: /public
We then just use the 'naked' Nginx official image since the config will be passed with another volume, the ConfigMap.
We will use a separate pod for Nginx in front of Rails.
Readiness and Liveness: Youtube Google
A pod may need a readiness and/or liveness
probe. The first means that we have a signal when the pod is ready to accept traffic, and the second is when a pod is dead or alive. By default, everything is green.
The kube-proxy
will then avoid sending traffic to a pod whose readiness probe failed.
A readiness probe shouldn't rely on dependencies on services such as a database, a migration.
The main kinds of probes are httpGet
and exec
.
An example: the readiness
probe for the Rails pod sends an HTTP request to an (internal) endpoint. The method #ready
is render json: { status: 200 }
.
readinessProbe:
httpGet:
path: /ready
port: 3000
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 2
Readiness probes of type exec
can be:
- Redis:
redis-cli ping
- PostgreSQL:
pg_isready -U postgres
.
Resources Youtube Google
The resources are of two types:
- CPU consumption, measured in vCPU/Core: 100m <=> 0.1%. For example, 8 threads can consume 1 CPU second in 0.125 seconds.
- memory, measured in mega-bytes.
The request
of resource is for Kubernetes to schedule that resource on a node that can give this request. This can be set by retroaction.
The limit
of resource is to ensure that a container won't exceed this number, and then scaling comes into play when monitoring the resources. You can monitor the resources with the Kubernetes dashboard or by using the command kubectl describe node minikube
.
You can set values:
resources:
requests:
memory: "25Mi"
cpu: "25m"
limits:
cpu: "50m"
memory: "50Mi"
The limits may be set by retroaction.
Scaling
Source Kubernetes
Source learnk8s blog
Then scale
verb modifies the number of replicas of a process:
kubectl scale deploy sidekiq-dep --replicas=2
kubectl get rs -w
and you watch the ReplicaSet working.
With these measures, but not limited to, Kubernetes can perform horizontal autoscaling. You give specific rules for how to scale up or down with autoscale
.
kubectl autoscale deploy rails-dep --min=1 --max=3 --cpu-percent=80
This means Kubernetes can continuously determine if there is a need for more or fewer pods running.
We can use external metrics; for example, monitor with Prometheus, or use a Redis queue length to monitor processes like Sidekiq.
No log collector is implemented here.
Costs! Beware of the costs!
Kubernetes is all about making your app easily scalable, so costs need to be monitored. For example, Kubecost has free tiers.
Rolling update and back
You can change for example the image used by a container (named "backend" below):
kubectl set image deploy rails-dep backend=usr/rails-base-2
then, check:
kubectl rollout status deploy rails-dep
#or
kubectl get rs -w
and in case of crash looping, for example, you can roll back:
kubectl rollout undo deploy rails-dep
We can also make canary deployments by putting labels.
Roles
As a normal user, we have an admin-user
Service Account and an associated Cluster Role Binding. By default, you have restricted privileges in a pod. You need to set a ServiceAccount
object with the RBAC policy, with Role
, RoleBinding
and ServiceAccount
to the Sidekiq deployment when you want to query from the container against the API with specific verbs (get
, watch
, list
as shown below).
{
"kind": "ServiceAccount",
"apiVersion": "v1",
"metadata": {
"name": "sidekiq" # <- serviceAccountName in deploy of this resource
}
}
kind: Role #<- more permissions with ClusterRole (for eg nodes)
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: query-pods #<- match rolebinding.roleref.name
# namespace: test
rules:
- apiGroups: [""] #<- core API group
resources: ["pods", "service"]
verbs: ["get", "watch", "list"]
---
kind: RoleBinding #<- ClusterRoleBnding if ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: query-pods
# namespace: test <- none if ClusterRole
subjects:
- kind: ServiceAccount
name: sidekiq
# apiGroup: rbac.authorization.k8s.io ??
roleRef:
kind: Role
name: query-pods #<- must match role.metadata.name
apiGroup: rbac.authorization.k8s.io
3. Local Kubernetes cluster with persistent volumes
3.1 Postgres
In this example, we bring the database into the cluster so we will set up local volumes.
The standard
PV available on the cluster is returned by the command kubectl get storageClass
on the system. This is an important topic in real life.
We bind a Persistent Volume (PV) ( admin role) with a Persistent Volume Claim (PVC) (dev role).
#pg-dep.yml
kind: Service
apiVersion: v1
metadata:
name: pg-svc
labels:
app: pg # <- must match with the pod
spec:
ports:
- protocol: TCP
port: 5432 # <- service port opened for Rails
selector:
# the set of pods with the name 'pg' is targeted by this service
app: pg
---
# Deployment
apiVersion: apps/v1
kind: StatefulSet
metadata:
# about the deployment itself. Gives a name of the DEPLOYMENT
name: pg-dep
labels:
app: pg
spec: # of the deployment
serviceName: pg-dep
replicas: 1
selector:
# the deployment must match all pods with the label "app: pg"
matchLabels:
# the label for the POD that the deployment is targeting
app: pg # match spec.template.labels for the pod
template: # blue print of a pod
metadata:
name: pg-pod
# label for the POD that the deployment is deploying
labels:
app: pg # match spec.selector.matchlabels
spec:
volumes:
- name: pg-pv # must match PV
persistentVolumeClaim:
claimName: pg-pvc # must match PVC
containers:
- name: pg-container
image: postgres:13.3-alpine
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 5432
volumeMounts:
- mountPath: $(PGDATA)
name: pg-pv # must match pv
readOnly: false
envFrom:
- configMapRef:
name: config
- secretRef:
name: secrets
readinessProbe:
exec:
command: ["pg_isready", "-U", "postgres"]
periodSeconds: 30
timeoutSeconds: 10
initialDelaySeconds: 30
resources:
requests:
cpu: 100m
---
# we bind the resource PV to the pod
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pg-pvc
spec:
#storageClassName: standard
accessModes:
- ReadWriteOnce #<- means only one node
resources:
requests:
storage: 100Mi
and the volume:
#pg-db-pv.yml
apiVersion: v1
kind: PersistentVolume
metadata:
name: pg-pv
labels:
app: pg
spec:
storageClassName: standard
capacity:
storage: 150Mi
accessModes:
- ReadWriteOnce #<- means only one node
hostPath: # for Minikube, emulate net. attached vol.
path: "/tmp/data"
3.2 Webserver Nginx
We use the web server image and set an external service for the Nginx deployment since this is the entry-point of the app.
#nginx-dep.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-dep
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: usr/nginx-ws
imagePullPolicy: "Always"
name: frontend
resources:
requests:
cpu: 500m
ports:
- containerPort: 9000
volumeMounts:
- mountPath: "/etc/nginx/conf.d" # mount nginx-conf volume to /etc/nginx
readOnly: true
name: nginx-conf
volumes:
- name: nginx-conf
configMap:
name: nginx-conf # place ConfigMap `nginx-conf` on /etc/nginx
items:
- key: myconfig.conf
path: default.conf
---
apiVersion: v1
kind: Service
metadata:
name: nginx-svc
labels:
app: nginx
spec:
type: NodePort # with LoadBalancer, you pay!
selector:
app: nginx
ports:
- protocol: TCP
# port exposed by the container
port: 80
# the port the app is listening on targetPort
targetPort: 9000
nodePort: 31000
This illustrates the use of a ConfigMap volume to pass the Nginx config via a mountPath into the pod source
#nginx-conf.yml
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-conf
data:
myconfig.conf: |
upstream puma {
server rails-svc:3000;
keepalive 1024;
}
access_log /dev/stdout main;
error_log /dev/stdout info;
server {
listen 9000 default_server;
root /usr/share/nginx/html;
try_files $uri @puma;
access_log off;
gzip_static on;
expires max;
add_header Cache-Control public;
add_header Last-Modified "";
add_header Etag "";
location @puma {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header Host $http_host;
proxy_pass_header Set-Cookie;
proxy_redirect off;
proxy_pass http://puma;
}
location /cable {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_pass "http://cable-svc:28080";
}
error_page 500 502 503 504 /500.html;
error_page 404 /404.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
3.3 Redis
For the Redis store, we didn't pass a config here. We should have used a ConfigMap.
#redis-dep.yml
apiVersion: v1
kind: Service
metadata:
name: redis-svc
spec:
ports:
- port: 6379
targetPort: 6379
name: client
selector:
app: redis
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis-dep
labels:
app: redis # match spec.template.labels
spec:
serviceName: redis
selector:
matchLabels:
app: redis
replicas: 1
template:
metadata:
name: redis-pod
labels:
app: redis # # match spec.selector.matchLabels
spec:
containers:
- name: redis
image: redis:6.2.4-alpine
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 6379
command: ["redis-server"]
resources:
requests:
cpu: 100m
readinessProbe:
exec:
command:
- redis-cli
- ping
initialDelaySeconds: 20
periodSeconds: 30
timeoutSeconds: 3
volumeMounts:
- name: data
mountPath: "/data"
readOnly: false
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
#storageClassName: "standard"
resources:
requests:
storage: 100Mi
and the PersistentVolume:
#redis-pv.yml
apiVersion: v1
kind: PersistentVolume
metadata:
name: data
labels:
app: data
spec:
#storageClassName: standard
capacity:
storage: 150Mi
accessModes:
- ReadWriteOnce
hostPath: # for Minikube, to emulate net. attached vol.
path: "/data"
3.4 Rails
We will use one image named "usr/rails-base" here. For production deployment, we would use the URL of a repository in a container registry (e.g. ECR).
The Rails image is build to be slim for faster loads. It is a two stage built image around 240Mb, run without root privileges. It is used by Rails, Sidekiq and Cable, with different commands. The Nginx pod will also use it to extract the static assets, and the migration Job will use it too.
❗ Do not tag it! Tilt needs to timestamp it through the tag.
All ENV vars and credentials will be set within Kubernetes. We will use two volumes, a ConfigMap for the ENV vars and a Secret for credentials:
#config.yml
#rails-dep.yml
apiVersion: v1
kind: ConfigMap
metadata:
name: config
data:
POSTGRES_HOST: "pg-svc" # <- name of the service that exposes the Postgres pod
POSTGRES_DB: "kubedb"
BUNDLE_PATH: "vendor/bundle" #<- gems are "localized"
RAILS_ENV: "production"
RAILS_LOG_TO_STDOUT: "true"
RAILS_SERVE_STATIC_FILES: "false"
REDIS_DB: "redis://redis-svc:6379/0"
REDIS_SIDEKIQ: "redis://redis-svc:6379/1"
REDIS_CACHE: "redis://redis-svc:6379/2"
REDIS_CABLE: "redis://redis-svc:6379/3"
The credentials need to be converted with echo -n 'postgres' | base64
which gives
#secrets.yml
apiVersion: v1
kind: Secret
metadata:
name: secrets
type: Opaque
data:
POSTGRES_USER: cG9zdGdyZXM= # postgres
POSTGRES_PASSWORD: ZG9ja2VycGFzc3dvcmQ= # dockerpassword
RAILS_MASTER_KEY: NWE0YmU0MzVjNmViODdhMWE5NTA3M2Y0YTRjYWNjYTg=
The manifest of the Rails deployment and its service:
#rails-dep.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: rails-dep
labels: # must match the service
app: rails
spec:
replicas: 1
selector:
matchLabels: # which pods are we deploying
app: rails
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
template:
metadata:
labels: # must match service and replicaset matchlabel
app: rails
spec:
containers:
- name: backend
image: usr/rails-base
imagePullPolicy: "Always"
command: ["bundle"]
args: ["exec", "rails", "server", "-b", "0.0.0.0"]
resources:
requests:
cpu: 500m
ports:
- containerPort: 3000
envFrom:
- configMapRef:
name: config
- secretRef:
name: secrets
readinessProbe:
httpGet:
path: /ready
port: 3000
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
name: rails-svc
labels:
app: rails
spec:
selector:
app: rails
type: ClusterIP # default type
ports:
- protocol: TCP
# port exposed by the service
port: 3000
# the port the app is listening on targetPort
targetPort: 3000
name: http
3.5 Cable and Sidekiq
The deployment of the "cable" is identical except for the args for the command and port. It has its service to expose the cable pod.
The "worker" deployment is also based on the same image, with its own "command" and "args" but no service is required for the worker since it communicates with Rails via a Redis queue.
Cable
#cable-dep.yml
spec:
containers:
- name: cable
image: usr/rails-base
imagePullPolicy: "Always"
command: ["bundle"]
args: ["exec", "puma", "-p", "28080", "./cable/config.ru"]
resources:
requests:
cpu: 150m
ports:
- containerPort: 28080
envFrom:
- configMapRef:
name: config
- secretRef:
name: secrets
---
apiVersion: v1
kind: Service
metadata:
name: cable-svc
labels:
app: cable
spec:
selector:
app: cable
type: ClusterIP # default type
ports:
- protocol: TCP
# port exposed by the service
port: 28080
# the port the app is listening on targetPort
targetPort: 28080
name: http
Sidekiq:
#sidekiq-dep.yml
spec:
containers:
- name: sidekiq
image: usr/rails-base
imagePullPolicy: "Always"
command: ["bundle"]
args: ["exec", "sidekiq", "-C", "./config/sidekiq.yml"]
resources:
requests:
cpu: 150m
envFrom:
- configMapRef:
name: config
- secretRef:
name: secrets
We can add a liveness
probe by using the gem sidekiq_alive.
3.6 Calling the Kubernetes API
As a normal user, we have an admin-user
Service Account and an associated Cluster Role Binding. Within a pod, by default everything is restricted. We have created a special role used by the Sidekiq deployment.
If we want to call the Kubernetes API to get data from the cluster within our app, we can:
- directly access to the REST API. The credentials are present in each pod so that we can read them; within a thread or Sidekiq job, we execute a
CURL
call to theapiserver
. The result should be parsed and filtered.
`curl --cacert #{cacert} -H "Authorization: Bearer #{token}" https://kubernetes.default.svc/api/... `
Note that the pod check against the server with
cacert
(PEM files) happens at the TLS level, not at the HTTPS level, thus this exec form is the easiest to use and harmelss since no external input.
Note that you may take profit from using the gemOj
for speeding up the JSON parsing.
- do it from within a pod by tunnelling with
kubectl proxy
. The endpoint is on localhost, so we run a side-car pod with a Kubernetes server along with Sidekiq (since obviously the call will be run within a job). This solution is "more expensive", a Kubernetes server is a 55 Mb image runningkubectl proxy --port 8001
.
URI.open(http://localhost:8001/api/v1/namespaces/{namespace}/pods)
To render in the browser, we broadcast the result through a web socket where listener will mutate a state to render (with React).
4. Run this with TILT
We have Docker, the Kubernetes CLI and Minikube installed. A normal workflow is to build and push images from Dockerfiles. You can automate this with a github action. Then, Kubernetes will pull and run the manifests that use these images.
We can let Tilt do all this for us and be reactive to changes in the code.
Note that as a Rails user, you may have the gem "tilt" present, so alias
tilt
with "/usr/local/bin/tilt".
Launch Minikube
We may want to namespace the project so that you can isolate different versions or modes (staging, prod) and also clean everything easily. We used the utilities kubectx, kubens to set the namespace. After it is set, any pod (except volumes) will use the current namespace.
kubectl create namespace test
# use the utility "kubens" from "kubectx" to assign a namespace
kubens stage-v1
# check:
kubens # => "stage-v1" marked
kubectl config get-contexts # => Minikube marked
#or
kubectl config current-context
A good practice is to document the namespace: apply the following manifest and use namespace: prod
in the metadata
of each kind
.
kind: Namespace
apiVersion: v1
metadata:
name: stage-v1
Now the project is isolated within a namespace, we can launch Minikube with minikube start
.
List of files
Our files are:
/app
/config
/public
...
/dckf
|_ _builder.Dockerfile
|_ _alpine.Dockerfile
|_ _nginx.Dockefile
Tiltfile
/kube
|_ config.yml
|_ secrets.yml
|_ nginx-config.yml
|_ pg-pv.yml
|_ postgres-dep.yml
|_ redis-pv.yml
|_ redis-dep.yml
|_ rails-dep.yml
|_ sidekiq-dep.yml
|_ cable-dep.yml
|_ nginx-dep.yml
|_ migrate.yml
Deploy
To deploy on the Minikube cluster, once we have built and pulled the images, we run the following commands against all our manifests ❗ in an orderly manner: first run the configs (and secrets) needed by the others processes.
kubectl apply -f ./kube/config.yml
kubectl apply -f ./kube/secrets.yml
...
kubectl apply -f ./kube/rails-dep.yml
...
kubectl apply -f ./kube/migrate.yml
minikube service nginx-svc #<- our app entry point
We will automate all this with Tilt. The Tilt engine will read a Tiltfile
and build the full project. It only needs the code (the Dockerfiles and the Kubernetes manifests) and one command.
In this Tiltfile, we describe all the actions we want Tilt and Kubernetes to perform: building images, running manifests and managing dependencies. The one below is a very basic image builder and runner.
Note that you should not tag your images as Tilt will put a timestamp, and the named used in
docker_build
should match the name of the image in the deployment manifest.
#Tilfile
# apply configs and create volumes
k8s_yaml(['./kube/config.yml','./kube/secrets.yml','./kube/nginx-config.yml','./kube/pg-db-pv.yml',
])
# apply databases adapters
k8s_yaml(['./kube/postgres-dep.yml','./kube/redis-dep.yml'])
# <- building images and live changes
docker_build( 'builder', # <- Bob
'.',
dockerfile="./dckf/builder.Dockerfile",
build_args={
"RUBY_VERSION": "3.0.2-alpine",
"NODE_ENV": "production",
"RAILS_ENV": "production",
"BUNDLER_VERSION": "2.2.25"
}
)
docker_build('usr/rails-base', #<- uses Bob
'.',
build_args={
"RUBY_VERSION": "3.0.2-alpine",
"RAILS_ENV": "production",
"RAILS_LOG_TO_STDOUT": "true",
},
dockerfile='./docker/rails.Dockerfile'
)
docker_build("usr/nginx-ws", # <- uses Bob
".",
dockerfile="./dckf/nginx-split.Dockerfile",
build_args={
"RUBY_VERSION": "3.0.2-alpine", "RAILS_ENV": "production"
}
)
# -> end images
# dependencies
k8s_resource('sidekiq-dep', resource_deps=['redis-dep'])
k8s_resource('rails-dep', resource_deps=['pg-dep', 'redis-dep'])
k8s_resource('cable-dep', resource_deps=['redis-dep'])
k8s_resource('nginx-dep', resource_deps=['rails-dep'])
# apply processes
k8s_yaml(['./kube/rails-dep.yml', './kube/sidekiq-dep.yml', './kube/cable-dep.yml', k8s_yaml('./kube/nginx-dep.yml'])
# <- creates manual/auto action button in the Tilt GUI
# migration
k8s_resource('db-migrate',
resource_deps=['rails-dep','pg-dep'],
trigger_mode=TRIGGER_MODE_MANUAL,
auto_init=False
)
k8s_yaml('./kube-split/migrate.yml' )
# auto (&manual) display pods in GUI
local_resource('All pods',
'kubectl get pods',
resource_deps=['rails-dep','sidekiq-dep','cable-dep','nginx-dep']
)
# ->
allow_k8s_contexts('minikube')
k8s_resource('nginx-dep', port_forwards='31000')
Notice the two stage build: the image is only build once even if reused four times. You may skip the build phase but you need to push the images to the container registry.
Notice the resources dependencies with
k8s_resource
which makes things easy!
To run this:
tilt up
or all in one (unless you applied the namespace manifest or needed to create the Postgres database): run the Docker daemon (open -a Docker
on OSX)
minikube start && kubectl create ns test && kubens test && tilt up
Then we need to migrate
(see below), and with our settings, we can navigate to http://localhost:31000 to visualise the app.
For a quick clean up, run (as docker-compose down
):
tilt down
#or
kubectl delete ns test
Live update
We can use the function live_update
that syncs the code between the host and the container.
Instead of a slim image, you need all the Rails tooling for hot-reloading and work RAILS_ENV=development
so use this dockerfile. You target /app
directory which contains the "active" code since any modification in any other folder needs a complete rebuild; for this, you can use live_update=[sync('app','/app/app')]
. Then any change in the backend code will be immediately available. Any other change in the code will trigger a rebuild.
If you want to exclude some file, add ignore=['README.md', 'log','tmp']
for example.
You remove the "Cable" process" and "Nginx" and keep only Rails (and Sidekiq since it communicates via the pub/sub Redis).
❗ Front-end code change needs page refresh.
The new Tilfile looks like (the files specific to this mode are put in a seperate folder "/kube-dev" and the common are unchanged).
#dev-tilfile
k8s_yaml(['./kube-dev/config.yml','./kube/secrets.yml', './kube/pg-db-pv.yml', './kube/redis-pv.yml', './kube/role.yml', './kube/service-account.yml'
])
k8s_yaml(['./kube/postgres-dep.yml', './kube/redis-dep.yml'])
docker_build( 'builder-dev', '.',
dockerfile="./docker/dev.Dockerfile",
build_args={
"RUBY_VERSION": "3.0.2-alpine", "NODE_ENV": "development", "RAILS_ENV": "development", "BUNDLER_VERSION": "2.2.25",
},
live_update=[sync('app','/app/app/')],
ignore=['tmp','log']
)
k8s_yaml(['./kube-dev/rails-dep.yml','./kube-dev/sidekiq-dep.yml'])
k8s_resource('rails-dep', resource_deps=['pg-dep', 'redis-dep', 'db-migrate'])
k8s_resource('sidekiq-dep', resource_deps=['redis-dep','pg-dep', 'db-migrate'])
k8s_resource('db-migrate',
resource_deps=['pg-dep'],
trigger_mode=TRIGGER_MODE_MANUAL,
auto_init=False
)
k8s_yaml('./kube-dev/migrate.yml' )
allow_k8s_contexts('minikube')
k8s_resource('rails-dep', port_forwards='31000') #<- "dev"
Run tilt -f dev.tiltfile up
.
Annotations
You may use annotations to prevent for example the lengthy rebuild of Postgres after you tilt down
. Add an annotation to keep them running:
metadata.annotations.tilt.dev/down-policy: keep
#postgres-dep.yml
apiVersion: apps/v1
kind: StatefulSet
metadata:
# about the deployment itself. Gives a name of the DEPLOYMENT
name: pg-dep
labels:
app: pg
annotations:
tilt.dev/down-policy: keep
Migration Job
The topic of performing safe migrations is a very important subject of its own (new migration on old code, old migration with new code).
Every time you stop Minikube and Tilt, you need to run a migration. You can an initContainer that waits for the db, or a Job.
#migrate.yml
apiVersion: batch/v1
kind: Job
metadata:
name: db-migrate
spec:
template:
spec:
restartPolicy: Never
containers:
- name: db-migrate
image: usr/rails-base
imagePullPolicy: IfNotPresent
command: ["bundle", "exec", "rails", "db:migrate"]
envFrom:
- secretRef:
name: secrets
- configMapRef:
name: config
You can run k8s_yaml('./kube/migrate.yml')
to apply this.
Alternatively, you can do this programatically with Tilt with k8s_resource
where you specify the needed dependencies, and then apply
with k8s_yaml
.
Below is an example of the manual custom db-migrate
action. With the two last flags, we have an action button in the GUI to trigger this migration when ready.
k8s_resource('db-migrate',
resource_deps=['rails-dep','pg-dep'],
trigger_mode=TRIGGER_MODE_MANUAL,
auto_init=False
)
k8s_yaml('./kube-split/migrate.yml' )
[K8s EVENT: Pod db-migrate-vh459 (ns: test)] Successfully pulled image "/rails-base" in 1.481341763s
Migrating to CreateCounters (20210613160256)
== 20210613160256 CreateCounters: migrating ===================================
-- create_table(:counters)
-> 0.4681s
== 20210613160256 CreateCounters: migrated (0.4694s) ==========================
If needed, run kubectl delete job db-migrate
to be able to run the job.
Using local_resource
If we want to list automatically the Rails pods, we add a "local_resource" in the Tiltfile. As such, it will be run automatically when Rails is ready. This will also add a button in the UI that we can rerun on demand.
local_resource('Rails pods',
"kubectl get pods -l app=rails -o go-template --template '{{range .items}}{{.metadata.name}}{{\"\n\"}}{{end}}' -o=name ",
resource_deps=['rails-dep']
)
Tilt UI
The Tilt UI comes in two flavours:
- a terminal, essentially an easy log collector
- a nice GUI at "http://localhost:10350" from which you can for example easily read the logs or trigger rebuilds.
Kubernetes dashboard
To enable the Kubernetes dashboard, run:
minikube addons list
minikube addons enable dashboard
minikube addons enable metrics-server
The command minikube dashboard
shows the full state of the cluster. Just select the namespace. We can see the logs for each pod, the resource consumption, globally and per process.
Since you have admin credentials, you can deploy your app just using the Dashboard. Your images should already be built.
Resources with kubectl
The command:
kubectl describe node minikube
gives the details of the resource usage of the cluster.
HPA
Once we have the measures, we can start to autopilot selected processes with HorizontalPodAutoscaler.
We can run the command kubectl autoscale
local_resource('hpa',
'kubectl autoscale deployment rails-dep --cpu-percent=80 --min=1 --max=3',
resource_deps=['rails-dep'],
trigger_mode=TRIGGER_MODE_MANUAL,
auto_init=False
)
or apply the manifest:
#rails-hpa.yml
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: rails-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: rails-dep
minReplicas: 1
maxReplicas: 3
#targetCPUUtilizationPercentage: 80
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
with
k8s_yaml('./kube/rails-hpa.yml')
k8s_resource('rails-hap', resource_deps=['rails-dep'])
We can use simple load tests to check.
5. Misc files
Cheat sheet
# waiting pods checking
kubectl get pods -w -o wide
# follow the logs of a pod
kubectl logs rails-nginx-dep-1231313 -c rails -f
# delete a deployment (pod+svc+replicaset) or job
kubectl delete deploy rails-dep
kubectl delete job db-migrate
# launch a browser
minikube service rails-nginx-svc # !! OSX !!
# launch the Kubernetes dashboard
minikube dashboard
# ssh into the VM (user docker, private key)
minikube ssh
# generate a token which grants admin permissions:
kubectl --namespace kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')
# ssh
ssh -i ~/.minikube/machines/minikube/id_rsa docker@$(minikube ip)
# get programmatically info
kubectl get service nginx-svc -o jsonpath='{.spec.ports[0].nodePort}'
# execute some commands:
export POD_NAME=$(kubectl get pods -l app=rails -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}') &&
# kubectl exec $POD_NAME -- bundle exec rails db:migrate' ,resource_deps=['rails-dep','sidekiq-dep','cable-dep',','nginx-dep'])
# execute a command in a pod with several containers (name it with "-c")
kubectl exec it -rails-nginx-dep-1234 -c rails -- irb
irb(main):001:0> require 'socket'; Socket.gethostname
=> "rails-nginx-dep-55bc5f77dc-48wg4"
# execute a command in a pod
kubectl exec -it pg-dep-0 -- psql -w -U postgres
kubectl exec -it redis-dep-0 -- redis-cli
# rolling update by setting a new image to a deployment
kubectl set image deployment/rails-dep usr/rails-base2
# check
kubectl rollout status deployment/rails-dep
# undo in case of a problem
kubectl rollout undo depoyment/rails-dep
yaml <=> json
You can convert a bunch of files from yaml into json (or vice-versa) from your current directory with yq
:
for file in *.yml; do
base=${file%.yml};
# yml > json
yq eval -o=j $base.yml > $base.json;
# or json > yml
yq eval -P $base.json > $base.yml;
done
Dev.Dockerfile
ARG RUBY_VERSION
FROM ruby:${RUBY_VERSION:-3.0.2-alpine}
ARG BUNDLER_VERSION
ARG NODE_ENV
ARG RAILS_ENV
ENV BUNDLER_VERSION=${BUNDLER_VERSION:-2.2.26} \
RAILS_ENV=${RAILS_ENV:-development} \
NODE_ENV=${NODE_ENV:-development}
RUN apk -U upgrade && apk add --no-cache postgresql-dev nodejs yarn build-base tzdata curl \
&& rm -rf /var/cache/apk/*
WORKDIR /app
COPY Gemfile Gemfile.lock package.json yarn.lock ./
ENV LANG=C.UTF-8 BUNDLE_JOBS=4 BUNDLE_RETRY=3
RUN gem install bundler:${BUNDLER_VERSION} --no-document \
&& bundle install --quiet \
&& rm -rf $GEM_HOME/cache/* \
&& yarn install --check-files --silent && yarn cache clean
COPY . ./
Builder
ARG RUBY_VERSION=3.0.2-alpine
FROM ruby:${RUBY_VERSION:-3.0.1-alpine}
ARG BUNDLER_VERSION
ARG NODE_ENV
ARG RAILS_ENV
RUN apk -U upgrade && apk add --no-cache \
postgresql-dev nodejs yarn build-base tzdata
ENV PATH /app/bin:$PATH
WORKDIR /app
COPY Gemfile Gemfile.lock package.json yarn.lock ./
ENV LANG=C.UTF-8 \
BUNDLE_JOBS=4 \
BUNDLE_RETRY=3 \
BUNDLE_PATH='vendor/bundle'
RUN gem install bundler:${BUNDLER_VERSION} --no-document \
&& bundle config set --without 'development test' \
&& bundle install --quiet \
&& && rm -rf $GEM_HOME/cache/* \
&& yarn --check-files --silent --production && yarn cache clean
COPY . ./
RUN bundle exec rails webpacker:compile assets:clean
Rails Dockerfile
FROM builder AS bob
FROM ruby:${RUBY_VERSION:-3.0.2-alpine}
ARG RAILS_ENV
ARG NODE_ENV
ARG RAILS_LOG_TO_STDOUT
RUN apk -U upgrade && apk add --no-cache libpq tzdata netcat-openbsd curl \
&& rm -rf /var/cache/apk/* \
&& adduser --disabled-password app-user
# -disabled-password doesn't assign a password, so cannot login
USER app-user
COPY --from=bob --chown=app-user /app /app
ENV RAILS_ENV=$RAILS_ENV \
RAILS_LOG_TO_STDOUT=$RAILS_LOG_TO_STDOUT \
BUNDLE_PATH='vendor/bundle'
WORKDIR /app
RUN rm -rf node_modules
Web-server Dockerfile
#_nginx-ws.Dockerfile
FROM usr/rails-base AS bob
FROM nginx:1.21.1-alpine
COPY --from=bob ./app/public /usr/share/nginx/html
Simple load test
You can run a simple load test:
kubectl run -i --tty load-generator --rm --image=busybox --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://localhost:31000; done"
If we have a domain, we can also use the "ab" Apache Bench: we test here how fast the app can handle 1000 requests, with a maximum of 50 requests running concurrently:
ab -n 1000 -c 50 https://my-domain.top/
[sources]:
- Docker
- Kubernetes
- Youtube Kubernetes Best pratices
- Patterns with containers
- Learnk8s blog
- Youtube-Marcel-Dempers and Youtube-Techworld-with-Nana
Happy coding!
Top comments (2)
Nice article !
One question: How do you manage to run commands which update app files? For example
rails g migration ...
,yarn add ...
and so on.Very nice article, lots of learning, keep posting. I also created a full course on Kubernetes in 10hr on YouTube, please have a look and motivate me. Thanks
youtu.be/toLAU_QPF6o