DEV Community

Cover image for Rails on Kubernetes with Minikube and Tilt
NDREAN
NDREAN

Posted on • Edited on

Rails on Kubernetes with Minikube and Tilt

#rails #kubernetes #docker #devops

We expose notes on Kubernetes manifests with the focus on deploying a full stack Rails app on a local Kubernetes cluster, powered by Minikube, with the help of Tilt.

no Helm charts, just plain Kubernetes yaml manifests.

To orchestrate the containers with Kubernetes, you firstly build and push images and then send manually manifests with kubectl apply to the Kubernetes engine. These manifests represent the desired state of your containers. The first part can be automated with a Github action. However, the deployment can be tedious to manage in dev mode.
We will use Tilt to orchestrate the (local) deployment: it just needs the Dockerfiles, the Kubernetes manifests, and a Tiltfile to conduct the deployment. It also automatically builds the images, no need to push them. You can also use a dev mode where changes in code are synchronised with the containers with the live_update function: a cool tool. You can also add Starlark code.

Testing and setting up a Minikube cluster is only part of the job. In real life, you may need to secure your databases, manage permissions, DNS, service discovery and more. In a following post, we will use another tool GruCloud. It is designed to go live in the cloud with the main providers, AWS EKS or GCC and Azure.

1. The Demo app

We want to run on Kubernetes a simple app to visualise the load-balancing of the pods as scaled. It uses the classic full stack tooling: PostgreSQL, Redis, background job framework Sidekiq, web sockets with ActionCable, and a web server Nginx in front of Rails. It is SSR with the lightweight Preact. It connects to the Kubernetes API to broadcast the existing pods-ID: on every new connection (or page refresh), the counters of the existing pods are refreshed and broadcasted. It also broadcasts the number of page hits and a click counter to any other connected browser. If you change the number of replicas, you can see the load balancing on page refresh or new connected client.
This is done async by using workers, broadcasting the data and rendering with state mutation client side.

Alt Text

2. Vocabulary

Best practices

Follow the guide!

REST objects

Most objects in Kubernetes are REST objects, that can be created, deleted, patched. This is done by sending requests with the appropriate verb to the API: kubectl create or kubectl delete or kubectl apply for example.
All Kubernetes objects can be written in json or yaml form . You can use the yaml processor yq.

⬆️

Configs

All environment variables and configs will be set by Kubernetes, not coded in the app (cf 12factor). The data is passed to the pods with volumes such as ConfigMap and Secret objects.

kind: ConfigMap
apiVersion: v1
metadata:
  name: config
data:
  POSTGRES_HOST: "pg-svc" 
  BUNDLE_PATH: "vendor/bundle" 
  ...
Enter fullscreen mode Exit fullscreen mode

For the credentials, the text string values have to be base64 encoded. For example, with the string "postgres", you get:

echo -n 'postgres' | base64
> cG9zdGdyZXM=
Enter fullscreen mode Exit fullscreen mode

and you add this to a Secret object:

kind: Secret
apiVersion: v1
metadata:
  name: secrets
type: Opaque
data:
  POSTGRES_USER: cG9zdGdyZXM= # postgres
  ...
Enter fullscreen mode Exit fullscreen mode

Then you can pass the data to the containers with:

envFrom:
  - configMapRef:
    name: config
  - secretRef:
    name: secrets
Enter fullscreen mode Exit fullscreen mode

⬆️

Deployment

We use a "Deployment" object for stateless processes. This wraps the "ReplicaSet" controller which guarantees the number of living pods and enables rolling updates or rolling back. The pods englobe the containers. This Russian doll nesting explains the way manifests are built with matching labels.

You understand the ReplicaSet action when a pod is deployed with a Deployment and you try to delete it: a new one will immediately be created because of the ReplicaSet. It will aim to recreate one unless you set the count to zero. You should delete the Deployment instead.

StatefulSet

Stateful processes, typically databases, should be deployed as a StatefulSet. We will use local storage here, but in real life, we would use network attached storage or cloud volumes or managed database services. Indeed, databases should be clustered, for high availability and disaster recovery.
This object is used because there is a master/slave relation to sync the pods, with a unique indexing. A StatefulSet comes with a PersistentVolume and a PersistentVolumeClaim.

⬆️

Services

A pod needs a service to be able to communicate with the cluster's network. Because pods can be created and deleted (think of a version update for example), the IP will change. The Service will take care of the follow-up.
The Service object links ports with any pod matching the defined metadata.name label::

kind: Service
apiVersion: v1
metadata:
  name: pgsql
spec:
  type: ClusterIP
  ports:
    port: 5432
    targetPort: 5432
Enter fullscreen mode Exit fullscreen mode

Every service is a load balancer, but unless described with spec.type: NodePort, all are internal services, of default type ClusterIP, meaning no access from outside the cluster.

The entry point of the app is via Nginx, thus we want the Nginx service to be external. We can use either type: LoadBalancer or type: NodePort to expose the app outside the cluster. However, in the case of LoadBalancer, you have to pay for each service. For NodePort, the ports.nodePort value needs to be above 30.000. There exists an alternative, the Ingress. To shorten, we do not use it here with Minikube.

⬆️

Mapping external services Youtube Google

You may want to use remote hosted databases instead of deploying databases into the cluster.
In your playground, you may use free tiers such as ElephantSQL or Redislabs. They provide an URL.

You can put the IP address directly into an env variable, so into a ConfigMap and pass this data to the pods.

Kubernetes can abstract the URLs and handle CNAME redirections and Endpoints objects.

Take for example an ElephantSQL service whose database-url is:

postgres://usr:pwd@batyr.db.elephantsql.com/mydb
Enter fullscreen mode Exit fullscreen mode

We use a Service object of type: ExternalName. This will perform a CNAME redirection. There is no port involved here, and no reference to a pod.

kind: Service
apiVersion: v1
metadata:
  name: pqsql
spec:
  type: ExternalName
  externalName: batyr.db.elephantsql.com
Enter fullscreen mode Exit fullscreen mode

We can now use:

postgres://usr:pwd@pgsql/db
Enter fullscreen mode Exit fullscreen mode

thus the ConfigMap is:

kind: ConfigMap
apiVersion: v1
metadata:
  name: env
data:
  POSTGRES_HOST: "pgsql"
  ...
Enter fullscreen mode Exit fullscreen mode

If you change the provider, just change the spec.externalName.

If we want to address the port, we need more. Take for example:

redis://user:pwd@redis-13444.c256.us-east-1-4.ec2.cloud.redislabs.com:13444
Enter fullscreen mode Exit fullscreen mode

First we can discover the IP from the URI with nslookup or ping against the URI (server name):

nslookup redis-13444.c256.[..].redislabs.com
 -> 54.164.246.16
Enter fullscreen mode Exit fullscreen mode

Then we create a Service object but without any pod selector:

kind: Service
apiVersion: v1
metadata:
  name: redisdb 
  namespace: stage-v1
spec:
  type: ClusterIP
  ports:
    - port: 6379
      targetPort: 13444
Enter fullscreen mode Exit fullscreen mode

Then create Endpoints object and insert manually the IP address (but no loopback) and use the same name. It doesn't use selectors but a DNS. It will receive the traffic from the service with matching name.

kind: Endpoints
apiVersion: v1
metadata:
  name: redisdb #<- Service name
  namespace: stage-v1
subsets:
  - addresses:
      - ip: 54.164.246.16 #<- manual discovery
    ports:
      - port: 13444
Enter fullscreen mode Exit fullscreen mode

You now can access the database without any IP address anywhere in the code.

redis://user:pwd@redisdb
Enter fullscreen mode Exit fullscreen mode

and once base64 encoded, we put this value in a Secret object for the credentials.

kind: Secret
apiVersion: v1
metadata:
  name: env
data:
  REDIS_DB: "cmVkaXM6Ly91c2VyOnB3ZEByZWRpc2Ri"
  # redis://user:pwd@redisdb
  # redis://user:pwd@redis-13444.c256.us-east-1-4.ec2.cloud.redislabs.com:13444
  ...
Enter fullscreen mode Exit fullscreen mode

⬆️

Ephemeral volumes with pods

We can illustrate the host based ephemeral volume emptyDir with the ambassador pattern. In other words, Nginx is proxying Rails within the same pod. They also share the same network.

spec.volumes:
  - name: shared-data
    emptyDir: {}
spec.containers:
  - name: nginx
    image: nginx:1.21.1-alpine
    volumeMounts:
      - name: shared-data
        mountPath: /usr/share/nginx/html
  - name: rails
    image: usr/rails-base
    volumeMounts:
    - name: shared-data
      mountPath: /public
Enter fullscreen mode Exit fullscreen mode

We then just use the 'naked' Nginx official image since the config will be passed with another volume, the ConfigMap.

We will use a separate pod for Nginx in front of Rails.

⬆️

Readiness and Liveness: Youtube Google

A pod may need a readiness and/or liveness probe. The first means that we have a signal when the pod is ready to accept traffic, and the second is when a pod is dead or alive. By default, everything is green.
The kube-proxy will then avoid sending traffic to a pod whose readiness probe failed.
A readiness probe shouldn't rely on dependencies on services such as a database, a migration.
The main kinds of probes are httpGet and exec.
An example: the readiness probe for the Rails pod sends an HTTP request to an (internal) endpoint. The method #ready is render json: { status: 200 }.

readinessProbe:
  httpGet:
    path: /ready
    port: 3000
    scheme: HTTP
  initialDelaySeconds: 10
  periodSeconds: 30
  timeoutSeconds: 2
Enter fullscreen mode Exit fullscreen mode

Readiness probes of type exec can be:

  • Redis: redis-cli ping
  • PostgreSQL: pg_isready -U postgres.

⬆️

Resources Youtube Google

The resources are of two types:

  • CPU consumption, measured in vCPU/Core: 100m <=> 0.1%. For example, 8 threads can consume 1 CPU second in 0.125 seconds.
  • memory, measured in mega-bytes.

The request of resource is for Kubernetes to schedule that resource on a node that can give this request. This can be set by retroaction.
The limit of resource is to ensure that a container won't exceed this number, and then scaling comes into play when monitoring the resources. You can monitor the resources with the Kubernetes dashboard or by using the command kubectl describe node minikube.

You can set values:

resources:
   requests:
     memory: "25Mi"
     cpu: "25m"
  limits:
    cpu: "50m"
    memory: "50Mi"
Enter fullscreen mode Exit fullscreen mode

The limits may be set by retroaction.

⬆️

Scaling

Source Kubernetes
Source learnk8s blog

Then scale verb modifies the number of replicas of a process:

kubectl scale deploy sidekiq-dep --replicas=2
kubectl get rs -w
Enter fullscreen mode Exit fullscreen mode

and you watch the ReplicaSet working.

With these measures, but not limited to, Kubernetes can perform horizontal autoscaling. You give specific rules for how to scale up or down with autoscale.

kubectl autoscale deploy rails-dep --min=1 --max=3 --cpu-percent=80
Enter fullscreen mode Exit fullscreen mode

This means Kubernetes can continuously determine if there is a need for more or fewer pods running.

We can use external metrics; for example, monitor with Prometheus, or use a Redis queue length to monitor processes like Sidekiq.
No log collector is implemented here.

Costs! Beware of the costs!

Kubernetes is all about making your app easily scalable, so costs need to be monitored. For example, Kubecost has free tiers.

⬆️

Rolling update and back

You can change for example the image used by a container (named "backend" below):

kubectl set image deploy rails-dep backend=usr/rails-base-2
Enter fullscreen mode Exit fullscreen mode

then, check:

kubectl rollout status deploy rails-dep
#or
kubectl get rs -w
Enter fullscreen mode Exit fullscreen mode

and in case of crash looping, for example, you can roll back:

kubectl rollout undo deploy rails-dep
Enter fullscreen mode Exit fullscreen mode

We can also make canary deployments by putting labels.

⬆️

Roles

As a normal user, we have an admin-user Service Account and an associated Cluster Role Binding. By default, you have restricted privileges in a pod. You need to set a ServiceAccount object with the RBAC policy, with Role, RoleBinding and ServiceAccount to the Sidekiq deployment when you want to query from the container against the API with specific verbs (get, watch, list as shown below).

{
  "kind": "ServiceAccount",
  "apiVersion": "v1",
  "metadata": {
    "name": "sidekiq" # <- serviceAccountName in deploy of this resource
  }
}
Enter fullscreen mode Exit fullscreen mode 
kind: Role #<- more permissions with ClusterRole (for eg nodes)
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: query-pods #<- match rolebinding.roleref.name
  # namespace: test
rules:
  - apiGroups: [""] #<- core API group
    resources: ["pods", "service"]
    verbs: ["get", "watch", "list"]

---
kind: RoleBinding #<- ClusterRoleBnding if ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: query-pods
  # namespace: test <- none if ClusterRole
subjects:
  - kind: ServiceAccount
    name: sidekiq
    # apiGroup: rbac.authorization.k8s.io ??
roleRef:
  kind: Role
  name: query-pods #<- must match role.metadata.name
  apiGroup: rbac.authorization.k8s.io
Enter fullscreen mode Exit fullscreen mode

⬆️

3. Local Kubernetes cluster with persistent volumes

3.1 Postgres

In this example, we bring the database into the cluster so we will set up local volumes.
The standard PV available on the cluster is returned by the command kubectl get storageClass on the system. This is an important topic in real life.

Alt Text

We will make two PVC:
Alt Text

and set two StatefulSet:
Alt Text

We bind a Persistent Volume (PV) ( admin role) with a Persistent Volume Claim (PVC) (dev role).

#pg-dep.yml
kind: Service
apiVersion: v1
metadata:
  name: pg-svc
  labels:
    app: pg # <- must match with the pod
spec:
  ports:
    - protocol: TCP
      port: 5432 # <- service port opened for Rails
  selector:
    # the set of pods with the name 'pg' is targeted by this service
    app: pg

---
# Deployment
apiVersion: apps/v1
kind: StatefulSet
metadata:
  # about the deployment itself. Gives a name of the DEPLOYMENT
  name: pg-dep
  labels:
    app: pg
spec: # of the deployment
  serviceName: pg-dep
  replicas: 1
  selector:
    # the deployment must match all pods with the label "app: pg"
    matchLabels:
      # the label for the POD that the deployment is targeting
      app: pg # match spec.template.labels for the pod
  template: # blue print of a pod
    metadata:
      name: pg-pod
      # label for the POD that the deployment is deploying
      labels:
        app: pg # match spec.selector.matchlabels
    spec:
      volumes:
        - name: pg-pv # must match PV
          persistentVolumeClaim:
            claimName: pg-pvc # must match PVC
      containers:
        - name: pg-container
          image: postgres:13.3-alpine
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 5432
          volumeMounts:
            - mountPath: $(PGDATA)
              name: pg-pv # must match pv
              readOnly: false
          envFrom:
            - configMapRef:
                name: config
            - secretRef:
                name: secrets
         readinessProbe:
            exec:
              command: ["pg_isready", "-U", "postgres"]
            periodSeconds: 30
            timeoutSeconds: 10
            initialDelaySeconds: 30
         resources:
           requests:
             cpu: 100m
---
# we bind the resource PV to the pod
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pg-pvc
spec:
  #storageClassName: standard
  accessModes:
    - ReadWriteOnce #<- means only one node
  resources:
    requests:
      storage: 100Mi
Enter fullscreen mode Exit fullscreen mode

and the volume:

#pg-db-pv.yml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pg-pv
  labels:
    app: pg
spec:
  storageClassName: standard
  capacity:
    storage: 150Mi
  accessModes:
    - ReadWriteOnce #<- means only one node
  hostPath: # for Minikube, emulate net. attached vol.
    path: "/tmp/data"
Enter fullscreen mode Exit fullscreen mode

⬆️

3.2 Webserver Nginx

We use the web server image and set an external service for the Nginx deployment since this is the entry-point of the app.

#nginx-dep.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-dep
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - image: usr/nginx-ws
          imagePullPolicy: "Always"
          name: frontend
          resources:
            requests:
              cpu: 500m    
          ports:
            - containerPort: 9000
          volumeMounts:
            - mountPath: "/etc/nginx/conf.d" # mount nginx-conf volume to /etc/nginx
              readOnly: true
              name: nginx-conf
      volumes:
        - name: nginx-conf
          configMap:
            name: nginx-conf # place ConfigMap `nginx-conf` on /etc/nginx
            items:
              - key: myconfig.conf
                path: default.conf

---
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
  labels:
    app: nginx
spec:
  type: NodePort # with LoadBalancer, you pay! 
  selector:
    app: nginx
  ports:
    - protocol: TCP
      # port exposed by the container
      port: 80
      # the port the app is listening on targetPort
      targetPort: 9000
      nodePort: 31000
Enter fullscreen mode Exit fullscreen mode

This illustrates the use of a ConfigMap volume to pass the Nginx config via a mountPath into the pod source

#nginx-conf.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
data:
  myconfig.conf: |
    upstream puma {
      server rails-svc:3000;
      keepalive 1024;
    }

    access_log  /dev/stdout main;
    error_log   /dev/stdout info;

    server {
      listen 9000 default_server;
      root /usr/share/nginx/html;
      try_files  $uri @puma;
      access_log  off;
      gzip_static on; 
      expires     max;
      add_header  Cache-Control public;
      add_header  Last-Modified "";
      add_header  Etag "";

      location @puma {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header Host $http_host;
        proxy_pass_header   Set-Cookie;
        proxy_redirect off;
        proxy_pass http://puma;
      }

      location /cable {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_pass "http://cable-svc:28080";
      }

      error_page 500 502 503 504 /500.html;
      error_page 404             /404.html;
      location = /50x.html {
        root /usr/share/nginx/html;
      }
    }
Enter fullscreen mode Exit fullscreen mode

⬆️

3.3 Redis

For the Redis store, we didn't pass a config here. We should have used a ConfigMap.

#redis-dep.yml
apiVersion: v1
kind: Service
metadata:
  name: redis-svc
spec:
  ports:
    - port: 6379
      targetPort: 6379
      name: client
  selector:
    app: redis

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-dep
  labels:
    app: redis # match spec.template.labels
spec:
  serviceName: redis
  selector:
    matchLabels:
      app: redis 
  replicas: 1
  template:
    metadata:
      name: redis-pod
      labels:
        app: redis # # match spec.selector.matchLabels
    spec:
      containers:
        - name: redis
          image: redis:6.2.4-alpine
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 6379
          command: ["redis-server"]
          resources:
            requests:
              cpu: 100m
          readinessProbe:
            exec:
              command:
              - redis-cli
              - ping
            initialDelaySeconds: 20
            periodSeconds: 30
            timeoutSeconds: 3
          volumeMounts:
            - name: data
              mountPath: "/data"
              readOnly: false
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        #storageClassName: "standard"
        resources:
          requests:
            storage: 100Mi
Enter fullscreen mode Exit fullscreen mode

and the PersistentVolume:

#redis-pv.yml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: data
  labels:
    app: data
spec:
  #storageClassName: standard
  capacity:
    storage: 150Mi
  accessModes:
    - ReadWriteOnce
  hostPath: # for Minikube, to emulate net. attached vol.
    path: "/data"
Enter fullscreen mode Exit fullscreen mode

⬆️

3.4 Rails

We will use one image named "usr/rails-base" here. For production deployment, we would use the URL of a repository in a container registry (e.g. ECR).

The Rails image is build to be slim for faster loads. It is a two stage built image around 240Mb, run without root privileges. It is used by Rails, Sidekiq and Cable, with different commands. The Nginx pod will also use it to extract the static assets, and the migration Job will use it too.
❗ Do not tag it! Tilt needs to timestamp it through the tag.
All ENV vars and credentials will be set within Kubernetes. We will use two volumes, a ConfigMap for the ENV vars and a Secret for credentials:

#config.yml
#rails-dep.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: config
data:
  POSTGRES_HOST: "pg-svc" # <- name of the service that exposes the Postgres pod
  POSTGRES_DB: "kubedb"
  BUNDLE_PATH: "vendor/bundle"   #<- gems are "localized"
  RAILS_ENV: "production"
  RAILS_LOG_TO_STDOUT: "true"
  RAILS_SERVE_STATIC_FILES: "false"
  REDIS_DB: "redis://redis-svc:6379/0"
  REDIS_SIDEKIQ: "redis://redis-svc:6379/1"
  REDIS_CACHE: "redis://redis-svc:6379/2"
  REDIS_CABLE: "redis://redis-svc:6379/3"
Enter fullscreen mode Exit fullscreen mode

The credentials need to be converted with echo -n 'postgres' | base64 which gives

#secrets.yml
apiVersion: v1
kind: Secret
metadata:
  name: secrets
type: Opaque
data:
  POSTGRES_USER: cG9zdGdyZXM= # postgres
  POSTGRES_PASSWORD: ZG9ja2VycGFzc3dvcmQ= # dockerpassword
  RAILS_MASTER_KEY: NWE0YmU0MzVjNmViODdhMWE5NTA3M2Y0YTRjYWNjYTg=
Enter fullscreen mode Exit fullscreen mode

The manifest of the Rails deployment and its service:

#rails-dep.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rails-dep
  labels: # must match the service
    app: rails
spec:
  replicas: 1
  selector:
    matchLabels: # which pods are we deploying
      app: rails
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    metadata:
      labels: # must match service and replicaset matchlabel
        app: rails
    spec:
      containers:
        - name: backend
          image: usr/rails-base
          imagePullPolicy: "Always"
          command: ["bundle"]
          args: ["exec", "rails", "server", "-b", "0.0.0.0"]
          resources:
            requests:
              cpu: 500m
          ports:
            - containerPort: 3000
          envFrom:
            - configMapRef:
                name: config
            - secretRef:
                name: secrets
          readinessProbe:
            httpGet:
              path: /ready
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 30
            timeoutSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: rails-svc
  labels:
    app: rails
spec:
  selector:
    app: rails
  type: ClusterIP # default type
  ports:
    - protocol: TCP
      # port exposed by the service
      port: 3000
      # the port the app is listening on targetPort
      targetPort: 3000
      name: http
Enter fullscreen mode Exit fullscreen mode

⬆️

3.5 Cable and Sidekiq

The deployment of the "cable" is identical except for the args for the command and port. It has its service to expose the cable pod.
The "worker" deployment is also based on the same image, with its own "command" and "args" but no service is required for the worker since it communicates with Rails via a Redis queue.

Cable 
#cable-dep.yml
spec:
  containers:
    - name: cable
      image: usr/rails-base
      imagePullPolicy: "Always"
      command: ["bundle"]
      args: ["exec", "puma", "-p", "28080", "./cable/config.ru"]
      resources:
        requests:
          cpu: 150m
      ports:
        - containerPort: 28080
      envFrom:
        - configMapRef:
            name: config
        - secretRef:
            name: secrets
---
apiVersion: v1
kind: Service
metadata:
  name: cable-svc
  labels:
    app: cable
spec:
  selector:
    app: cable
  type: ClusterIP # default type
  ports:
    - protocol: TCP
      # port exposed by the service
      port: 28080
      # the port the app is listening on targetPort
      targetPort: 28080
      name: http
Enter fullscreen mode Exit fullscreen mode
Sidekiq: 
#sidekiq-dep.yml
spec:
  containers:
    - name: sidekiq
      image: usr/rails-base
      imagePullPolicy: "Always"
      command: ["bundle"]
      args: ["exec", "sidekiq", "-C", "./config/sidekiq.yml"]
      resources:
        requests:
          cpu: 150m
      envFrom:
        - configMapRef:
            name: config
        - secretRef:
            name: secrets
Enter fullscreen mode Exit fullscreen mode

We can add a liveness probe by using the gem sidekiq_alive.

⬆️

3.6 Calling the Kubernetes API

As a normal user, we have an admin-user Service Account and an associated Cluster Role Binding. Within a pod, by default everything is restricted. We have created a special role used by the Sidekiq deployment.

If we want to call the Kubernetes API to get data from the cluster within our app, we can:

  • directly access to the REST API. The credentials are present in each pod so that we can read them; within a thread or Sidekiq job, we execute a CURL call to the apiserver. The result should be parsed and filtered. 
`curl --cacert #{cacert} -H "Authorization: Bearer #{token}" https://kubernetes.default.svc/api/... `
Enter fullscreen mode Exit fullscreen mode

Note that the pod check against the server with cacert (PEM files) happens at the TLS level, not at the HTTPS level, thus this exec form is the easiest to use and harmelss since no external input.
Note that you may take profit from using the gem Oj for speeding up the JSON parsing.

Alt Text

  • do it from within a pod by tunnelling with kubectl proxy. The endpoint is on localhost, so we run a side-car pod with a Kubernetes server along with Sidekiq (since obviously the call will be run within a job). This solution is "more expensive", a Kubernetes server is a 55 Mb image running kubectl proxy --port 8001. 
URI.open(http://localhost:8001/api/v1/namespaces/{namespace}/pods)
Enter fullscreen mode Exit fullscreen mode

To render in the browser, we broadcast the result through a web socket where listener will mutate a state to render (with React).

4. Run this with TILT

We have Docker, the Kubernetes CLI and Minikube installed. A normal workflow is to build and push images from Dockerfiles. You can automate this with a github action. Then, Kubernetes will pull and run the manifests that use these images.
We can let Tilt do all this for us and be reactive to changes in the code.

Note that as a Rails user, you may have the gem "tilt" present, so alias tilt with "/usr/local/bin/tilt".

Launch Minikube

We may want to namespace the project so that you can isolate different versions or modes (staging, prod) and also clean everything easily. We used the utilities kubectx, kubens to set the namespace. After it is set, any pod (except volumes) will use the current namespace.

kubectl create namespace test
# use the utility "kubens" from "kubectx" to assign a namespace
kubens stage-v1

# check:
kubens # => "stage-v1" marked
kubectl config get-contexts # => Minikube marked
#or
kubectl config current-context
Enter fullscreen mode Exit fullscreen mode

A good practice is to document the namespace: apply the following manifest and use namespace: prod in the metadata of each kind.

kind: Namespace
apiVersion: v1
metadata:
  name: stage-v1
Enter fullscreen mode Exit fullscreen mode

namespace

Now the project is isolated within a namespace, we can launch Minikube with minikube start.

⬆️

List of files

Our files are:

/app
/config
/public
...
/dckf
   |_  _builder.Dockerfile
   |_  _alpine.Dockerfile
   |_  _nginx.Dockefile
Tiltfile

/kube
  |_ config.yml
  |_ secrets.yml
  |_ nginx-config.yml

  |_ pg-pv.yml
  |_ postgres-dep.yml
  |_ redis-pv.yml
  |_ redis-dep.yml

  |_ rails-dep.yml
  |_ sidekiq-dep.yml
  |_ cable-dep.yml
  |_ nginx-dep.yml

  |_ migrate.yml
Enter fullscreen mode Exit fullscreen mode

Deploy

To deploy on the Minikube cluster, once we have built and pulled the images, we run the following commands against all our manifests ❗ in an orderly manner: first run the configs (and secrets) needed by the others processes.

kubectl apply -f ./kube/config.yml
kubectl apply -f ./kube/secrets.yml
...
kubectl apply -f ./kube/rails-dep.yml
...
kubectl apply -f ./kube/migrate.yml

minikube service nginx-svc #<- our app entry point
Enter fullscreen mode Exit fullscreen mode

We will automate all this with Tilt. The Tilt engine will read a Tiltfile and build the full project. It only needs the code (the Dockerfiles and the Kubernetes manifests) and one command.

In this Tiltfile, we describe all the actions we want Tilt and Kubernetes to perform: building images, running manifests and managing dependencies. The one below is a very basic image builder and runner.

Note that you should not tag your images as Tilt will put a timestamp, and the named used in docker_build should match the name of the image in the deployment manifest.

#Tilfile
# apply configs and create volumes
k8s_yaml(['./kube/config.yml','./kube/secrets.yml','./kube/nginx-config.yml','./kube/pg-db-pv.yml',
])
# apply databases adapters
k8s_yaml(['./kube/postgres-dep.yml','./kube/redis-dep.yml'])

# <- building images and live changes
docker_build( 'builder', # <- Bob
  '.', 
  dockerfile="./dckf/builder.Dockerfile",
  build_args={
     "RUBY_VERSION": "3.0.2-alpine",
     "NODE_ENV": "production",
     "RAILS_ENV": "production",
     "BUNDLER_VERSION": "2.2.25"
  }
)

docker_build('usr/rails-base', #<- uses Bob
   '.',
   build_args={
     "RUBY_VERSION": "3.0.2-alpine",      
     "RAILS_ENV": "production",   
     "RAILS_LOG_TO_STDOUT": "true",
   },
   dockerfile='./docker/rails.Dockerfile'
)

docker_build("usr/nginx-ws", # <- uses Bob
   ".",
   dockerfile="./dckf/nginx-split.Dockerfile",
   build_args={
      "RUBY_VERSION": "3.0.2-alpine",  "RAILS_ENV": "production"
   }
)
# -> end images

# dependencies
k8s_resource('sidekiq-dep', resource_deps=['redis-dep'])
k8s_resource('rails-dep', resource_deps=['pg-dep', 'redis-dep'])
k8s_resource('cable-dep', resource_deps=['redis-dep'])
k8s_resource('nginx-dep', resource_deps=['rails-dep'])

# apply processes
k8s_yaml(['./kube/rails-dep.yml', './kube/sidekiq-dep.yml', './kube/cable-dep.yml', k8s_yaml('./kube/nginx-dep.yml'])

# <- creates manual/auto action button in the Tilt GUI
# migration
k8s_resource('db-migrate', 
   resource_deps=['rails-dep','pg-dep'],
   trigger_mode=TRIGGER_MODE_MANUAL,
   auto_init=False
)
k8s_yaml('./kube-split/migrate.yml' )

# auto (&manual) display pods in GUI
local_resource('All pods',
   'kubectl get pods',
   resource_deps=['rails-dep','sidekiq-dep','cable-dep','nginx-dep']
)
# -> 

allow_k8s_contexts('minikube')
k8s_resource('nginx-dep', port_forwards='31000')
Enter fullscreen mode Exit fullscreen mode

Notice the two stage build: the image is only build once even if reused four times. You may skip the build phase but you need to push the images to the container registry.

Notice the resources dependencies with k8s_resource which makes things easy!

To run this:

tilt up
Enter fullscreen mode Exit fullscreen mode

or all in one (unless you applied the namespace manifest or needed to create the Postgres database): run the Docker daemon (open -a Docker on OSX)

minikube start && kubectl create ns test && kubens test && tilt up
Enter fullscreen mode Exit fullscreen mode

Then we need to migrate (see below), and with our settings, we can navigate to http://localhost:31000 to visualise the app.

For a quick clean up, run (as docker-compose down):

tilt down
#or
kubectl delete ns test
Enter fullscreen mode Exit fullscreen mode

Live update

We can use the function live_update that syncs the code between the host and the container.
Instead of a slim image, you need all the Rails tooling for hot-reloading and work RAILS_ENV=development so use this dockerfile. You target /app directory which contains the "active" code since any modification in any other folder needs a complete rebuild; for this, you can use live_update=[sync('app','/app/app')]. Then any change in the backend code will be immediately available. Any other change in the code will trigger a rebuild.
If you want to exclude some file, add ignore=['README.md', 'log','tmp'] for example.
You remove the "Cable" process" and "Nginx" and keep only Rails (and Sidekiq since it communicates via the pub/sub Redis).
❗ Front-end code change needs page refresh.

The new Tilfile looks like (the files specific to this mode are put in a seperate folder "/kube-dev" and the common are unchanged).

#dev-tilfile
k8s_yaml(['./kube-dev/config.yml','./kube/secrets.yml', './kube/pg-db-pv.yml', './kube/redis-pv.yml', './kube/role.yml', './kube/service-account.yml'
])
k8s_yaml(['./kube/postgres-dep.yml', './kube/redis-dep.yml'])

docker_build( 'builder-dev', '.', 
  dockerfile="./docker/dev.Dockerfile", 
  build_args={
     "RUBY_VERSION": "3.0.2-alpine", "NODE_ENV": "development", "RAILS_ENV": "development", "BUNDLER_VERSION": "2.2.25",
   },
   live_update=[sync('app','/app/app/')],
   ignore=['tmp','log']
)

k8s_yaml(['./kube-dev/rails-dep.yml','./kube-dev/sidekiq-dep.yml'])

k8s_resource('rails-dep', resource_deps=['pg-dep', 'redis-dep', 'db-migrate'])
k8s_resource('sidekiq-dep', resource_deps=['redis-dep','pg-dep', 'db-migrate'])

k8s_resource('db-migrate', 
   resource_deps=['pg-dep'],
   trigger_mode=TRIGGER_MODE_MANUAL,
   auto_init=False
)
k8s_yaml('./kube-dev/migrate.yml' )

allow_k8s_contexts('minikube')
k8s_resource('rails-dep', port_forwards='31000') #<- "dev"
Enter fullscreen mode Exit fullscreen mode

Run tilt -f dev.tiltfile up.

⬆️

Annotations

You may use annotations to prevent for example the lengthy rebuild of Postgres after you tilt down. Add an annotation to keep them running:

metadata.annotations.tilt.dev/down-policy: keep
Enter fullscreen mode Exit fullscreen mode 
#postgres-dep.yml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  # about the deployment itself. Gives a name of the DEPLOYMENT
  name: pg-dep
  labels:
    app: pg
  annotations:
    tilt.dev/down-policy: keep
Enter fullscreen mode Exit fullscreen mode

Migration Job

The topic of performing safe migrations is a very important subject of its own (new migration on old code, old migration with new code).

Every time you stop Minikube and Tilt, you need to run a migration. You can an initContainer that waits for the db, or a Job.

#migrate.yml
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: db-migrate
          image: usr/rails-base
          imagePullPolicy: IfNotPresent
          command: ["bundle", "exec", "rails", "db:migrate"]
          envFrom:
            - secretRef:
                name: secrets
            - configMapRef:
                name: config
Enter fullscreen mode Exit fullscreen mode

You can run k8s_yaml('./kube/migrate.yml') to apply this.
Alternatively, you can do this programatically with Tilt with k8s_resource where you specify the needed dependencies, and then apply with k8s_yaml.
Below is an example of the manual custom db-migrate action. With the two last flags, we have an action button in the GUI to trigger this migration when ready.

k8s_resource('db-migrate', 
   resource_deps=['rails-dep','pg-dep'],
   trigger_mode=TRIGGER_MODE_MANUAL,
   auto_init=False
)
k8s_yaml('./kube-split/migrate.yml' )
Enter fullscreen mode Exit fullscreen mode

Alt Text

[K8s EVENT: Pod db-migrate-vh459 (ns: test)] Successfully pulled image "/rails-base" in 1.481341763s
Migrating to CreateCounters (20210613160256)
== 20210613160256 CreateCounters: migrating ===================================
-- create_table(:counters)
   -> 0.4681s
== 20210613160256 CreateCounters: migrated (0.4694s) ==========================
Enter fullscreen mode Exit fullscreen mode

If needed, run kubectl delete job db-migrate to be able to run the job.

Using local_resource

If we want to list automatically the Rails pods, we add a "local_resource" in the Tiltfile. As such, it will be run automatically when Rails is ready. This will also add a button in the UI that we can rerun on demand.

local_resource('Rails pods',
   "kubectl get pods -l app=rails -o go-template --template '{{range .items}}{{.metadata.name}}{{\"\n\"}}{{end}}' -o=name ",
   resource_deps=['rails-dep']
)
Enter fullscreen mode Exit fullscreen mode

⬆️

Tilt UI

The Tilt UI comes in two flavours:

  • a terminal, essentially an easy log collector Alt Text
  • a nice GUI at "http://localhost:10350" from which you can for example easily read the logs or trigger rebuilds. Alt Text

⬆️

Kubernetes dashboard

To enable the Kubernetes dashboard, run:

minikube addons list
minikube addons enable dashboard
minikube addons enable metrics-server
Enter fullscreen mode Exit fullscreen mode

The command minikube dashboard shows the full state of the cluster. Just select the namespace. We can see the logs for each pod, the resource consumption, globally and per process.

Alt Text

Alt Text

Since you have admin credentials, you can deploy your app just using the Dashboard. Your images should already be built.

Resources with kubectl

The command:

kubectl describe node minikube
Enter fullscreen mode Exit fullscreen mode

gives the details of the resource usage of the cluster.

Alt Text

HPA

Once we have the measures, we can start to autopilot selected processes with HorizontalPodAutoscaler.

We can run the command kubectl autoscale

local_resource('hpa', 
   'kubectl autoscale deployment rails-dep --cpu-percent=80 --min=1 --max=3',
   resource_deps=['rails-dep'],
   trigger_mode=TRIGGER_MODE_MANUAL,
   auto_init=False
)
Enter fullscreen mode Exit fullscreen mode

or apply the manifest:

#rails-hpa.yml
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: rails-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: rails-dep
  minReplicas: 1
  maxReplicas: 3
  #targetCPUUtilizationPercentage: 80
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
Enter fullscreen mode Exit fullscreen mode

with

k8s_yaml('./kube/rails-hpa.yml')
k8s_resource('rails-hap', resource_deps=['rails-dep'])
Enter fullscreen mode Exit fullscreen mode

We can use simple load tests to check.

⬆️

5. Misc files

Cheat sheet 
# waiting pods checking
kubectl get pods -w -o wide

# follow the logs of a pod
kubectl logs rails-nginx-dep-1231313 -c rails -f

# delete a deployment (pod+svc+replicaset) or job
kubectl delete deploy rails-dep
kubectl delete job db-migrate

# launch a browser
minikube service rails-nginx-svc   # !! OSX !!

# launch the Kubernetes dashboard
minikube dashboard

# ssh into the VM (user docker, private key)
minikube ssh

# generate a token which grants admin permissions:
kubectl --namespace kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')

# ssh
ssh -i ~/.minikube/machines/minikube/id_rsa docker@$(minikube ip)

# get programmatically info
kubectl get service nginx-svc -o jsonpath='{.spec.ports[0].nodePort}'

# execute some commands:
export POD_NAME=$(kubectl get pods -l app=rails -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}') &&
#   kubectl exec $POD_NAME -- bundle exec rails db:migrate' ,resource_deps=['rails-dep','sidekiq-dep','cable-dep',','nginx-dep'])

# execute a command in a pod with several containers (name it with "-c")
kubectl exec it -rails-nginx-dep-1234 -c rails -- irb
  irb(main):001:0> require 'socket'; Socket.gethostname
  => "rails-nginx-dep-55bc5f77dc-48wg4"

# execute a command in a pod 
kubectl exec -it pg-dep-0 -- psql -w -U postgres
kubectl exec -it redis-dep-0  -- redis-cli

# rolling update by setting a new image to a deployment
kubectl set image deployment/rails-dep usr/rails-base2

# check
kubectl rollout status deployment/rails-dep

# undo in case of a problem
kubectl rollout undo depoyment/rails-dep
Enter fullscreen mode Exit fullscreen mode

⬆️

yaml <=> json

You can convert a bunch of files from yaml into json (or vice-versa) from your current directory with yq:

for file in *.yml; do
  base=${file%.yml};
  # yml > json
  yq eval -o=j $base.yml > $base.json;
  # or json > yml
  yq eval -P $base.json > $base.yml;
done
Enter fullscreen mode Exit fullscreen mode

Dev.Dockerfile 

ARG RUBY_VERSION
FROM ruby:${RUBY_VERSION:-3.0.2-alpine}
ARG BUNDLER_VERSION
ARG NODE_ENV
ARG RAILS_ENV

ENV BUNDLER_VERSION=${BUNDLER_VERSION:-2.2.26} \
   RAILS_ENV=${RAILS_ENV:-development} \
   NODE_ENV=${NODE_ENV:-development}

RUN apk -U upgrade && apk add --no-cache postgresql-dev nodejs yarn build-base tzdata curl \ 
   && rm -rf /var/cache/apk/*

WORKDIR /app
COPY Gemfile Gemfile.lock package.json yarn.lock ./

ENV LANG=C.UTF-8 BUNDLE_JOBS=4 BUNDLE_RETRY=3 

RUN gem install bundler:${BUNDLER_VERSION} --no-document \
   && bundle install --quiet \
   && rm -rf $GEM_HOME/cache/* \
   && yarn install --check-files --silent  && yarn cache clean

COPY . ./
Enter fullscreen mode Exit fullscreen mode

⬆️

Builder 

ARG RUBY_VERSION=3.0.2-alpine
FROM ruby:${RUBY_VERSION:-3.0.1-alpine}
ARG BUNDLER_VERSION
ARG NODE_ENV
ARG RAILS_ENV
RUN apk -U upgrade && apk add --no-cache \
   postgresql-dev nodejs yarn build-base tzdata
ENV PATH /app/bin:$PATH
WORKDIR /app
COPY Gemfile Gemfile.lock package.json yarn.lock ./
ENV LANG=C.UTF-8 \
   BUNDLE_JOBS=4 \
   BUNDLE_RETRY=3 \
   BUNDLE_PATH='vendor/bundle' 
RUN gem install bundler:${BUNDLER_VERSION} --no-document \
   && bundle config set --without 'development test' \
   && bundle install --quiet \
   && && rm -rf $GEM_HOME/cache/* \
   && yarn --check-files --silent --production && yarn cache clean
COPY . ./
RUN bundle exec rails webpacker:compile assets:clean
Enter fullscreen mode Exit fullscreen mode

Rails Dockerfile 

FROM builder AS bob
FROM ruby:${RUBY_VERSION:-3.0.2-alpine}

ARG RAILS_ENV
ARG NODE_ENV
ARG RAILS_LOG_TO_STDOUT

RUN apk -U upgrade && apk add --no-cache  libpq tzdata netcat-openbsd curl \
   && rm -rf /var/cache/apk/* \
   && adduser --disabled-password app-user
# -disabled-password doesn't assign a password, so cannot login 
USER app-user
COPY --from=bob --chown=app-user /app /app
ENV RAILS_ENV=$RAILS_ENV \
   RAILS_LOG_TO_STDOUT=$RAILS_LOG_TO_STDOUT \
   BUNDLE_PATH='vendor/bundle'

WORKDIR /app
RUN rm -rf node_modules
Enter fullscreen mode Exit fullscreen mode

Web-server Dockerfile 

#_nginx-ws.Dockerfile
FROM usr/rails-base AS bob
FROM nginx:1.21.1-alpine
COPY --from=bob  ./app/public /usr/share/nginx/html
Enter fullscreen mode Exit fullscreen mode

⬆️

Simple load test

You can run a simple load test:

kubectl run -i --tty load-generator --rm --image=busybox --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://localhost:31000; done"
Enter fullscreen mode Exit fullscreen mode

If we have a domain, we can also use the "ab" Apache Bench: we test here how fast the app can handle 1000 requests, with a maximum of 50 requests running concurrently:

ab -n 1000 -c 50 https://my-domain.top/
Enter fullscreen mode Exit fullscreen mode

⬆️

[sources]:

Happy coding!

Top comments (2)

Collapse
 
zedtux profile image
ZedTuX •

Nice article !

One question: How do you manage to run commands which update app files? For example rails g migration ..., yarn add ... and so on.

Collapse
 
jhawithu profile image
Alok Kumar •

Very nice article, lots of learning, keep posting. I also created a full course on Kubernetes in 10hr on YouTube, please have a look and motivate me. Thanks
youtu.be/toLAU_QPF6o

Read next

guiqui profile image

Pod Security with K8Studio

Guillermo Quiros -

eduardinni profile image

Kamal: How to use Grafana Loki as a logging driver

Eduardo Lomeli -

depp57 profile image

Kubernetes homelab - Learning by doing, Part 6: Automation

Sacha Thommet -

depp57 profile image

Kubernetes homelab - Learning by doing, Part 5: Monitoring

Sacha Thommet -