Disable Effects of a Controller On its Pods in Kubernetes

A controller is the name for several different elements in Kubernetes. One of these elements designates the global name of any resource responsible, among other things, for the creation and lifecycle management of the pods.

A typical controller is a Deployment, which is used to deploy pods in several replicas but is also responsible for destroying or recreating them as needed. The other three most used types of controllers are StatefulSet, DaemonSet, and Job. When working with controllers, we may want to prevent them from making changes to pods temporarily. So, how do we do it?

The cleanest solution (STS only)

In the case of an StatefulSet (STS), we have a significant difference compared to a dDeployment, it is the presence of Rolling Update Strategies, at the root of spec, of which here is the entire API with the values ​​by default:

spec:
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate

What we are interested in is spec.updateStrategy.rollingUpdate.partition. This option is only available if spec.updateStrategy.type is set to RollingUpdate. It allows you to specify, in a StatefulSet, the pod ordinal above which modifications made to the controller will be propagated. In other words, with the default value of 0, all pods will be updated when the StatefulSet changes. If set to 1, then all pods will be updated except the first pod, whose name ends with -0.

This allows for canary deployment at a lower cost, such as modifying the image of only 30% of the pods to gradually introduce an update. The command to patch a StatefulSet is as follows:

sts=postgres
part=2
kubectl patch statefulset $sts -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":'$part'}}}}'

In the case of a canary deployment, we could first modify the image of a container in the pod template, then deploy the modifications gradually by regularly patching the controller and lowering the partition value step by step.

Note that this will not prevent a pod from restarting if it is deleted; it only affects modifications to the pod template. For handling pod restarts, continue reading.

Without a controller, there won't be any deletion or recreation of pods against your will. However, if you delete a controller, the pods created by it will disappear. To prevent this, you can use an option with the kubectl delete command that prevents a controller from eliminating its child pods. This option is --cascade=false (since it is true by default).

kubectl delete sts postgresql --cascade=false

When deleting a controller this way, Kubernetes still removes the ownerReferences objects from all child pods, but does nothing else. You can then, for example, delete a pod and mount its data volume in a new pod for a special operation (such as an advanced backup). Using the backup example, this method can also be used to turn off the pods of a StatefulSet one by one and take the time to perform a cold backup of the data on the storage side.

However, you should remember to recreate the deleted controller if you want to keep the pods alive after the operation. Here is a bash script that provides the basics of using this method properly:

This script ensures that the controller is recreated as is once the operation is completed or if the script encounters a problem, or if the user kills it prematurely (except in the case of a kill -9 of course).