How to: Delete a stubborn Kubernetes namespaces.

First posted on my blog.

The Situation

Recently I was introduced to a new project. The getting started documentation was decent and I was able to get the project started up. When it came time to clear the project out of my local machines Kubernetes cluster however, the same space for to the terminating life cycle phase and stayed there for days. Fast forward a couple of weeks and I notice it there, still terminating. Turns out the project had a finalizer that was not responding back. Think of remote service call. In this case the remote service never responded. So below is a little script I put together to force a namespace deletion.

The Code

#!/bin/bash
k8s_delete_ns=$1
echo "Provided namesapce: ${k8s_delete_ns}..."
echo "Exporting namespace configuration..."
kubectl get namespaces -o json | grep "${k8s_delete_ns}"
kubectl get namespace ${k8s_delete_ns} -o json > temp.json
echo "Opening editor..."
wait 3
vi temp.json
echo "Sending configuration to k8s master for processing..."
curl -H "Content-Type: application/json" -X PUT --data-binary @temp.json http://127.0.0.1:8080/api/v1/namespaces/${k8s_delete_ns}/finalize
echo "Waiting for namespace deletion to process..."
wait 12
kubectl get namespaces
echo "...done."

So what does this do? Lets remove the shell, echo and wait statements since we know the machine does not really do anything with those.

k8s_delete_ns=$1
kubectl get namespaces -o json | grep "${k8s_delete_ns}"
kubectl get namespace ${k8s_delete_ns} -o json > temp.json
vi temp.json
curl -H "Content-Type: application/json" -X PUT --data-binary @temp.json http://127.0.0.1:8080/api/v1/namespaces/${k8s_delete_ns}/finalize
kubectl get namespaces

So what does this do exactly?

Ok, so what do we have here. Line one takes the first argument from the command invocation and assigns it to a local variable. ./script.sh stuck_namespace. So stuck_namespace is the value that is assigned to k8s_delete.

The next two commands get's the current namespace configuration and writes it to temp.json.

The fourth command opens the temp.json configuration file using vi.

With the configuration open we want to remove any items listed in the finalizers array. This is the real magic moment here. Removing the finalizers with allow kubectl to remove the namespace.

Speaking of the next command uses curl to pass the newly edited configuration to kubectl. This updates the namespace configuration with the config that does not contain finalizers.

And finally, get namespaces will output the existing namespace after the update. If all went as expected the list will not contain the stuck namespace.

Conclusion

So there it is. There are a number of solutions out floating around the internet but the release cadence of Kubernetes renders some of then out-dated in a matter of months. So let me state this: this solution works for Kubernetes 1.13. Other than that, I hope you it helps you out as it helped me.

P.S.: If you would like to keep up with Kubernetes helpers I make, checkout the repo on GitHub.