Clean up resources with kube-janitor

About

Clean up (delete) Kubernetes resources after a configured TTL (time to live)

Implementation

The application is deployed as a deployment with escalated privileges. It listens to API requests to the API server, has an internal schedule queue (probably) and then deletes resources upon a rule match.

Reference repo

https://codeberg.org/hjacobs/kube-janitor.git

Installation

# pick files from https://codeberg.org/hjacobs/kube-janitor/src/branch/main/deploy/
# update rules.yaml as necessary
> cat rules.yaml
rules:
# remove deployments and statefulsets with a "demo" label set after 3 days
- id: cleanup-demo-objects
  resources:
    - deployments
    - statefulsets
  jmespath: "(spec.template.metadata.labels.demo)"
  ttl: 3d
# remove all deployments and jobs named "pr-*" after 6 hours
- id: cleanup-pr-deployments
  resources:
    - deployments
    - jobs
  jmespath: "starts_with(metadata.name, 'pr-')"
  ttl: 6h
# delete all resources within the "temp-*" namespace after 3 days
- id: cleanup-temp-namespaces
  resources:
    - namespaces
  jmespath: "starts_with(metadata.name, 'temp-')"
  ttl: 3d
# delete all PVCs which are not mounted and not referenced by StatefulSets after 4 days
- id: remove-unused-pvcs
  resources:
    - persistentvolumeclaims
  jmespath: "_context.pvc_is_not_mounted && _context.pvc_is_not_referenced"
  ttl: 4d

> kubectl apply -k .

Configuration

There are 3 ways of using this tool

1. Annotate the object with a janitor/ttl annotation. Useful for CI/CD scenario
   > kubectl annotate deploy test-app-dep janitor/ttl=24h

2. Annotate the object with a janitor/expires annotation. Useful for dangling jobs/cronjobs
   > kubectl annotate deploy nginx janitor/expires=2022-03-31

3. Update the rules file (edit it and (re)deploy or > kubectl edit configmap kube-janitor). This is the server side automation applicable for policy enforcement

4. refer rules.yaml for example

5. use jmespath for writing rules (refer this)

Note:

• namespace level cleanup is not working and there is a PR to add this enhancement

More info

• Sample usage in tekton pipeline scenario

Fascinating to see a python application working almost as a kubernetes CRD without all the complexities.