Swagger-Operator, let groovy operate your cluster

WARNING

This post is still under review

In this post I’ll (try to) explain how I’ve created a kubernetes operator using Groovy and Micronaut, because …​ yes, you don’t need to use Go for it!!

Situation

Say we are working in a microservice architecture with several services (nodejs, spring, micronaut, quarkus, …​) and some (or all) of them are using OpenAPI to expose their API and they are deployed in a kubernetes cluster.

INFO

Basically a OpenApi spec is a resource (typically a json or yaml) the service serve via an http GET where all endpoints, payloads, documentation, etc are structured following a well-know structure

From time to time a new service is deployed, or deprecated and removed from the cluster.

Typical situation is every service include an html interface to render its spec in a human friendly way and it is very common to use swagger-ui for it.

INFO

Swagger-ui is basically a Javascript application able to understand an OpenApi spec and generate a playground on the fly

Another posible solution is to use a single swagger-ui instance and configure a list of OpenApi spec (for example configuring the SERVERS_URL environment) so using a single javascript application we can play with different services

Usually QA and/or Frontend uses this interface to check their implementation and also to execute some requests to the backend microservice (yes, yes, I know, is not the best practique, but …​ ) so it’s important to have this swagger-ui updated with the right list of specs.

Manual solution

Our current solution consists in 2 artifacts:

• a pod running standard swagger-ui docker image

• a configmap with a JavaScript file similar to the oficial but with the lists of servers

configmap

window.ui = SwaggerUIBundle({
    urls: [
        {
            name:"user-rest",
            url:"/user-rest/swagger/service-example-0.0.yml"
        }
    ...
})

When we deploy the swagger-ui overwriting the original javascript with this configmap we have a Swagger playground where our QA team can use to send requests to every service.

When a new microservice is required by the architecture and is deployed in the cluster is so simple as edit the configmap, add the new entry in the list, and delete the current pod. As soon the cluster detect this pod was deleted a new one will be created using new configmap, so QA only need to refresh the browser to see the new service in the list. (Same if what we want is to remove a microservice from the list)

As you can imagine, although is a simple process is very error-prone, and most of the time we react when the QA report can’t find the new service in the swagger-ui application.

What’s and operator?

Basically an operator is a "typical" application deployed into the cluster who will be "talking" with the cluster, no with the user.

The cluster will be asking to our operator to check if all looks good every few seconds and our operator need to "reconcile" current status with the desired state.

Imagine we want to have running 2 instances of a deployment and suddenly one of them reach and exception and finish.

A "few seconds" later the cluster will ask the operator to check if all looks good in the deployment so the operator will retrieve the list of running instances, will compare with the desired state, and it will decide to create a new pod.

The logic of our SwaggerOperator to decide if all looks good is similar.

Swagger-Operator

So basically what we’ll create is a kubernetes operator to perform these actions:

• create a deployment (running the swagger-ui docker image) and a service in case they not exist

• maintain a configmap updated with the list of current services present into the cluster.

• delete the current pod in case an update in the configmap is done

As an operator is in fact a typical application we’ll create our swagger-operator using the micronaut command line to create it:

mn create-app --lang groovy --features kubernetes-informer swagger-operator

CRD, Custom Resource Definition

First thing is to define a new Kind resource (and deploy it into the cluster). The CRD is where we’ll instruct to the cluster how the new resource will look, so yes, it’s a kind of meta-resource.

In a CRD we need to specify the name of the kind, the group we want to include it and the properties the user need to fill to deploy a new swagger-operator resource

Swagger-operator CRD looks like:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: swaggers.puravida.com
spec:
  group: puravida.com
  scope: Namespaced
  names:
    plural: swaggers
    singular: swagger
    kind: Swagger
  versions:
    - name: v1
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                serviceSelector:
                  type: string
                configMap:
                  type: string
                deployment:
                  type: string
                service:
                  type: string
...

Although a long file basically what we’re doing is instructing to the cluster about a new kind of resource (Swagger). When the user will want to create a new resource of this kind he will need to provide 4 properties calledserviceSelector, configMap …​

INFO

In our case these 4 properties will be used by the operator to create configurable "named" resources instead to use hardcoded values

To let the cluster manage this new resource we need to apply it into the cluster:

$ kubectl apply -f crd.yml

CRD to Java

As we want to use Groovy/Java in our operator, we need to convert this CRD to Java objects. The easy way is to use a command line from kubernetes-client project similar to:

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v "$(pwd)":"$(pwd)" \
      -ti --network host ghcr.io/kubernetes-client/java/crd-model-gen:v1.0.6 \
      /generate.sh -u $(pwd)/src/k8s/crd.yml -n com.puravida -p com.puravida \
      -o $(pwd)

Mounting our local project as a volume the generate.sh process can read the crd and generate some Java files. They are basically a kind of POJO Java representations of the CRD.

Operator

Swagger-operator is a simple operator and requires only one class, SwaggerOperator.groovy

@Operator(
    informer = @Informer(
        apiType = V1Swagger,
        apiListType = V1SwaggerList,
        apiGroup = V1SwaggerWrapper.GROUP,
        resourcePlural = V1SwaggerWrapper.PLURAL,
        resyncCheckPeriod = 10000L
    )
)
class SwaggerOperator implements ResourceReconciler<V1Swagger>{
    // The implementation
}

As you can see basically we need to annotate our class with @Operator and implement ResourceReconciler<V1Swagger>interface

This interface requires we implement only one method:

@Override
Result reconcile(@NonNull Request request, @NonNull OperatorResourceLister<V1Swagger> lister) {
    //
    return new Result(false) (1)
}

| 1 | Returning false we inform to the cluster we don’t need a new reconcile "right now" |

reconcile is the method will be called every X millis (10s in our case) by the cluster once a V1Swagger resource is deployed by the user. Our operator must check if all resources are aligned with the desired state.

To do it the operator needs/can "talk" with different APIs exposed by the cluster as CoreApi or AppApi, so it can list all services present in a namespace, create a configmap, etc

Basically the main logic of the swagger-operator reconcile method is:

• if a Swagger resource is present the operator need to check if the configmap, the deployment and the service exist

• also, if the Swagger resource exist it must to check if the configmap is up to date checking the list of services and if they is any different it needs to update the configmap and delete the current pod

• if the resource is marked to be deleted the operator needs to "clean" the resources created deleting the configmap, service and deployment

Checking the list of services

The operator requests to the cluster a list of current services in the namespace and select all that contains the desired annotation:

def services = coreApi.listNamespacedService(wrapper.namespace)
    .items
    .findAll({ service->
        service.metadata.annotations?.containsKey(wrapper.serviceSelector)
    })
def map = services.inject([:],{ map, it ->
    map[it.metadata.name] = it.metadata.annotations[wrapper.serviceSelector]
    map
}) as Map<String, String>

services is the current list of services we want to show in the list of swagger-ui and map is a Map to be "injected" in the ConfigMap.

Checking if ConfigMap is up to date

def configMap = coreApi
        .readNamespacedConfigMap(wrapper.configMap,
                wrapper.namespace,null,null,null)

def currentJS = configMap.data[CONFIG_YML]

if( currentJS.contains("urls: [$urlServices]") ){
        return false
}

The swagger-operator read current ConfigMap data entry CONFIG_YML and check if the current value contains a string similar to the current list of services. If it is the same no action is required

If they are not equals this means some service is not in the list or a new service was deployed so the swagger-operator create a new updated ConfigMap

currentJS = currentJS.replaceFirst(/urls: \[(.*?)]/,"urls: [$urlServices]")
configMap.data[CONFIG_YML] = currentJS

coreApi.replaceNamespacedConfigMap(wrapper.configMap,
        wrapper.namespace,
        configMap)

and mark current deployment to be restarted

def deployment = deploymentList.items.first()
deployment.spec
        .template
        .metadata
        .annotations[V1SwaggerWrapper.RESTARTED_AT_ANNOTATION]=Instant.now().toString()

appsApi.replaceNamespacedDeployment(deployment.metadata.name,
        deployment.metadata.namespace,
        deployment)

INFO

You can check out the repository (link at the end of the post) to see full code

Roles and Service account

Probably your operator will require to be run with an special service account how allow them to access to cluster resources. For swagger-operator this is specify in this resource

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: swagger-operator-role
rules:
  - apiGroups: ["", "apps"]
    resources: ["services", "configmaps", "deployments", "pods"]
    verbs: ["get", "watch", "list", "create", "delete", "update"]
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["get", "create", "update"]
  - apiGroups: ["puravida.com"]
    resources: ["swaggers"]
    verbs: ["*"]

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: swagger-operator-sa

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: swagger-operator-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: swagger-operator-role
subjects:
  - kind: ServiceAccount
    name: swagger-operator-sa
    namespace: default

As you can see we are creating a new service account swagger-operator-sa and allowing to it different access to different resources (full control for swaggers.puravida.com resources for example)

Deploy

As a final step we need to build and deploy our application to a docker registry (in swagger-opeator case using the gradle task jib ) and deploy it into the cluster using a typical deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: swagger-deployment-operator
  name: swagger-deployment-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      app: swagger-deployment-operator
  template:
    metadata:
      labels:
        app: swagger-deployment-operator
    spec:
      serviceAccountName: swagger-operator-sa
      containers:
        - image: registry.localhost:5000/swagger-operator
          name: swagger-deployment-operator
          imagePullPolicy: Always

As you can see we specify the serviceAccountName swagger-operator-sa created previously.

WARNING

For this example I’m using a local docker registry. In a near future I hope to have time and deploy in docker hub

Last step

So now our cluster is ready and waiting an user (admin, QA, …​) create a new Swagger resource specifying wich services to include and wich deployment create

apiVersion: puravida.com/v1
kind: Swagger
metadata:
  name: swagger-operator
spec:
  serviceSelector: swagger-path
  configMap: swagger-config
  deployment: swagger
  service: swagger

As soon the user apply this file into the cluster, swagger-operator will start receiving events from the cluster to reconcile it.

The operator will list all current services and select some of them, check the ConfigMap is not present and it will create a new one using a classpath resource as template, check the Deployment is not present and create one, etc

After a few seconds we’ll have a new pod running into our cluster with the swagger-ui interface and a list of services configured

If we remove some service (kubectl delete -f user-rest for example) the operator will recreate all resources and the swagger-ui will be updated automatically

Conclusion

In this article we are creating a Micronaut Kubernetes Operator using Groovy able to create and maintain resources into the cluster

I've created a repo with the code and a service to be used as example at https://github.com/jagedn/swagger-operator