- WARNING
-
This post is still under review
In this post I’ll (try to) explain how I’ve created a kubernetes operator using Groovy and Micronaut, because … yes, you don’t need to use Go for it!!
Situation
Say we are working in a microservice architecture with several services (nodejs, spring, micronaut, quarkus, …) and some (or all) of them are using OpenAPI to expose their API and they are deployed in a kubernetes cluster.
- INFO
-
Basically a OpenApi spec is a resource (typically a json or yaml) the service serve via an http GET where all endpoints, payloads, documentation, etc are structured following a well-know structure
From time to time a new service is deployed, or deprecated and removed from the cluster.
Typical situation is every service include an html interface to render its spec in a human friendly way and it is very common to use swagger-ui for it.
- INFO
-
Swagger-ui is basically a Javascript application able to understand an OpenApi spec and generate a playground on the fly
Another posible solution is to use a single swagger-ui instance and configure a list of OpenApi spec (for example configuring the SERVERS_URL environment) so using a single javascript application we can play with different services
Usually QA and/or Frontend uses this interface to check their implementation and also to execute some requests to the backend microservice (yes, yes, I know, is not the best practique, but … ) so it’s important to have this swagger-ui updated with the right list of specs.
Manual solution
Our current solution consists in 2 artifacts:
a pod running standard swagger-ui docker image
a configmap with a JavaScript file similar to the oficial but with the lists of servers
configmap
window.ui = SwaggerUIBundle({
urls: [
{
name:"user-rest",
url:"/user-rest/swagger/service-example-0.0.yml"
}
...
})
When we deploy the swagger-ui overwriting the original javascript with this configmap we have a Swagger playground where our QA team can use to send requests to every service.
When a new microservice is required by the architecture and is deployed in the cluster is so simple as edit the configmap, add the new entry in the list, and delete the current pod. As soon the cluster detect this pod was deleted a new one will be created using new configmap, so QA only need to refresh the browser to see the new service in the list. (Same if what we want is to remove a microservice from the list)
As you can imagine, although is a simple process is very error-prone, and most of the time we react when the QA report can’t find the new service in the swagger-ui application.
What’s and operator?
Basically an operator is a "typical" application deployed into the cluster who will be "talking" with the cluster, no with the user.
The cluster will be asking to our operator to check if all looks good every few seconds and our operator need to "reconcile" current status with the desired state.
Imagine we want to have running 2 instances of a deployment and suddenly one of them reach and exception and finish.
A "few seconds" later the cluster will ask the operator to check if all looks good in the deployment so the operator will retrieve the list of running instances, will compare with the desired state, and it will decide to create a new pod.
The logic of our SwaggerOperator to decide if all looks good is similar.
Swagger-Operator
So basically what we’ll create is a kubernetes operator to perform these actions:
create a deployment (running the swagger-ui docker image) and a service in case they not exist
maintain a configmap updated with the list of current services present into the cluster.
delete the current pod in case an update in the configmap is done
As an operator is in fact a typical application we’ll create our swagger-operator using the micronaut command line to create it:
mn create-app --lang groovy --features kubernetes-informer swagger-operator
CRD, Custom Resource Definition
First thing is to define a new Kind resource (and deploy it into the cluster). The CRD is where we’ll instruct to the cluster how the new resource will look, so yes, it’s a kind of meta-resource.
In a CRD we need to specify the name of the kind, the group we want to include it and the properties the user need to fill to deploy a new swagger-operator resource
Swagger-operator CRD looks like:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: swaggers.puravida.com
spec:
group: puravida.com
scope: Namespaced
names:
plural: swaggers
singular: swagger
kind: Swagger
versions:
- name: v1
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
serviceSelector:
type: string
configMap:
type: string
deployment:
type: string
service:
type: string
...
Although a long file basically what we’re doing is instructing to the cluster about a new kind of resource (Swagger). When the user will want to create a new resource of this kind he will need to provide 4 properties calledserviceSelector
, configMap
…
- INFO
-
In our case these 4 properties will be used by the operator to create configurable "named" resources instead to use hardcoded values
To let the cluster manage this new resource we need to apply it into the cluster:
$ kubectl apply -f crd.yml
CRD to Java
As we want to use Groovy/Java in our operator, we need to convert this CRD to Java objects. The easy way is to use a command line from kubernetes-client project similar to:
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v "$(pwd)":"$(pwd)" \
-ti --network host ghcr.io/kubernetes-client/java/crd-model-gen:v1.0.6 \
/generate.sh -u $(pwd)/src/k8s/crd.yml -n com.puravida -p com.puravida \
-o $(pwd)
Mounting our local project as a volume the generate.sh
process can read the crd and generate some Java files. They are basically a kind of POJO Java representations of the CRD.
Operator
Swagger-operator is a simple operator and requires only one class, SwaggerOperator.groovy
@Operator(
informer = @Informer(
apiType = V1Swagger,
apiListType = V1SwaggerList,
apiGroup = V1SwaggerWrapper.GROUP,
resourcePlural = V1SwaggerWrapper.PLURAL,
resyncCheckPeriod = 10000L
)
)
class SwaggerOperator implements ResourceReconciler<V1Swagger>{
// The implementation
}
As you can see basically we need to annotate our class with @Operator
and implement ResourceReconciler<V1Swagger>
interface
This interface requires we implement only one method:
@Override
Result reconcile(@NonNull Request request, @NonNull OperatorResourceLister<V1Swagger> lister) {
//
return new Result(false) (1)
}
| 1 | Returning false we inform to the cluster we don’t need a new reconcile "right now" |
reconcile
is the method will be called every X millis (10s in our case) by the cluster once a V1Swagger resource is deployed by the user. Our operator must check if all resources are aligned with the desired state.
To do it the operator needs/can "talk" with different APIs exposed by the cluster as CoreApi or AppApi, so it can list all services present in a namespace, create a configmap, etc
Basically the main logic of the swagger-operator reconcile method is:
if a Swagger resource is present the operator need to check if the configmap, the deployment and the service exist
also, if the Swagger resource exist it must to check if the configmap is up to date checking the list of services and if they is any different it needs to update the configmap and delete the current pod
if the resource is marked to be deleted the operator needs to "clean" the resources created deleting the configmap, service and deployment
Checking the list of services
The operator requests to the cluster a list of current services in the namespace and select all that contains the desired annotation:
def services = coreApi.listNamespacedService(wrapper.namespace)
.items
.findAll({ service->
service.metadata.annotations?.containsKey(wrapper.serviceSelector)
})
def map = services.inject([:],{ map, it ->
map[it.metadata.name] = it.metadata.annotations[wrapper.serviceSelector]
map
}) as Map<String, String>
services
is the current list of services we want to show in the list of swagger-ui and map
is a Map to be "injected" in the ConfigMap.
Checking if ConfigMap is up to date
def configMap = coreApi
.readNamespacedConfigMap(wrapper.configMap,
wrapper.namespace,null,null,null)
def currentJS = configMap.data[CONFIG_YML]
if( currentJS.contains("urls: [$urlServices]") ){
return false
}
The swagger-operator read current ConfigMap data entry CONFIG_YML and check if the current value contains a string similar to the current list of services. If it is the same no action is required
If they are not equals this means some service is not in the list or a new service was deployed so the swagger-operator create a new updated ConfigMap
currentJS = currentJS.replaceFirst(/urls: \[(.*?)]/,"urls: [$urlServices]")
configMap.data[CONFIG_YML] = currentJS
coreApi.replaceNamespacedConfigMap(wrapper.configMap,
wrapper.namespace,
configMap)
and mark current deployment to be restarted
def deployment = deploymentList.items.first()
deployment.spec
.template
.metadata
.annotations[V1SwaggerWrapper.RESTARTED_AT_ANNOTATION]=Instant.now().toString()
appsApi.replaceNamespacedDeployment(deployment.metadata.name,
deployment.metadata.namespace,
deployment)
- INFO
-
You can check out the repository (link at the end of the post) to see full code
Roles and Service account
Probably your operator will require to be run with an special service account how allow them to access to cluster resources. For swagger-operator this is specify in this resource
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: swagger-operator-role
rules:
- apiGroups: ["", "apps"]
resources: ["services", "configmaps", "deployments", "pods"]
verbs: ["get", "watch", "list", "create", "delete", "update"]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "create", "update"]
- apiGroups: ["puravida.com"]
resources: ["swaggers"]
verbs: ["*"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: swagger-operator-sa
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: swagger-operator-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: swagger-operator-role
subjects:
- kind: ServiceAccount
name: swagger-operator-sa
namespace: default
As you can see we are creating a new service account swagger-operator-sa
and allowing to it different access to different resources (full control for swaggers.puravida.com resources for example)
Deploy
As a final step we need to build and deploy our application to a docker registry (in swagger-opeator case using the gradle task jib
) and deploy it into the cluster using a typical deployment
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: swagger-deployment-operator
name: swagger-deployment-operator
spec:
replicas: 1
selector:
matchLabels:
app: swagger-deployment-operator
template:
metadata:
labels:
app: swagger-deployment-operator
spec:
serviceAccountName: swagger-operator-sa
containers:
- image: registry.localhost:5000/swagger-operator
name: swagger-deployment-operator
imagePullPolicy: Always
As you can see we specify the serviceAccountName swagger-operator-sa
created previously.
- WARNING
-
For this example I’m using a local docker registry. In a near future I hope to have time and deploy in docker hub
Last step
So now our cluster is ready and waiting an user (admin, QA, …) create a new Swagger resource specifying wich services to include and wich deployment create
apiVersion: puravida.com/v1
kind: Swagger
metadata:
name: swagger-operator
spec:
serviceSelector: swagger-path
configMap: swagger-config
deployment: swagger
service: swagger
As soon the user apply this file into the cluster, swagger-operator will start receiving events from the cluster to reconcile it.
The operator will list all current services and select some of them, check the ConfigMap is not present and it will create a new one using a classpath resource as template, check the Deployment is not present and create one, etc
After a few seconds we’ll have a new pod running into our cluster with the swagger-ui interface and a list of services configured
If we remove some service (kubectl delete -f user-rest
for example) the operator will recreate all resources and the swagger-ui will be updated automatically
Conclusion
In this article we are creating a Micronaut Kubernetes Operator using Groovy able to create and maintain resources into the cluster
I've created a repo with the code and a service to be used as example at https://github.com/jagedn/swagger-operator
Top comments (0)