DEV Community

Ajeet Singh Raina
Ajeet Singh Raina

Posted on • Edited on

Kubernetes and AI: 3 Open Source Tools Powered by OpenAI

As per the recent Cloud-Native AI report generated by CNCF during the KubeCon + CloudNativeCon Europe conference, AI is empowering operators and developers to work smarter, not harder. The convergence of Cloud Native methodologies with Artificial Intelligence (AI) has emerged as a transformative force reshaping industries and driving innovation. Using AI to improve cloud-native systems is no longer science fiction.

Image1

Cloud Native technologies have revolutionized the way applications are developed, deployed, and managed in modern IT environments. By leveraging containerization, microservices architecture, and orchestration platforms like Kubernetes, organizations can achieve scalability, resilience, and agility in their operations. On the other hand, Artificial Intelligence and Machine Learning have paved the way for data-driven decision-making, predictive analytics, and automation across various domains.

Kubernetes, the container orchestration platform, is a powerful tool for managing applications in the cloud. But what if you could add some artificial intelligence muscle to your Kubernetes workflow? This is where OpenAI and these 3 open-source projects come in: Kubectl OpenAI client, K8sGPT, and KoPylot. These innovative tools leverage OpenAI's capabilities to streamline tasks, automate processes, and gain deeper insights from your Kubernetes clusters.

Let's dive in and explore how they can supercharge your Kubernetes experience.

https://www.youtube.com/watch?v=SoEAFawQ9y4

1. KoPylot

KoPylot is a cloud-native application performance monitoring (APM) solution that runs on Kubernetes. It is designed to help developers and operations teams diagnose and troubleshoot issues in complex distributed systems. It provides real-time insights into application performance, including metrics, traces, and logs, to help teams identify and resolve issues quickly.

KoPylot is designed to help teams monitor and diagnose Kubernetes applications with ease. It provides a comprehensive view of application performance, including real-time metrics, traces, and logs, to help teams identify and resolve issues quickly.

Features

KoPylot provides a wide range of features to help teams monitor and diagnose Kubernetes applications, including:

  • Real-time Metrics: KoPylot provides real-time metrics for Kubernetes workloads, including CPU, memory, and network usage. It also provides metrics for custom application-level metrics, which can be used to monitor specific application behaviors and performance.

  • Distributed Tracing: KoPylot provides distributed tracing capabilities, allowing teams to trace requests across multiple microservices and identify bottlenecks and performance issues.

  • Logs: KoPylot provides log aggregation capabilities, allowing teams to centralize logs from multiple containers and pods running on Kubernetes.

  • Audit: KoPylot provides auditing capabilities, allowing teams to track changes to Kubernetes resources and monitor access to Kubernetes API server.

  • Chat: KoPylot provides a chat interface, allowing teams to collaborate and share insights in real-time.

  • Diagnose: KoPylot provides a diagnose feature, allowing teams to quickly identify issues and find potential solutions.

How it Works

KoPylot works by integrating with Kubernetes clusters and exposing a web-based interface that provides users with a comprehensive view of their workloads. KoPilot’s architecture consists of a set of microservices that communicate with each other over HTTP and gRPC.

When a user logs into KoPylot, they are presented with a dashboard that shows an overview of all the workloads running on their Kubernetes cluster. From the dashboard, users can drill down into individual workloads to view details such as resource utilization, status, and logs.

KoPylot communicates with Kubernetes clusters using the Kubernetes API server. The API server provides a standard interface for interacting with Kubernetes clusters and enables KoPilot to manage and monitor workloads running on any Kubernetes cluster, regardless of the underlying infrastructure.

KoPylot’s microservices are designed to be highly scalable and fault-tolerant. The microservices are deployed as Kubernetes pods and can be scaled horizontally as workload demand increases. Additionally, KoPilot’s microservices are designed to recover automatically from failures, ensuring that the system remains available and responsive at all times.

To diagnose issues in a Kubernetes cluster, KoPylot provides a diagnostic tool that can help identify problems quickly. The diagnostic tool checks for common issues and misconfigurations in the cluster and provides suggestions for resolving them. For example, it can check if any pods are not running, if there are any pending pods, or if any nodes are unreachable. The diagnostic tool can also help identify performance issues in the cluster by analyzing the resource usage of nodes and pods. It provides detailed information about CPU and memory usage, as well as network and disk I/O metrics.

In addition to monitoring and diagnosing issues, KoPylot also provides a chat interface that allows users to interact with the system using Natural Language. This feature enables users to ask questions and get answers about the status of the cluster, the configuration of specific resources, or any other aspect of the system.

Finally, KoPylot includes an audit log that records all actions taken by users within the system. This log can be used to trace changes made to the cluster and to identify any potential security issues.

The following is an example of using KoPylot to perform a health check on a service:



kopylot diagnose check -s my-service -n my-namespace
Enter fullscreen mode Exit fullscreen mode

This command will run a series of checks on the specified service, including checking for pod readiness, HTTP status codes, and TCP connectivity. If any issues are found, KoPylot will provide a detailed report of the problem.

Installation

To install KoPylot on Kubernetes, you can follow these steps:

1. Requests an API key from OpenAI

Export the key using the following command:



export KOPYLOT_AUTH_TOKEN=your_api_key
Enter fullscreen mode Exit fullscreen mode

2. Install KoPylot using pip: 



# pip install kopylot
Enter fullscreen mode Exit fullscreen mode

3. Run KoPylot 



# kopylot --help
Enter fullscreen mode Exit fullscreen mode 


Usage: kopylot [OPTIONS] COMMAND [ARGS]...                                           

╭─ Options ──────────────────────────────────────────────────────────────────────────╮
│ --version                                                                          │
│ --install-completion        [bash|zsh|fish|powershell  Install completion for the  │
│                             |pwsh]                     specified shell.            │
│                                                        [default: None]             │
│ --show-completion           [bash|zsh|fish|powershell  Show completion for the     │
│                             |pwsh]                     specified shell, to copy it │
│                                                        or customize the            │
│                                                        installation.               │
│                                                        [default: None]             │
│ --help                                                 Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ─────────────────────────────────────────────────────────────────────────╮
│ audit     Audit a pod, deployment, or service using an LLM model.                  │
│ chat      Start a chat with kopylot to generate kubectl commands based your        │
│           inputs.                                                                  │
│ ctl       A wrapper around kubectl. The arguments passed to the ctl subcommand are │
│           interpreted by kubectl.                                                  │
│ diagnose  Diagnose a resource e.g. pod, deployment, or service using an LLM model. │
╰────────────────────────────────────────────────────────────────────────────────────╯
Enter fullscreen mode Exit fullscreen mode

Local Setup

If you prefer to set up your development environment locally, make sure you have Poetry installed on your system. Then, follow these steps:

Clone the KoPylot repository: 



git clone https://github.com/avsthiago/kopylot
Enter fullscreen mode Exit fullscreen mode

Navigate to the project folder 



cd kopylot
Enter fullscreen mode Exit fullscreen mode

Install the project dependencies using Poetry 



make install
Enter fullscreen mode Exit fullscreen mode

Real Workload Example

Now that we have covered the features and functionality of KoPylot, let’s take a look at a real workload example to see how it can be used in practice.
In this example, we will use KoPylot to diagnose a problem with a Kubernetes deployment.

Deploy a sample workload using the following YAML manifest:



apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
Enter fullscreen mode Exit fullscreen mode 


kubectl apply -f nginx-deployment.yaml
Enter fullscreen mode Exit fullscreen mode

Run the following command to diagnose the deployment using KoPylot:



kopilot diagnose deployment nginx --namespace default
Enter fullscreen mode Exit fullscreen mode

KoPylot will output the following result:



===========================================================================
Diagnosing Deployment nginx in namespace default
===========================================================================

---------------------------------------------------------------------------
Deployment nginx is running
---------------------------------------------------------------------------

  Reason: The deployment is running correctly.

---------------------------------------------------------------------------
Deployment nginx is accessible
---------------------------------------------------------------------------

  Reason: The deployment is accessible via the service.

---------------------------------------------------------------------------
Deployment nginx has enough resources
---------------------------------------------------------------------------

  Reason: The deployment has enough resources.

---------------------------------------------------------------------------
Deployment nginx is not outdated
---------------------------------------------------------------------------

  Reason: The deployment is using the latest available image.

===========================================================================
Diagnosis complete.
===========================================================================
Enter fullscreen mode Exit fullscreen mode

This indicates that the deployment is running correctly and no issues were found.

Overall, KoPylot is a useful tool for diagnosing and troubleshooting Kubernetes workloads. Its web-based chat interface and CLI make it easy to use and accessible to all levels of users.

2. K8sGPT

K8sGPT is a tool that uses NLP to analyze logs and other data from Kubernetes clusters to identify and diagnose issues. It has a set of built-in analyzers that are designed to identify common issues such as pod crashes, service failures, and ingress misconfigurations. K8sGPT is built on top of OpenAI's GPT-3 language model, which allows it to understand natural language and provide explanations that are easy to understand.

K8sGPT is focused on triaging and diagnosing issues in your cluster. It is a tool for SRE, Platform, and DevOps engineers to help them understand what is going on in their cluster and find the root cause of an issue. It can help you cut through the noise of logs and multiple tools to find the root cause of an issue.

How K8sGPT Works

K8sGPT uses a set of analyzers that are designed to identify issues in Kubernetes clusters. These analyzers use NLP to analyze logs, metrics, and other data from your cluster to identify potential issues. When an issue is identified, K8sGPT provides an explanation in natural language that is easy to understand. This allows you to quickly understand the issue and take the necessary steps to resolve it.

K8sGPT is built on top of OpenAI's GPT-3 language model, which allows it to understand natural language. This means that you can ask K8sGPT questions about your cluster in plain English, and it will provide a response that is easy to understand. For example, you can ask K8sGPT "Why is my pod crashing?" and it will provide an explanation of why the pod is crashing and what steps you can take to fix the issue.

Installation

K8sGPT can be installed on Linux, Mac, and Windows. The easiest way to install K8sGPT on Linux or Mac is via Homebrew. To install K8sGPT via Homebrew, run the following commands:



brew tap k8sgpt-ai/k8sgptbrew install k8sgpt
Enter fullscreen mode Exit fullscreen mode

If you encounter an error when installing K8sGPT on WSL or Linux, you may need to install the build-essential package. You can do this by running the following commands:
 



sudo apt-get updatesudo apt-get install build-essential
Enter fullscreen mode Exit fullscreen mode

 
If you are running on Windows, you can download the latest Windows binaries from the Release tab on GitHub.
Once k8sgpt is installed, you can verify that it is working by running the version command:



k8sgpt version
Enter fullscreen mode Exit fullscreen mode

Generating an API key

Before we can use k8sgpt, we need to generate an API key from OpenAI. To generate an API key, run below command , this will open a link in your default web browser where you can generate an API key. Once you have generated an API key, you can set it in k8sgpt



k8sgpt generateK8sgpt auth
Enter fullscreen mode Exit fullscreen mode

This will prompt you to enter your API key. Once you have entered your API key, k8sgpt will be able to use it to analyze your Kubernetes clusters.

Analyzing your Kubernetes clusters

Now that we have k8sgpt installed and authenticated, we can start analyzing our Kubernetes clusters. To analyze a Kubernetes cluster, run the following command:



k8sgpt analyze
Enter fullscreen mode Exit fullscreen mode

This will scan your Kubernetes cluster and look for any issues. By default, k8sgpt will use all of its built-in analyzers to analyze your cluster.

If k8sgpt finds any issues, it will output a summary of the issues it found. For example:



Analyzer: podAnalyzer
  Namespace: default
  Name: nginx-66b6c48dd5-zk6jt
  Kind: Pod
  Reason: CrashLoopBackOff
  Message: Back-off 5m0s restarting failed container=nginx pod=nginx-66b6c48dd5-zk6jt_default(25f13c57-04eb-4a0a-a2f7-17b7564a7944)
 

Refer:   https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/ 
Enter fullscreen mode Exit fullscreen mode

This output tells us that there is an issue with the nginx pod in the default namespace. The pod is in a CrashLoopBackOff state, and the nginx container in the pod has failed and is being restarted every 5 minutes.

If you want to see more information about the issue, you can use the --explain flag:



k8sgpt analyze --explain
Enter fullscreen mode Exit fullscreen mode

This will output a more detailed explanation of the issues that k8sgpt found.

Filtering resources

By default, k8sgpt will analyze all resources in your Kubernetes cluster. However, you can filter the resources that k8sgpt analyzes by using the --filter flag.
Filters are used to control which Kubernetes resources are analyzed. By default, all filters are enabled. You can manage filters using the filters subcommand.
To view the list of filters, run:
Add filter , remove , list and more



k8sgpt filters list
k8sgpt filters add [filter]
k8sgpt filters add Service,Pod
k8sgpt filters remove [filter]
k8sgpt filters remove Service
Enter fullscreen mode Exit fullscreen mode

You can filter the resources by namespace using the --namespace flag:



k8sgpt analyze --namespace=default
Enter fullscreen mode Exit fullscreen mode

Custom Analyzers

k8sgpt comes with a set of built-in analyzers that cover a variety of Kubernetes resources. However, you can also write your own custom analyzers to analyze your own resources.

To write a custom analyzer, you will need to create a new Go package that implements the Analyzer interface:



type Analyzer interface {
// Analyze analyzes the given Kubernetes resource and returns any issues found
Analyze(resource *unstructured.Unstructured) ([ ]*Issue, error)
// Name returns the name of the analyzer
Name() string
// Enabled returns whether the analyzer is enabled
Enabled() bool
}
Enter fullscreen mode Exit fullscreen mode

The Analyze method takes a Kubernetes resource as input and returns an array of issues found. The Name method returns the name of the analyzer, and the Enabled method returns whether the analyzer is enabled.
Once you have written your custom analyzer, you can register it with k8sgpt by calling the RegisterAnalyzer function:



RegisterAnalyzer(name string, analyzer Analyzer)
k8sgpt.RegisterAnalyzer("MyAnalyzer", &MyAnalyzer{})
Enter fullscreen mode Exit fullscreen mode

You can then use the --filter flag to filter by your custom analyzer:



k8sgpt analyze --filter=MyAnalyzer
Enter fullscreen mode Exit fullscreen mode

You can also combine multiple filters together to create more complex filters. For example, the following command will only analyze Pod resources in the "default" namespace with a label of "app=myapp":



k8sgpt analyze --filter=Pod --namespace=default --label-selector=app=myapp
Enter fullscreen mode Exit fullscreen mode

K8sGPT is a powerful tool that can help you diagnose and triage issues in your Kubernetes clusters. Its ability to analyze logs and Kubernetes resources using natural language processing and AI makes it stand out from other monitoring tools.

By installing and configuring K8sGPT, you can easily scan your clusters, identify issues, and get recommendations on how to fix them. Additionally, its built-in analyzers and filters make it easy to customize the analysis to fit your specific needs.Whether you're an SRE, Platform or DevOps engineer, K8sGPT can help you gain insights into your Kubernetes clusters and make your job easier. Try it out today and see how it can help you improve your Kubernetes monitoring and troubleshooting workflows!

3. Kubectl OpenAI Client

The Kubectl OpenAI client project is a kubectl plugin to generate and apply Kubernetes manifests using OpenAI GPT.

Getting Started

  • Install Docker Desktop
  • Install Kubectl-ai

You can install it on your Macbook directly using Homebrew:



brew tap sozercan/kubectl-ai 
https://github.com/sozercan/kubectl-ai
brew install kubectl-ai
Enter fullscreen mode Exit fullscreen mode

Get OpenAI Keys

You can get the OpenAI keys from https://platform.openai.com/account/api-keys

Please Note: kubectl-ai requires an OpenAI API key or an Azure OpenAI Service API key and endpoint, and a valid Kubernetes configuration.



export OPENAI_API_KEY=<your OpenAI key>
Enter fullscreen mode Exit fullscreen mode

Installing on CentOS 



yum install wget
wget https://github.com/sozercan/kubectl-ai/releases/download/v0.0.10/kubectl-ai_linux_amd64.tar.gz
tar xvf kubectl-ai_linux_amd64.tar.gz
mv kubectl-ai /usr/local/bin/kubectl-ai
Enter fullscreen mode Exit fullscreen mode

Setting up Kubeview

Using Helm

Assuming that you have already installed Git and Helm on your laptop, follow the below steps



git clone https://github.com/benc-uk/kubeview
cd kubeview/charts/
helm install kubeview kubeview
Enter fullscreen mode Exit fullscreen mode

Testing it locally 



kubectl port-forward svc/kubeview -n default 80:80
Enter fullscreen mode Exit fullscreen mode

Deploying Pod using namespace 



kubectl ai "Create a namespace called ns1 and deploy a Nginx Pod"        
✨ Attempting to apply the following manifest:

---
apiVersion: v1
kind: Namespace
metadata:
  name: ns1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: ns1
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
Use the arrow keys to navigate: ↓ ↑ → ← 
? Would you like to apply this? [Reprompt/Apply/Don't Apply]: 
+   Reprompt
  ▸ Apply
    Don't Apply
Enter fullscreen mode Exit fullscreen mode

The YAML manifest you provided creates a basic Nginx pod with the name “nginx-pod” and exposes port 80. To apply this manifest and create the pod, you can use the kubectl apply command. Save the manifest in a file, for example, nginx-pod.yaml, and then execute the following command in your terminal:

Difference between “Create” and “Deploy” [Be Careful] 



kubectl ai "Create a namespace called ns1 and create a Nginx Pod"
✨ Attempting to apply the following manifest:

apiVersion: v1
kind: Namespace
metadata:
  name: ns1
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: ns1
spec:
  containers:
  - name: nginx
    image: nginx:1.7.9
    ports:
    - containerPort: 80
✔ Apply
Enter fullscreen mode Exit fullscreen mode

Accessing the Nginx Pod via Web Browser 



kubectl port-forward nginx 8000:80 -n ns1
Forwarding from 127.0.0.1:8000 -> 80
Forwarding from [::1]:8000 -> 80
Handling connection for 8000
Handling connection for 8000
Enter fullscreen mode Exit fullscreen mode

Deployment

This is an example for deploying 3 replicas in the specific namespace



kubectl ai "create an nginx deployment with 3 replicas under namespace ns1"
✨ Attempting to apply the following manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: ns1
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: webpage
        image: ajeetraina/webpage
        ports:
        - containerPort: 80
✔ Apply
Enter fullscreen mode Exit fullscreen mode

Services 



kubectl ai "create an nginx deployment with 3 replicas under namespace ns1 and this time create service type as NodePort"
✨ Attempting to apply the following manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: ns1
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: webpage
        image: ajeetraina/webpage
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: ns1
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
    nodePort: 30080
Use the arrow keys to navigate: ↓ ↑ → ← 
? Would you like to apply this? [Reprompt/Apply/Don't Apply]: 
+   Reprompt
  ▸ Apply
    Don't Apply
Enter fullscreen mode Exit fullscreen mode

Listing the Kubernetes Resources 



kubectl get po,deploy,svc -n ns1
NAME                                    READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-58945458f5-5pk6b   1/1     Running   0          28s
pod/nginx-deployment-58945458f5-7htd7   1/1     Running   0          28s
pod/nginx-deployment-58945458f5-s6cxm   1/1     Running   0          28s

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   3/3     3            3           28s

NAME                    TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/nginx-service   NodePort   10.100.230.251   <none>        80:30080/TCP   28s
Enter fullscreen mode Exit fullscreen mode

krs

Krs is here to change the game! This project utilizes GenAI technology to recommend the perfect Kubernetes tools for your unique environment. Say goodbye to endless searches and hello to a streamlined, efficient workflow.

What makes Krs unique?

Krs is a Kubernetes cluster health monitoring and tools recommendation service. The primary goal of KRS is to provide insights into the current state of a Kubernetes cluster, identify potential issues, and suggest relevant tools and resources to enhance the cluster's efficiency and security.

The project is designed to work with a local or remote Kubernetes cluster, and it utilizes various data sources, such as CNCF tools, Kubernetes landscape, and LLM (Language Model) for contextual analysis. KRS aims to provide actionable recommendations based on the cluster's current state and the latest trends in the Kubernetes ecosystem.

How does it work?

Image2

The project is built using Python and is designed to be easily installable and configurable. It provides a command-line interface (CLI) for users to interact with the tool. The project is open-source and available on GitHub at https://github.com/kubetoolsca/krs.

To achieve this, KRS follows a multi-step process:

  • Scans the Kubernetes cluster for resource usage, configuration, and potential issues.
  • Fetches data from CNCF tools, Kubernetes landscape, and other relevant sources.
  • Utilizes LLM for contextual analysis and understanding of the cluster's state.

  • Provides recommendations for improving the cluster's efficiency, security, and resource utilization.

  • Reduced Time Spent Searching: Krs helps you find the right tools quickly and easily.

  • Improved Efficiency: Get matched with tools that perfectly align with your needs.

  • Cost Optimization: Reduce wasted resources and optimize your Kubernetes cluster performance.

We're excited to share Krs with the developer community! We believe this open-source project has the potential to revolutionize the way DevOps and DevSecOps teams approach Kubernetes tooling.

Getting Started

Clone the repository 



git clone https://github.com/kubetoolsca/krs.git
Enter fullscreen mode Exit fullscreen mode

Install the Krs Tool

Change directory to /krs and run the following command to install krs locally on your system:



pip install .
Enter fullscreen mode Exit fullscreen mode

Krs CLI 



 krs --help

 Usage: krs [OPTIONS] COMMAND [ARGS]...

 krs: A command line interface to scan your Kubernetes Cluster, detect errors, provide resolutions using LLMs and recommend latest tools for your cluster

╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --install-completion          Install completion for the current shell.                                                                                       │
│ --show-completion             Show completion for the current shell, to copy it or customize the installation.                                                │
│ --help                        Show this message and exit.                                                                                                     │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ exit         Ends krs services safely and deletes all state files from system. Removes all cached data.                                                       │
│ export       Exports pod info with logs and events.                                                                                                           │
│ health       Starts an interactive terminal using an LLM of your choice to detect and fix issues with your cluster                                            │
│ init         Initializes the services and loads the scanner.                                                                                                  │
│ namespaces   Lists all the namespaces.                                                                                                                        │
│ pods         Lists all the pods with namespaces, or lists pods under a specified namespace.                                                                   │
│ recommend    Generates a table of recommended tools from our ranking database and their CNCF project status.                                                  │
│ scan         Scans the cluster and extracts a list of tools that are currently used.                                                                          │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Enter fullscreen mode Exit fullscreen mode

Initialise and load the scanner

Run the following command to initialize the services and loads the scanner.



krs init
Enter fullscreen mode Exit fullscreen mode

Scan your cluster

Run the following command to scan the cluster and extract a list of tools that are currently used.



krs scan
Enter fullscreen mode Exit fullscreen mode

You will see the following results:




Scanning your cluster...

Cluster scanned successfully...

Extracted tools used in cluster...


The cluster is using the following tools:

+-------------+--------+------------+---------------+
| Tool Name   | Rank   | Category   | CNCF Status   |
+=============+========+============+===============+
+-------------+--------+------------+---------------+
Enter fullscreen mode Exit fullscreen mode

Lists all the namespaces 



krs namespaces
Namespaces in your cluster are:

1. default
2. kube-node-lease
3. kube-public
4. kube-system
Enter fullscreen mode Exit fullscreen mode

Installing sample Kubernetes Tools

Assuming that you already have a bunch of Kubernetes tools running in your infrastructure.
If not, you can leverage samples/install-tools.sh script to install these sample tools.



cd samples
sh install-tools.sh
Enter fullscreen mode Exit fullscreen mode

Use scanner 



krs scan

Scanning your cluster...

Cluster scanned successfully...

Extracted tools used in cluster...


The cluster is using the following tools:

+-------------+--------+----------------------+---------------+
| Tool Name   |   Rank | Category             | CNCF Status   |
+=============+========+======================+===============+
| kubeshark   |      4 | Alert and Monitoring | unlisted      |
+-------------+--------+----------------------+---------------+
| portainer   |     39 | Cluster Management   | listed        |
+-------------+--------+----------------------+---------------+
Enter fullscreen mode Exit fullscreen mode

Kubetools Recommender System

Generates a table of recommended tools from our ranking database and their CNCF project status.



krs recommend

Our recommended tools for this deployment are:

+----------------------+------------------+-------------+---------------+
| Category             | Recommendation   | Tool Name   | CNCF Status   |
+======================+==================+=============+===============+
| Alert and Monitoring | Recommended tool | grafana     | listed        |
+----------------------+------------------+-------------+---------------+
| Cluster Management   | Recommended tool | rancher     | unlisted      |
+----------------------+------------------+-------------+---------------+
Enter fullscreen mode Exit fullscreen mode

Krs health

Assuming that there is a Nginx Pod under the namespace ns1



krs pods --namespace ns1

Pods in namespace 'ns1':

1. nginx-pod
Enter fullscreen mode Exit fullscreen mode 


krs health

Starting interactive terminal...


Choose the model provider for healthcheck:

[1] OpenAI
[2] Huggingface

>>
Enter fullscreen mode Exit fullscreen mode

The user is prompted to choose a model provider for the health check.
The options provided are "OpenAI" and "Huggingface". The selected option determines which LLM model will be used for the health check.

Let's say you choose the option "1", then it will install the necessary libraries.



Enter your OpenAI API key: sk-3iXXXXXTpTyyOq2mR

Enter the OpenAI model name: gpt-3.5-turbo
API key and model are valid.

Namespaces in the cluster:

1. default
2. kube-node-lease
3. kube-public
4. kube-system
5. ns1

Which namespace do you want to check the health for? Select a namespace by entering its number: >> 5

Pods in the namespace ns1:

1. nginx-pod

Which pod from ns1 do you want to check the health for? Select a pod by entering its number: >>
Checking status of the pod...

Extracting logs and events from the pod...

Logs and events from the pod extracted successfully!


Interactive session started. Type 'end chat' to exit from the session!

>>  The provided log entries are empty, as there is nothing between the curly braces {}. Therefore, everything looks good and there are no warnings or errors to report.
Enter fullscreen mode Exit fullscreen mode

Let us pick up an example of Pod that throws an error:




krs health



Starting interactive terminal...



Do you want to continue fixing the previously selected pod ? (y/n): >> n



Loading LLM State..



Model:  gpt-3.5-turbo



Namespaces in the cluster:



    

  1. default
    2. 

  2. kube-node-lease
    3. 

  3. kube-public
    4. 

  4. kube-system
    5. 

  5. portainer
    6. 




Which namespace do you want to check the health for? Select a namespace by entering its number: >> 4



Pods in the namespace kube-system:



    

  1. coredns-76f75df574-mdk6w
    2. 

  2. coredns-76f75df574-vg6z2
    3. 

  3. etcd-docker-desktop
    4. 

  4. kube-apiserver-docker-desktop
    5. 

  5. kube-controller-manager-docker-desktop
    6. 

  6. kube-proxy-p5hw4
    7. 

  7. kube-scheduler-docker-desktop
    8. 

  8. storage-provisioner
    9. 

  9. vpnkit-controller
    10. 




Which pod from kube-system do you want to check the health for? Select a pod by entering its number: >> 4



Checking status of the pod...



Extracting logs and events from the pod...



Logs and events from the pod extracted successfully!



Interactive session started. Type 'end chat' to exit from the session!



>>  Warning/Error 1:

"Unable to authenticate the request" with err="[invalid bearer token, service account token has expired]"

This indicates that there was an issue with authenticating the request due to an invalid bearer token and an expired service account token. To resolve this issue, the bearer token needs to be updated or regenerated, and the service account token needs to be renewed.



Warning/Error 2:

"Failed to update lease" with err="StorageError: invalid object, Code: 4, Key: /registry/leases/kube-system/apiserver-457xdkumpkdqmkwikdt6khrbya"

This error indicates a storage error while trying to update a lease, possibly due to an invalid object or a conflict in UIDs. To resolve this issue, the lease object needs to be checked and possibly recreated with the correct UID.



Warning/Error 3:

"Error removing old endpoints from kubernetes service: no API server IP addresses were listed in storage, refusing to erase all endpoints for the kubernetes Service"

This warning indicates that there were no API server IP addresses listed in storage, causing the removal of old endpoints from the Kubernetes service to fail. To resolve this issue, the API server IP addresses need to be correctly listed in storage.



Warning/Error 4:

"Skipping API apps/v1beta1 because it has no resources."

This warning indicates that the API version 'apps/v1beta1' is being skipped because it has no associated resources. This might be an informational warning and may not necessarily require immediate action, depending on the use case.



Warning/Error 5:

"watch chan error: etcdserver: mvcc: required revision has been compacted"

This error indicates a watch channel error related to the revision being compacted in etcdserver. To resolve this issue, the etcd database revision or configuration may need to be checked and potentially adjusted.



Overall, it is important to investigate and address each warning and error to ensure the stability and reliability of the Kubernetes cluster. If there are no concerns in the log entries provided, the message 'Everything looks good!' can be returned.



>> can you help me in fixing this error message

>>  To help you fix the error messages mentioned in the log entries, here are some steps you can take:



    

  1. 

    For the "Unable to authenticate the request" error:
    


      

    • Generate a new bearer token and update it in the relevant configuration files.
      • 

    • Renew the service account token to prevent expiration issues.
      • 

    

    2. 

  2. 

    For the "Failed to update lease" error:
    


      

    • Investigate the storage error and check for any invalid objects.
      • 

    • Recreate the lease object with the correct UID if needed.
      • 

    

    3. 

  3. 

    For the "Error removing old endpoints from Kubernetes service" error:
    


      

    • Ensure that the API server IP addresses are correctly listed in the storage.
      • 

    • Verify the configuration of the Kubernetes service and update it if necessary.
      • 

    

    4. 

  4. 

    For the "watch chan error: etcdserver: mvcc: required revision has been compacted" error:
    


      

    • Check the etcd server configuration and ensure that the revision is maintained correctly.
      • 

    • Monitor the etcd server for any potential issues causing revision compaction.
      • 

    

    5. 




After performing these steps, restart relevant services or components as needed to apply the changes. Monitor the Kubernetes cluster for any further errors and ensure that the issues have been resolved successfully.



Feel free to provide more specific details or additional logs if you need further assistance with resolving the error messages.
Enter fullscreen mode Exit fullscreen mode




Using Hugging Face





krs health



Starting interactive terminal...



Choose the model provider for healthcheck:



[1] OpenAI

[2] Huggingface



>> 2



Installing necessary libraries..........



transformers is already installed.



torch is already installed.

/opt/homebrew/lib/python3.11/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.

  torch.utils._pytree._register_pytree_node(



Enter the Huggingface model name: codellama/CodeLlama-13b-hf

tokenizer_config.json: 100%|█████████████████████████████████████████████| 749/749 [00:00<00:00, 768kB/s]

tokenizer.model: 100%|████████████████████████████████████████████████| 500k/500k [00:00<00:00, 1.94MB/s]

tokenizer.json: 100%|███████████████████████████████████████████████| 1.84M/1.84M [00:01<00:00, 1.78MB/s]

special_tokens_map.json: 100%|██████████████████████████████████████████| 411/411 [00:00<00:00, 1.49MB/s]

config.json: 100%|██████████████████████████████████████████████████████| 589/589 [00:00<00:00, 1.09MB/s]

model.safetensors.index.json: 100%|█████████████████████████████████| 31.4k/31.4k [00:00<00:00, 13.9MB/s]

...
Enter fullscreen mode Exit fullscreen mode




Get Involved!

We welcome your contributions and feedback!
Let's work together to build a smarter, more efficient future for Kubernetes!

Top comments (0)

Read next

ahmad_tibibi profile image

TypeScript: Understanding TS1005 Error

Ahmad Tibibi -

chirag_elixent profile image

Frontend vs. Backend Frameworks: A Full-Stack Perspective for Businesses

Chirag Elixent -

hannahanot profile image

Understanding Tailwind CSS

Hannah -

ahmad_tibibi profile image

TS1009: Trailing comma not allowed.

Ahmad Tibibi -