3 Tips for Deploying APISIX in Kubernetes (Part 2)

The era of cloud-native computing has seen widespread adoption of Kubernetes as a container orchestration platform, with Apache APISIX emerging as a high-performance, cloud-native dynamic API gateway. Deploying Apache APISIX in Kubernetes has become increasingly common. However, despite the relatively straightforward deployment process of Apache APISIX on Kubernetes, there are still some key issues to consider. In this series of articles, we will delve into the following topics:

1. Considerations for deployment methods
2. Health checks, logging, and monitoring
3. Handling custom plugins and configurations

In the previous article, we discussed the first point. This article will focus on the second point: considerations regarding health checks, logging, and monitoring.

Health Checks

When deploying APISIX in Kubernetes, health checks are particularly important, as they represent a fundamental requirement for applications in Kubernetes. In Kubernetes, by configuring Liveness and Readiness Probes, the health status and availability of APISIX instances can be ensured.

• Liveness Probe is utilized to determine whether the application is running. If the application is deemed unhealthy, Kubernetes will restart the instance.

• Readiness Probe is employed to ascertain if the application is ready to receive traffic. If the application is not yet ready, it will not receive any traffic. This helps prevent traffic from being sent to instances that are either not fully started or are damaged.

By properly configuring Liveness and Readiness Probes, Kubernetes can automatically manage unhealthy Pod instances. This implies that when instances encounter issues, Kubernetes will automatically restart them or stop sending traffic to unhealthy instances, thereby enhancing system availability and stability.

YAML Configuration Example:

apiVersion: v1
kind: Deployment
metadata:
  name: my-apisix-pod
spec:
  containers:
  - name: my-apisix-container
    image: my-apisix-image
    livenessProbe:
      httpGet:
        path: /healthz
        port: 9080
      initialDelaySeconds: 15
      periodSeconds: 10
    readinessProbe:
      httpGet:
        path: /readyz
        port: 9080
      initialDelaySeconds: 10
      periodSeconds: 5

This example defines the Liveness and Readiness Probes for the container. The Liveness Probe sends an HTTP GET request to the path /healthz every 10 seconds to check the container's health status. If the container does not respond or returns a non-200 status code, Kubernetes considers the container unhealthy and attempts to restart it. The Readiness Probe is similar but is used to check if the container is ready to receive traffic.

Monitoring

There are various methods to monitor APISIX at runtime, with integrating Prometheus being a recommended approach. In fact, Prometheus remains the most widely used monitoring component to date.

Integrating Prometheus assists in collecting and monitoring metrics for APISIX and the services it proxies. These metrics may encompass request rates, error rates, latency, and other critical performance indicators. By monitoring these metrics, issues can be promptly identified and performance tuning and troubleshooting can be conducted. Ensure proper configuration of metrics and alerting rules to take action promptly when issues arise.

Enabling the Prometheus plugin in APISIX is straightforward. First, set the export_uri in config.yaml.

plugin_attr:
  prometheus:
    export_uri: /apisix/metrics

Then enable the plugin on the API or service that needs to be statistically analyzed by Prometheus.

curl http://127.0.0.1:9180/apisix/admin/routes/1  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "uri": "/hello",
    "plugins": {
        "prometheus":{}
    },
    "upstream": {
        ...
    }
}'

Finally, the Prometheus server can periodically pull configurations from the export_uri.

scrape_configs:
  - job_name: "apisix"
    scrape_interval: 15s
    metrics_path: "/apisix/prometheus/metrics"
    static_configs:
      - targets: ["127.0.0.1:9091"]

In practical use, Prometheus also requires deployment in a highly available manner, such as using open-source solutions like Thanos. Integration with APISIX can be achieved using the Thanos sidecar mode.

Logging

In APISIX, important logs are broadly categorized into two types: traffic logs and audit logs.

• Traffic logs refer to the logging of each request when APISIX acts as a reverse proxy. These logs, containing both request traffic and returned information, along with internal operation logs of APISIX, are crucial for tracing and troubleshooting. Typically, appropriate log levels and formats are set for recording. In practical scenarios, consider outputting logs to a centralized logging system, such as ELK (Elasticsearch, Logstash, and Kibana), Fluentd, or Splunk. APISIX provides log plugins for selection.

• Audit logs primarily refer to the logs generated when managing APISIX configurations. They not only aid in meeting compliance requirements but also serve for security analysis. By analyzing audit logs, potential security risks and improper configurations or management behaviors can be identified, and corresponding measures can be taken to enhance system security.

The open-source APISIX provides an Admin API for easy configuration distribution but lacks configuration related to audit logs. Typically, users need to record these logs themselves or use the enterprise edition of APISIX.

There are not many differences in logging configuration between Kubernetes and other environments. It's worth noting that when configuring upstream and related information in APISIX, Kubernetes service discovery is commonly used. It's recommended to record the service name in logs for easier troubleshooting in subsequent issues.

Conclusion

By configuring health check mechanisms to detect unhealthy instances of APISIX, Kubernetes can promptly take action to migrate traffic and facilitate recovery, thereby ensuring the continuity and stability of API services. APISIX also supports integration with advanced monitoring tools such as Prometheus, enabling monitoring of API performance and stability, including key metrics such as request rates, error rates, and latency. This monitoring capability allows organizations to swiftly identify potential issues, and conduct timely performance tuning and optimization, ensuring the efficient operation of API services.