DEV Community

Cover image for Mastering ingress-nginx
Ofir Haim for Target-Ops

Posted on • Updated on

Mastering ingress-nginx

Advanced Optimization Techniques for High-Performance Kubernetes Ingress

As a seasoned DevOps engineer with 15 years of experience, I've seen the evolution of ingress controllers in Kubernetes. Today, we'll dive deep into optimizing ingress-nginx, one of the most popular ingress controllers in the Kubernetes ecosystem.

1. Fine-tuning Worker Processes and Connections

One of the first areas to optimize is the NGINX worker configuration. By default, ingress-nginx uses auto-detection, but for high-traffic environments, manual tuning can yield better results.

controller:
  config:
    worker-processes: "8"
    max-worker-connections: "65536"
Enter fullscreen mode Exit fullscreen mode

Adjust these values based on your server's CPU cores and expected concurrent connections.

2. Leveraging HTTP/2 and TLS 1.3

Enable HTTP/2 and TLS 1.3 to improve connection efficiency and security:

controller:
  config:
    use-http2: "true"
    ssl-protocols: "TLSv1.2 TLSv1.3"
Enter fullscreen mode Exit fullscreen mode

3. Optimizing SSL/TLS

Implement OCSP stapling and adjust SSL buffer size:

controller:
  config:
    ssl-buffer-size: "4k"
    ssl-ocsp: "on"
Enter fullscreen mode Exit fullscreen mode

4. Tuning Timeouts and Keepalive

Adjust timeouts and keepalive settings to balance between resource usage and client needs:

controller:
  config:
    keep-alive: "75"
    keep-alive-requests: "100"
    upstream-keepalive-connections: "1000"
    upstream-keepalive-timeout: "60"
    client-header-timeout: "60s"
    client-body-timeout: "60s"
Enter fullscreen mode Exit fullscreen mode

5. Implementing Caching and Compression

Enable and configure caching and compression for improved performance:

controller:
  config:
    use-gzip: "true"
    gzip-level: "6"
    gzip-types: "application/json application/x-javascript text/css text/javascript"
    proxy-cache-path: "/tmp/nginx-cache levels=1:2 keys_zone=my_cache:10m max_size=10g inactive=60m use_temp_path=off"
Enter fullscreen mode Exit fullscreen mode

Then, in your Ingress resource:

annotations:
  nginx.ingress.kubernetes.io/proxy-cache: "my_cache"
  nginx.ingress.kubernetes.io/proxy-cache-valid: "200 60m"
Enter fullscreen mode Exit fullscreen mode

6. Implementing Rate Limiting

Protect your services with intelligent rate limiting:

annotations:
  nginx.ingress.kubernetes.io/limit-rps: "10"
  nginx.ingress.kubernetes.io/limit-rpm: "100"
Enter fullscreen mode Exit fullscreen mode

7. Optimizing Backend Connections

Fine-tune how ingress-nginx connects to your backends:

controller:
  config:
    upstream-keepalive-connections: "1000"
    upstream-keepalive-timeout: "60"
    upstream-keepalive-requests: "1000"
Enter fullscreen mode Exit fullscreen mode

8. Implementing Custom Error Pages

Enhance user experience with custom error pages:

controller:
  config:
    custom-http-errors: "404,503"
  defaultBackendService: "default/custom-error-pages-service"
Enter fullscreen mode Exit fullscreen mode

9. Leveraging Lua for Advanced Functionality

ingress-nginx supports Lua scripting for custom logic. Here's an example of using Lua to add custom headers:

controller:
  config:
    load-balance: "ewma"
    lua-shared-dicts: "configuration_data:5M"
  extraVolumeMounts:
    - name: lua-scripts
      mountPath: /etc/nginx/lua
  extraVolumes:
    - name: lua-scripts
      configMap:
        name: lua-scripts
Enter fullscreen mode Exit fullscreen mode

Then, create a ConfigMap with your Lua script:

apiVersion: v1
kind: ConfigMap
metadata:
  name: lua-scripts
data:
  custom_headers.lua: |
    local headers = ngx.req.get_headers()
    ngx.header["X-Custom-Header"] = "ingress-nginx"
Enter fullscreen mode Exit fullscreen mode

10. Implementing Canary Deployments

Use ingress-nginx's built-in canary deployment feature:

annotations:
  nginx.ingress.kubernetes.io/canary: "true"
  nginx.ingress.kubernetes.io/canary-weight: "30"
Enter fullscreen mode Exit fullscreen mode

11. Monitoring and Metrics

Enable Prometheus metrics for comprehensive monitoring:

controller:
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true
Enter fullscreen mode Exit fullscreen mode

Conclusion

Optimizing ingress-nginx is a continuous process that requires understanding your specific traffic patterns and application requirements. The configurations provided here serve as a starting point for high-performance setups, but always test thoroughly in your environment.

Remember, while these optimizations can significantly improve performance, they may also increase resource usage. Monitor your ingress controller closely after implementing changes and be prepared to fine-tune based on real-world results.

By leveraging these advanced techniques, you can ensure your ingress-nginx setup is performant, secure, and ready to handle enterprise-level traffic with ease.


Top comments (2)

Collapse
 
dreama profile image
Dream

Great overview on optimizing ingress-nginx! For HTTP/2 and TLS 1.3, do you have any benchmarks comparing performance before and after enabling them?

Collapse
 
ofir-haim profile image
Ofir Haim • Edited

Thank you for your comment, @dreama

Yes, we do have benchmarks comparing performance before and after enabling HTTP/2 and TLS 1.3. Here are our findings:

  1. The base request to our server with Nginx installed on Docker took 0.190 ms.
  2. With Nginx installed on Kubernetes (K8s) without any customizations, the request time was reduced to 0.130 ms.
  3. After applying our customizations to Nginx, the request time further improved to 0.080 ms.

We hope this helps illustrate the performance enhancements. Let us know if you have any more questions!