Nginx Ingress Controller Load Balancing and Retry with proxy_next_upstream

By default, Nginx/Nginx Ingress provides sensible default settings. In the real world, there are a few combinations of configurations that you must deliberately change according to different upstreams (your backend apps) behaviour.

Log format

Structured logging is always better if JSONified. This is documented here.

log-format-upstream: '{"ts": "$time_iso8601", "requestID": "$req_id", "upstreamAddr": "$upstream_addr", "proxyUpstreamName": "$proxy_upstream_name", "proxyAlternativeUpstreamName": "$proxy_alternative_upstream_name", "requestMethod": "$request_method", "requestUrl": "$host$uri?$args", "status": $status, "upstreamStatus": "$upstream_status", "requestSize": "$request_length", "responseSize": "$upstream_response_length", "userAgent": "$http_user_agent", "remoteIp": "$remote_addr", "referer": "$http_referer", "latency": "$upstream_response_time"}'

Unfortunately, error logs can not be customised in Nginx.

The power of proxy-next-upstream

In Kubernetes, upstreams are Pod IPs from each Kubernetes Services’ Endpoints in your Ingress configurations. They are watched and fetched periodically from the Kubernetes API. This way, a controller does not need to reload its nginx.conf every time an upstream Pod is restarted or deleted. See Avoiding reloads for reference. This means that Nginx Ingress Controller does not depend on DNS.

Example Scenario

Now, what if a request hits an Nginx Pod upstream that is being restarted or was deleted? Obviously, we expect Nginx to load balance properly to the next available Pod right? Yes, this is how it works.

• A Pod with IP 10.8.9.11 is deleted.
• Nginx tries to connect to this IP until the proxy_connect_timeout.
• Nginx will then look at the proxy_next_upstream settings and see if the error matches any of its value. If there is a match, the request is retried to the next upstream (other IPs from the Kubernetes Service endpoints).

Retry configurations

Gotchas

If you are going to customise proxy-next-upstream-timeout, the value should be greater than the proxy-connect-timeout.

Observation

The first error logs.

2020/01/20 13:20:48 [error] 2224#2224: *133630 connect() failed (111: Connection refused) while connecting to upstream, client: 10.8.9.11 ...
2020/01/20 13:20:48 [error] 2224#2224: *133630 connect() failed (111: Connection refused) while connecting to upstream, client: 10.8.9.11 ...
2020/01/20 13:20:48 [error] 2224#2224: *133630 connect() failed (111: Connection refused) while connecting to upstream, client: 10.8.9.11 ...

... 3 seconds later ...

13:20:56 [error] 2190#2190: *133679 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.8.9.11...

And then finally, Nginx still managed to make the request successful.

You should see something in the logs like.

{
  "upstreamAddr": "10.8.9.11:8080, 10.8.8.2:8080",
  "status": 200,
  "upstreamStatus": "504, 200",
  "latency": "3.001, 0.005"
}

Summary

• The $upstream_addr displays the upstream Pod IPs used during the retry. This means that in a single request context, multiple Pod IPs can be used.
• $upstream_status shows that in the first upstream the response code is 504 which is included in our proxy_next_upstream settings as http_504.
• $upstream_response_time shows the response time for each upstream that was used. The first one is 3 seconds which is also our proxy_connect_timeout duration.

Known Issues

Sometimes, identical IPs are used in the $upstream_addr. See https://github.com/kubernetes/ingress-nginx/issues/4944#issuecomment-575754005.