Problem
I'm running Nginx Ingress Controller deployed on GKE and I want to structure my web server logs so I can create metrics and alerts in Google Cloud Stackdriver.
References
- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
- https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/log-format/
Solution
If you are using GKE, there is already a Fluentd DaemonSet that forwards all container logs to Stackdriver.
All you need to do is ensure that Nginx Ingress Controller's ConfigMap has the following log-format-upstream settings:
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-ingress-controller
labels:
app: nginx-ingress
data:
log-format-escape-json: "true"
log-format-upstream: '{"timestamp": "$time_iso8601", "requestID": "$req_id", "proxyUpstreamName":
"$proxy_upstream_name", "proxyAlternativeUpstreamName": "$proxy_alternative_upstream_name","upstreamStatus":
"$upstream_status", "upstreamAddr": "$upstream_addr","httpRequest":{"requestMethod":
"$request_method", "requestUrl": "$host$request_uri", "status": $status,"requestSize":
"$request_length", "responseSize": "$upstream_response_length", "userAgent": "$http_user_agent",
"remoteIp": "$remote_addr", "referer": "$http_referer", "latency": "$upstream_response_time s",
"protocol":"$server_protocol"}}'
If you are deploying Nginx using the Helm Chart, the values file will look something like this:
controller:
config:
log-format-escape-json: "true"
log-format-upstream: '{"timestamp": "$time_iso8601", "requestID": "$req_id", "proxyUpstreamName":
"$proxy_upstream_name", "proxyAlternativeUpstreamName": "$proxy_alternative_upstream_name","upstreamStatus":
"$upstream_status", "upstreamAddr": "$upstream_addr","httpRequest":{"requestMethod":
"$request_method", "requestUrl": "$host$request_uri", "status": $status,"requestSize":
"$request_length", "responseSize": "$upstream_response_length", "userAgent": "$http_user_agent",
"remoteIp": "$remote_addr", "referer": "$http_referer", "latency": "$upstream_response_time s",
"protocol":"$server_protocol"}}'
Top comments (4)
Is this properly indexed by Stackdriver? From reading the documentation, it seems that Stackdriver wants the HTTP info nested in the
httpRequest
key, and IIRC, JSON logs end up in thejsonPayload
key for k8s containers in Stackdriver. Or is Stackdriver smart enough to figure out it should be in thehttpRequest
key?UPDATE:
So I think I figured this out, the JSON should be nested in an
httpRequest
key. Here's the working version:I do not see the difference ... is the original post also updated?
Thanks for pointing this out.
Here's another example JSON format I found useful:
journal.petrausch.info/post/2020/0...