Openshift behind a proxy with SSL termination

The Openshift Container Platform is a great tool to deploy your applications and services in a container environment: It hides the complexity of Kubernetes and offers many facilities to manage builds, configurations, volumes and environment variables.

Everything was fine until I had this weird problem where I could not get Keycloak to work properly. I was unable to authenticate because the service kept telling me that it required HTTPS... and I had HTTPS!

In order to track down the problem, I wrote a small service (in Rust :) that just returned the HTTP headers it received and the result was like this:

[other headers]
X-Forwarded-Port: 80
X-Forwarded-Proto: http
[some more headers]

That was strange! The proxy was sending the right X-Forwarded-* headers, but somewhere they were being overwriden!

I searched this behavior and found this anwswer (needs a redhat account to visualize) about changing the openshift router configuration. My solution was a bit different but worked the same.

We need to create a custom router configuration according to the documentation. First let's get a copy of the default configuration:

oc -n default rsh dc/router cat haproxy-config.template > haproxy-config.template

Now open the haproxy-config.template file and search for the following text:

http-request set-header X-Forwarded-Port %[dst_port]
http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Forwarded-Proto-Version h2 if { ssl_fc_alpn -i h2 }

These lines above override the X-Forwarded-* headers sent by the proxy. My solution was to add a conditional to only set the headers if they are not present:

http-request set-header X-Forwarded-Port %[dst_port]  if { req.hdr_cnt(X-Forwarded-Port) eq 0 }
http-request set-header X-Forwarded-Proto http        if { req.hdr_cnt(X-Forwarded-Proto) eq 0 } !{ ssl_fc }
http-request set-header X-Forwarded-Proto https       if { req.hdr_cnt(X-Forwarded-Proto) eq 0 } { ssl_fc }
http-request set-header X-Forwarded-Proto-Version h2  if { req.hdr_cnt(X-Forwarded-Proto-Version) eq 0 } { ssl_fc_alpn -i h2 }

With the file changed, now we can add it to the router:

# create a configmap to hold the file
oc -n default create configmap customrouter --from-file=haproxy-config.template
# add the configmap as a volume
oc -n default set volume dc/router --add --overwrite \
    --name=config-volume \
    --mount-path=/var/lib/haproxy/conf/custom \
    --source='{"configMap": { "name": "customrouter"}}'
# set an environment variable to tell the router were to find the configuration
oc -n default set env dc/router \
    TEMPLATE_FILE=/var/lib/haproxy/conf/custom/haproxy-config.template

After that, Openshift will redeploy the router and the configuration will take effect.