I started a new job 6 weeks ago and I'm still learning the environment. I got a new task yesterday and it was an issue which had been bounced around for a while. The issue seemed simple enough, enable brotli on a virtual host. This is the story of how simple may not be so simple.

The first issue: this is a Centos 6 host.

Centos 6 is end of life as of November 2020. After a bit of discussion, the plan is to continue to investigate if we can do this as a stopgap until the entire solution can be reworked.

So, how would one investigate how to do this? It turns out, it isn't so trivial, and even just starting an investigate can be tricky. My first thought was, start a docker instance so I can poke around and try various RPMs.

docker pull centos:6.7
Great start!
docker run -ti centos:6.7

Cool, I've got a prompt, now what?

After searching the web, I see that nginx.org has rpm packages. I'll try one and see if it has brotli support.

curl -LO https://nginx.org/packages/mainline/centos/6/x86_64/RPMS/nginx-1.19.5-1.el6.ngx.x86_64.rpm
curl: (35) SSL connect error


WAT? ok, maybe the certs on this 2yo docker image (I had checked docker hub) are out of date. curl -k the same thing and get the same error. WTF? add the -v to curl and see the libnss message * warning: ignoring value of ssl.verifyhost Well ain't that something? At this point I mumble under my breath about how debian/ubuntu uses openssl linked curl by default and not libnss and I wonder if it would have behaved the same.

As I write this, I realize that maybe I should have used a more recent Centos 6 docker image, 6.10 perhaps. Unfortunately I'm not as experienced with Centos as I should be. Part of this fun is diving in and learning. I used 6.7 because that is what this server under question says it is.

Alright, so new problem...

How to update certificates on a 2yo Centos 6.7 docker image?

yum update says It can't do its thing.

$ yum update
Loaded plugins: fastestmirror
Setting up Update Process
Loading mirror speeds from cached hostfile
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
 Eg. Invalid release/repo/arch combination/
removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/6/base/mirrorlist.txt
Error: Cannot find a valid baseurl for repo: base

Alright, some searching tells me to use baseurl instead of mirrorlist and that it is now at vault.centos.org. I try that and get a new error: http://vault.centos.org/centos/6/os/x86_64/repodata/repomd.xml: [Errno 14] Peer cert cannot be verified or peer cert invalid Yes, that says http:, but port 80 just redirects to port 443 and then we get the cert error. We have a catch-22. We need to upgrade our package to get new certs but we need new certs to upgrade our package. We need to upgrade our package to get new certs but we need new certs to upgrade our package. We need to upgrade our package to get new certs but we need new certs to upgrade our package. We need to upgrade our package to get new certs but we need new certs to upgrade our package. cough ouch

(Update: I'm doing this often enough that editing that file takes too long and so there is this: sed -i 's/#baseurl=http:\/\/mirror/baseurl=http:\/\/vault/;s/mirrorlist/#mirrorlist/' /etc/yum.repos.d/CentOS-Base.repo )

For a solution, I got lucky. I knew to try to get the ca-certificates rpm manually and install it without yum. Worse come to worse I could download it on another host and copy it with docker cp or use netcat or python simple server or put it on a non-TLS webserver or any other method of moving data around. I was manually browsing the repo and got this URL curl -LO https://archive.kernel.org/centos-vault/6.10/updates/x86_64/Packages/ca-certificates-2020.2.41-65.1.el6_10.noarch.rpm and it turns out that the cert serving archive.kernel.org was acceptable.

rpm -U ./ca-cert*.rpm

And now I can yum update ; yum upgrade

Now we can look for nginx with brotli

Search search search, search the web and you will find no clear answers. There are posts about compiling it yourself. There are posts about using a subscription repo. (https://www.getpagespeed.com)

Incidentally, that subscription repo also had blog post which helped confirm my findings above: https://www.getpagespeed.com/server-setup/how-to-fix-yum-after-centos-6-went-eol along with https://forums.centos.org/viewtopic.php?f=13&t=78238 which verified what I had guessed about updating certificates.

Now my first thought was, nginx has this by default now, right? Use the newest package from nginx.org. Look at all of 'em here: https://nginx.org/packages/centos/6/x86_64/RPMS/

I curl -LOed (curllo is a verb ya see) that https://nginx.org/packages/mainline/centos/6/x86_64/RPMS/nginx-1.19.5-1.el6.ngx.x86_64.rpm and installed it (rpm -i ./nginx*rpm) and was disappointed that it had no brotli (strings /usr/sbin/nginx | grep brot confirms). (That is after I yum install initscripts as required by that package)

Well bummer.

Keep searching and reading...

The next thing I tried were the packages from https://repo.aerisnetwork.com but, in short they didn't do what I wanted or had some drawback.

Next, I tried poking at the packages from getpagespeed.com, I even browsed in my browser, got this URL: https://extras.getpagespeed.com/redhat/6/mainline/x86_64/RPMS/nginx-1.21.3-1.el6.ngx.x86_64.rpm and tried to download and install it, but it didn't actually download. Registration is required and apparently they whitelist registered users by IP. Jumping through those hoops would not be sustainable for me in my work environment so I discarded getpagespeed.com.

I have no idea why, but next I tried this https://nginx.org/packages/centos/6/x86_64/RPMS/nginx-1.18.0-2.el6.ngx.x86_64.rpm Older than the pervious nginx.org package, but it isn't in the mainline repo and I'm Centos n00b enough that I don't know the difference. Again, strings /usr/sbin/nginx|grep brot showed nothing, so that got a quick rpm -e nginx.

Next, I tried curl -LO https://repo.aerisnetwork.com/archive/nginx-more-1.13.6-1.el6.x86_64.rpm and https://repo.aerisnetwork.com/archive/nginx-more-1.14.2-4.el6.x86_64.rpm. (No idea why I grabbed 1.13 and 1.14.) There were a bunch of various nginx packages on repo.aerisnetwork.com For this to work there were some requirements so I ran yum install gd libxslt but there was still the case of libmaxminddb.so.0

Next, I tried repo.codeit.guru packages. I don't recall exactly where I found this repo. I think it was multiple sources. One of them was https://nixcp.com/brotli-compression-nginx/. I'd probably also seen https://codeit.guru/en_US/2020/04/nginx-1-18-0-stable-with-brotli-support-tls-1-3-final-rfc-8446-built-against-openssl-1-1-1g-for-red-hat-enterprise-linux-and-centos/. I was desperate enough to copy and page the root repo URL and hope for a Centos 6 directory. There was one!

curl -LO https://repo.codeit.guru/packages/mainline/centos/6/x86_64/nginx-1.19.5-1.el6.codeit.x86_64.rpm
rpm -i ./nginx-1.19.5-1.el6.codeit.x86_64.rpm

warning: ./nginx-1.19.5-1.el6.codeit.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 898b43f4: NOKEY
error: Failed dependencies:
    libbrotli = 1:1.0.7 is needed by nginx-1:1.19.5-1.el6.codeit.x86_64
    libmaxminddb.so.0()(64bit) is needed by nginx-1:1.19.5-1.el6.codeit.x86_64

curl -LO https://repo.codeit.guru/packages/mainline/centos/6/x86_64/libbrotli-1.0.7-1.codeit.el6.x86_64.rpm
rpm -i ./libbrotli-1.0.7-1.codeit.el6.x86_64.rpm 

There is obviously brotli support but also the same libmaxminddb.so.0 dependency.

libmaxminddb on Centos 6

Now I had to briefly forget nginx and focussing on where to find this dependency. Surely there is a package?

$ yum install libmaxmindddb
Loaded plugins: fastestmirror
Setting up Install Process
Loading mirror speeds from cached hostfile
No package libmaxmindddb available.
Error: Nothing to do
$ yum install libmaxmindddb-dev
...
No package libmaxmindddb-dev available.
$ yum install libmaxmindddb-devel
...
No package libmaxmindddb-devel available.

It required much searching. I wondered if it was in the nginx-module-geoip rpm, but no.

Finally, I found some comments about epel. I didn't have epel repo enabled. It is not part of vault AFAICT, but I was able to web browse epel and find a package.

curl -LO https://archives.fedoraproject.org/pub/archive/epel/6/x86_64/Packages/l/libmaxminddb-1.1.1-5.el6.x86_64.rpm
rpm -i ./libmaxminddb-1.1.1-5.el6.x86_64.rpm 

And now nginx package from codeit.guru can be installed.

$ rpm -i ./nginx-1.19.5-1.el6.codeit.x86_64.rpm 
warning: ./nginx-1.19.5-1.el6.codeit.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 898b43f4: NOKEY
---------------------------------------------------------------------------

Thanks for using nginx!

Please find the official documentation for nginx here:
* http://nginx.org/en/docs/

Please subscribe to nginx-announce mailing list to get
the most important news about nginx:
* http://nginx.org/en/support.html

Commercial subscriptions for nginx are available on:
* http://nginx.com/products/

---------------------------------------------------------------------------
$ rpm -ql nginx
/etc/logrotate.d/nginx
/etc/nginx
/etc/nginx/conf.d
/etc/nginx/conf.d/default-ssl.conf.example
/etc/nginx/conf.d/default.conf
/etc/nginx/fastcgi_params
/etc/nginx/koi-utf
/etc/nginx/koi-win
/etc/nginx/mime.types
/etc/nginx/modules
/etc/nginx/nginx.conf
/etc/nginx/php.inc
/etc/nginx/scgi_params
/etc/nginx/uwsgi_params
/etc/nginx/win-utf
/etc/rc.d/init.d/nginx
/etc/rc.d/init.d/nginx-debug
/etc/sysconfig/nginx
/etc/sysconfig/nginx-debug
/usr/lib64/nginx
/usr/lib64/nginx/modules
/usr/sbin/nginx
/usr/sbin/nginx-debug
/usr/share/doc/nginx-1.19.5
/usr/share/doc/nginx-1.19.5/COPYRIGHT
/usr/share/man/man8/nginx.8.gz
/usr/share/nginx
/usr/share/nginx/html
/usr/share/nginx/html/50x.html
/usr/share/nginx/html/index.html
/var/cache/nginx
/var/log/nginx
$ strings /usr/sbin/nginx | grep brot
ngx_http_brotli_static_module_ctx
ngx_http_brotli_filter_module
ngx_http_brotli_static_module
...

Look at those beautiful symbols!

Next task is to discuss with the team all of the reasons that we should NOT integrate this into our environment and instead migrate off of Centos 6 instead.