Running-multi server Dokku: problems and options

This blog post is a collection of resources and thoughts about running applications via Dokku on a High Available (HA) or multi-server setup. Since Dokku doesn't support multi-server out of the box but there are some efforts to make it work, this post is a meant of an overview of options that are out there.

For me, there are two reasons for running applications in a multi-server setup:

1. Most important: being able to apply maintenance updates, update a kernel, and reboot a server without application downtime.
2. Making sure our applications can scale "horizontally" to multiple servers at increased load.

So which problems do we need to solve to scale Dokku deployments horizontally? To figure that out, let's first define the simplest imaginable multi-server Dokku setup.

The simplest imaginable multi-server Dokku setup

A simple imaginable setup would be: one load balancer (or reverse-proxy), and two "backend" servers that run Dokku and run the apps.

In this case, we won't expect Dokku to deal with routing or load balancing logic. That's what a load balancer is for. We simply want Dokku to play nice with having a brother on another server nearby, and we want to make maintaining that as easy as can be.

So what problems do we have to tackle to make this happen?

What Dokku "lacks" for multi-server

The most important thing would be that all app definitions, configuration options, ENV vars, domain names, SSL certificates, etc. are all stored on-server. This means that when running two Dokku servers, all application configuration would have to be defined and kept in-sync on both servers in the "simplest imaginable multi-server Dokku setup".

So we have problem one: keeping configuration in sync

The second problem is moving up domain mapping and SSL termination to the load balancer. Dokku provides awesome mechanisms for mapping domains to apps, installing SSL certificates, or using LetsEncrypt via the dokku-letsencrypt plugin.

However, since traffic will be coming in on the load balancer (or reverse-proxy), it needs to take over the role of recognizing the domain names, owning the SSL certificates and forwarding the rest of the traffic via an internal network or internally encrypted self-signed certificate to the backend servers.

So we have problem two: moving the routing and certificate part of the infrastructure to the load balancer

So what are the options?

Here's a list of options I think might be useful to start thinking about multi-server Dokku setups:

1. (Not really an option) Set up up the two Dokku servers, and keeping them in sync manually. Then putting a load blancer from your cloud provider, or a reverse-proxy like Treafik in front to deal with SSL certificates and routing to the two Dokku servers.
2. Automating maintenance of your Dokku servers via Ansible. There is a new repository live on GitHub where josegonzalez is working on Ansible scripts to maintain Dokku servers. By managing Dokku installs and their application settings via Ansible, it will already make it way easier to keep two Dokku servers and it's app settings "in sync". You will still need to put a load balancer or reverse-proxy in front of the two servers. One bigger challenge here is managing secrets. If you configuring your servers via Ansible, you'll need to set up some kind of secrets vault or other method to inject secrets into the Ansible runs when they update your server and app definitions.
3. Using different tools for setting up your own PaaS on your own server. This article on ServerFault seems to be updated with recent options, but I haven't taken a look at them personally: https://serverfault.com/questions/640038/scaling-out-dokku-infrastructure
4. Wait for Intercity to support multi-server setups. Intercity is the management panel for Dokku that we've built internally at Firmhouse. It's an open source project that you can run yourself on your own server. We're currently working on a feature to keep application settings in-sync across multiple servers.