My team has been using vagrant for some time to manage our development environment. Vagrant automatically provisions a Linux virtual machine that contains a miniature version of our entire system, plus development tools. We even re-use these scripts when we are deploying the real system. It has been working well, but I recently became frustrated with the resource usage of the underlying VirtualBox VM. Continuous laptop fan.

The obvious solution was to try replacing VirtualBox with a more efficient hypervisor, something that wasn't spending all its time pretending to be an entire PC. Recent versions of OSX contain a hypervisor framework that lets unprivileged, app-store compliant programs manage virtual machines. There are just a few programs that take advantage of this framework, and one of them is Docker.

Docker Desktop for Mac runs an efficient Linux virtual machine inside the hypervisor framework and then runs your containers inside that virtual machine. Vagrant supports docker as a provider as well as virtualbox, vmware, and others. If we can figure out how to make a Docker container that behaves like our old VirtualBox VM, then we should be able to get our environment running in the hypervisor instead.

Are you supposed to do that with Docker?

At this point anyone who's read the docker documentation will ask whether you're supposed to use docker this way. They say change the world by running exactly one application in its own container, perhaps with an orchestration framework. And that would be very cool, but there's no technical reason why you can't run an entire Linux distribution in a container instead, that's what I need, and docker is really easy to install.

What does vagrant require from a container?

Once you've decided to use vagrant with docker, the main obstacle is that you have to provide your own base box. This is a box that is running a ssh server, with a vagrant user authenticated with vagrant's key pair, and passwordless sudo. Vagrant uses a Dockerfile to build this. In a Dockerfile adjacent my Vagrantfile,

RUN useradd vagrant \
  && echo "vagrant" | passwd --stdin vagrant \
  && usermod -a -G wheel vagrant

# allow vagrant to login
RUN cd ~vagrant \
  && mkdir .ssh \
  && echo "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzIw+niNltGEFHzD8+v1I2YJ6oXevct1YeS0o9HZyN1Q9qgCgzUFtdOKLv6IedplqoPkcmF0aYet2PkEDo3MlTBckFXPITAMzF8dJSIFo9D8HfdOV0IAdx4O7PtixWKn5y2hMNG0zQPyUecp4pzC6kivAIhyfHilFR61RGL+GPXQ2MWZWFYbAGjyiYJnAmCP3NOTd0jMZEnDkbUvxhMmBYSdETk1rRgm+R4LOzFUGaHqHDLKLX+FIPKcF96hrucXzcWyLbIbEgE98OHlnVYCzRdK8jlqm8tehUc9c9WhQ== vagrant insecure public key" > .ssh/authorized_keys \
  && chown -R vagrant:vagrant .ssh \
  && chmod 0700 .ssh \
&& chmod 0600 .ssh/authorized_keys \
&& echo "vagrant ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/vagrant_user

# install sudo, sshd, scp
RUN yum -y install sudo openssh-server openssh-clients; systemctl enable sshd.service

EXPOSE 22

CMD ["/usr/sbin/init"]

The other tricky bit is that in order for systemd, the init implementation used in the CentOS Linux distribution, to run inside our container, we have to expose a few filesystems by passing them to the docker run command. In Vagrantfile,

docker.create_args = ['--tmpfs', '/tmp:exec', '--tmpfs', '/run', '-v', '/sys/fs/cgroup:/sys/fs/cgroup:ro']

Vagrant will use these arguments when invoking docker run --tmpfs /tmp:exec ...

This tells Docker to give systemd a read-only view of the cgroup filesystem, and in-memory /tmp and /run directories. (Since /sys is not one of the OSX folders in Docker Desktop's file sharing preferences, it exposes the folder from the Linux box that's really running docker under the hood instead). If you don't expose these directories, systemd will not be able to start any services, including the ssh server.

Once this is all in place, vagrant up should be able to automatically build the Dockerfile, and then run the existing Vagrant provisioning steps by connecting to the container over ssh. From this point on the vagrant workflow is the same as if it was using a VirtualBox provider. Then you can go to town by moving more provisioning steps into the Dockerfile so that rebuilding the vagrant box is more efficient.

I hope this will help someone else to use Docker with Vagrant. A working configuration with this setup and more is at https://github.com/dholth/vagrant-docker/.