I've recently started at a new job, and I'm fortunate enough to have a new M1 Macbook Air. I'd heard from the wider community that everything "just works" on the M1, even though it's not an Intel chip. "Even Docker" they said. "Wow" I thought, this will be seamless, incredible.

So I sit there on my first day ready to get started. NPM installs everything fine, my python packages all install and both the backend and frontend are running, too easy! Then I hit an issue. This application requires a service which is run using Vagrant on Virtualbox.

Virtualbox very much doesn't support Apple Silicon. It's unclear whether Virtualbox will ever support Apple Silicon. The preferred alternative is a product by VMWare which also doesn't (yet) support Apple Silicon. I'm in a bit of a pickle.

But! Never fear! I can see that Vagrant also supports Docker. So over the next three days I dive into documentation, I frantically search the web for "vagrant docker M1" and "vagrant docker config" and "vagrant docker network issue" and "vagrant docker systemctl". After shaving at least three seperate yaks, and configuring a monstrosity I got it all running! Here's some lessons I learned.

Disclaimer: I am very much not a linux / ops / devops person. My experience with linux is limited but there's not much help out there for this issue so I'm doing my best.

Docker and Vagrant are frenemies

Docker and Vagrant have very different philosophies about how you should run your development setup. Docker's ideal is a minimalist setup, with just enough to run the single process you need. If you want more processes, Docker wants you to create more containers. Vagrant prefers a maximalist style, install everything on the one virtual machine and get it all going together.

To get Docker going you'll need a Dockerfile this describes the setup needed to create the Docker container. For our Docker container to be friends with Vagrant we're going to configure it as if its a traditional linux machine. By default you'll need at least sshd (which lets you SSH into the machine). I also needed systemd (which runs services). This is very much not how you should do Docker normally.

Here's the Dockerfile I used:

# Docker image to use with Vagrant
# Aims to be as similar to normal Vagrant usage as possible
# Adds Puppet, SSH daemon, Systemd
# Adapted from https://github.com/BashtonLtd/docker-vagrant-images/blob/master/ubuntu1404/Dockerfile

FROM ubuntu:18.04
ENV container docker
RUN apt-get update -y && apt-get dist-upgrade -y

# Install system dependencies, you may not need all of these
RUN apt-get install -y --no-install-recommends ssh sudo libffi-dev systemd openssh-client

# Needed to run systemd
# VOLUME [ "/sys/fs/cgroup" ]
# Doesn't appear to be necessary? See comments

RUN apt-get -y install puppet

# Add vagrant user and key for SSH
RUN useradd --create-home -s /bin/bash vagrant
RUN echo -n 'vagrant:vagrant' | chpasswd
RUN echo 'vagrant ALL = NOPASSWD: ALL' > /etc/sudoers.d/vagrant
RUN chmod 440 /etc/sudoers.d/vagrant
RUN mkdir -p /home/vagrant/.ssh
RUN chmod 700 /home/vagrant/.ssh
RUN echo "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzIw+niNltGEFHzD8+v1I2YJ6oXevct1YeS0o9HZyN1Q9qgCgzUFtdOKLv6IedplqoPkcmF0aYet2PkEDo3MlTBckFXPITAMzF8dJSIFo9D8HfdOV0IAdx4O7PtixWKn5y2hMNG0zQPyUecp4pzC6kivAIhyfHilFR61RGL+GPXQ2MWZWFYbAGjyiYJnAmCP3NOTd0jMZEnDkbUvxhMmBYSdETk1rRgm+R4LOzFUGaHqHDLKLX+FIPKcF96hrucXzcWyLbIbEgE98OHlnVYCzRdK8jlqm8tehUc9c9WhQ==" > /home/vagrant/.ssh/authorized_keys
RUN chmod 600 /home/vagrant/.ssh/authorized_keys
RUN chown -R vagrant:vagrant /home/vagrant/.ssh
RUN sed -i -e 's/Defaults.*requiretty/#&/' /etc/sudoers
RUN sed -i -e 's/\(UsePAM \)yes/\1 no/' /etc/ssh/sshd_config

# Start SSH
RUN mkdir /var/run/sshd
EXPOSE 22
RUN /usr/sbin/sshd

# Start Systemd (systemctl)
CMD ["/lib/systemd/systemd"]

If you don't need systemd in your setup then you can remove that and change the RUN of sshd to:

CMD ["/usr/sbin/sshd", "-D"]

All of this sets up a Docker container which doesn't work like a regular Docker container. It runs more like a virtual machine. This means it will be difficult to manage using the normal Docker commands. Once you take it down it will also be difficult to get up again. Best to control it with Vagrant. I've had to delete the container and re-create it with Vagrant many times.

Vagrant is pretty easy to configure

To get Vagrant running using your Dockerfile you just need to add a little bit of configuration:

Vagrant.configure(2) do |config|

# ... your existing config

  # Custom configuration for docker
  config.vm.provider "docker" do |docker, override|
    # docker doesnt use boxes
    override.vm.box = nil

    # this is where your Dockerfile lives
    docker.build_dir = "."

    # Make sure it sets up ssh with the Dockerfile
    # Vagrant is pretty dependent on ssh
    override.ssh.insert_key = true
    docker.has_ssh = true

    # Configure Docker to allow access to more resources
    docker.privileged = true
  end

# ...

end

You may need to override some other settings here. The override variable allows you to change top-level settings, while the docker variable allows you to setup docker-specific stuff. Have a read of the vagrant docker provider and vagrant docker provisioning documentation.

Once you've set this up, you can run your Vagrant machine with:

$ vagrant up --provider=docker

And if you need to do extra things inside the machine, you can ssh in with:

$ vagrant ssh

You should have sudo access as well if you need it.

Vagrant doesn't like to run multiple boxes

When I first ran Vagrant, before I realised Virtualbox wouldn't work it started and then failed. Then when I tried to run vagrant with the Docker provider it gave me errors about not having multiple boxes. I also couldn't delete my box because virtualbox wasn't running.

You can just delete the files in .vagrant to resolve that. It seems like Vagrant doesn't have a good way of resolving this itself.

You can re-provision Vagrant

Sometimes I found my setup would break, in my case it wasn't too hard to just run the Vagrant provisioning script again.

$ vagrant up --provision

Once that was complete some of my issues got resolved. It's not a good idea to blindly do this, but in my case I'm just trying to get it running.

Docker's public network bridge feature is broken

It doesn't seem to work on Mac! So if your Vagrant config has something like:

config.vm.network "public_network"

Try to see if you can replace it with:

config.vm.network "private_network", type: "dhcp"

In my case this substitution was sufficient, but you may have other requirements.

Your linux binaries need to be arm64

I ran into some issues because our provisioning scripts assumed that the architecture of the system would be amd64 (Intel and AMD 64-bit processors). It was a pretty safe bet that the CPU was amd64 up until Apple introduced their M1 chip!

Running linux on the M1 chip uses the arm64 architecture instead. As a handy shortcut you can get the architecture of the current machine in linux with:

$ dpkg --print-architecture

I used this in shell scripts by setting a variable:

readonly ARCH=`dpkg --print-architecture`

And then substituting that in when binaries were downloaded and installed, usually that was relatively simple, e.g.:

wget http://somedomain.com/some_binary_linux_amd64.zip

Became

wget "http://somedomain.com/some_binary_linux_${ARCH}.zip"

I learned a lot - but I still don't know much

I learned a lot about Docker (and how not to do things) Vagrant (and how to hack it) and some various linux tricks trying to get this all working. However I'm very much not a linux, Docker or Vagrant person, I'm just trying to make things work. So if I've written anything here horribly wrong, or extremely misguided, please throw down a comment.

If this helped you I'd love to hear it! I got stuck googling around for how to do this, and I imagine there are other people out there in the same boat.