Auto-route multiple web projects using Traefik

When I first joined a company, they had a method to deploy web projects onto dev and stage servers. However, I spotted critical flaws and inefficiency immediately.

Problem: The scenario was...

1. To manually build Docker image on personal machine, then upload to public Docker Hub! (Yes, you read it right, cooperate secrets leaked.)

2. To manually edit DNS records each time when setting up new projects. It's not a problem for "production" deployment, but it is not the case here. Those are short-lived projects for reviews, and there are many legacy records cluttered up.

3. There are multiple over-specced VMs just to serve static websites, and they had to choose which one to swap with new project each time.

4. Connections are not HTTPS. Although it is technically not a problem, but annoying.

5. They have also complained significant times wasted to push and pull from Docker Hub, and to manually SSH to VMs to spin up projects. (Though, I didn't experience it since this practice was killed by me immediately.)

Solution:

1. I set a wild-card in DNS record to point to a single VM.

2. That VM serves as Docker host.

3. I configured Traefix proxy (a container) to handle routing between projects (containers) and to handle HTTPS connections.

4. I wrote custom GitHub Actions workflow to handle deployment automatically.

5. Repeat above steps for dev and stage VMs.

The Main Topic:

I'll talk about how I setup Docker host and Traefix container in this article, and I have separate article about my custom workflow.

System Architecture

flowchart
  subgraph Local Dev Env
  LD1 --> LD2 --> LD3
  LD1[Developing]
  LD2[Testing]
  LD3[Commit]
  end

  LD3 --> GH1

  subgraph GitHub
  GH1 --> GA1
  GH1[main branch]
    subgraph GitHub Actions
    GA1 --> GA2 --> GA3 --> GA4 --> GA5 --> GA6 --> GA7 --> GA8
    GA1[Activate gcloud service account]
    GA2[Config docker to use Artifacts Registry]
    GA3[Generate static webpages]
    GA4[Build custom nginx docker image]
    GA5[Push to Artifacts Registry]
    GA6[Stop existing docker container on remote host]
    GA7[Copy docker-compose to remote host]
    GA8[Docker-compose up on remote host]
    end
  end

  IA1 -. credential .-> GA1
  GA5 -.-> AR1

  subgraph Google Cloud IAM
  IA1[Actifacts Registry Service Account]
  IA2[Cloud DNS Service Account]
  end

  CD1 -.-> TR1

  subgraph Google Cloud DNS
  CD1 --> CD2
  CD1[some-domain-name.com]
  CD2[Manually add records for *.some-domain-name.com]
  end

  subgraph "Docker Host (on Google Cloud)"
  GA8 --> VM1
  VM1 --> VM2 --> VM3 --> AP1
  VM1[docker-compose up]
  VM2[pull new image from repository]
  VM3[docker run]

    subgraph Traefik
    IA2 -. credential .-> TR2
    TR1 -.-> TR2 -.-> TR3 -.-> TR4 --> TR5
    TR1[Setup reverse proxy]
    TR2["Signing TLS (Let's Encrypt)"]
    TR3[Listen on docker.sock]
    TR4[Dynamic config routing]
    TR5[Dynamic config middlewares]
    end

    subgraph App1
    AP1 --> TR4
    AP1[Nginx web server]
    end

  end

  AR1 -.-> VM2
  subgraph Google Cloud Actifacts Registry
  AR1[Repository]
  end

As aforementioned, we will have:

1. An Ubuntu VM to run Docker engine.

2. Configure Traefik proxy (container) to handle routing between other containers, and to handle HTTPS connections.

3. Google Artifact Registry to store private Docker images.

4. Configure GitHub Actions workflow to glue all parts together.

5. Additionally, we will need a SSH key pair to connect to VM, and IAM credentials to access gcloud services.

Configure Traefik Proxy

Assuming we have Ubuntu VM with Docker and Docker Compose installed, and both Cloud Artifacts Registry and IAM setup properly. (Let me know in the comment if anyone wish for extra walkthrough)

Within the docker-compose.yml for the Traefik container, we will add:

version: "3.8"

services:
  traefik:
    # You may want to specify a version.
    image: "traefik:latest"

    # Name to your own liking.
    container_name: "traefik-proxy"

    command:
      - "--entrypoints.web.address=:80"
      - "--providers.docker"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=proxy"
      - "--api"

    ports:
      - "80:80"

    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

    labels:
      - "traefik.enable=true"
      - "traefik.port=80"
      - "traefik.http.routers.traefik.rule=Host(`traefik.some-domain-name.com`)"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
      - "traefik.http.routers.http-catchall.entrypoints=web"

    networks:
      - "proxy"

networks:
  proxy:
    driver: "bridge"
    name: "proxy"

Before you start the container, you should do the following once:

1. Create a bridge network: sh docker network create proxy

This configuration allows Traefik to setup new url whenever new docker container spins up, over HTTP for now.

For example, your VM is some-domain-name.com and a new container configured with hostname of aaa.some-domain-name.com. Traefik will configure the route of aaa.some-domain-name.com to the container automatically.

Configure Traefik to redirect to HTTPS

We will configure Traefik to renew certificate with Let's Encrypt by using DNS challenge. Merge following lines into docker-compose.yml:

services:
  traefik:
    # For simplicity, let's ignore aforementioned lines.

    command:
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.leresolver.acme.dnschallenge=true"
      - "--certificatesresolvers.leresolver.acme.dnschallenge.provider=gcloud"
      - "--certificatesresolvers.leresolver.acme.email=your-registered@email.com"
      - "--certificatesresolvers.leresolver.acme.storage=/acme.json"

    environment:
      - "GCE_PROJECT=your_GCP_project_id"
      - "GCE_SERVICE_ACCOUNT_FILE=lets-encrypt.json"

    ports:
      - "443:443"

    volumes:
      - "./acme.json:/acme.json"
      - "./lets-encrypt.json:/lets-encrypt.json"

    labels:
      - "traefik.http.routers.traefik.tls.certresolver=leresolver"
      - "traefik.http.routers.traefik.tls.domains[0].main=*.some-domain-name.com"
      - "traefik.http.routers.traefik.tls.domains[0].sans=some-domain-name.com"
      - "traefik.http.routers.traefik.entrypoints=websecure"
      - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
      - "traefik.http.routers.http-catchall.entrypoints=web"
      - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
      - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"

Please notes that the example shown is using Google Cloud DNS, you may have to set to a different challenger and different environment variables. Follow this doc for gcloud challenger, or follow this index for specific instruction of your DNS providers.

Before you start the container, you should do the following once:

1. Create empty ACME file for Traefik to store Let's Encrypt info:

   touch acme.json
   chmod 600 acme.json
   

2. Download the Google Cloud Artifacts Registry service account credentials (json file), and rename it to lets-encrypt.json. (or do the reverse in docker-compose.yml.)

Kindly note that both acme.json and lets-encrypt.json should be within the same folder of docker-compose.yml.

🎉 Walla! You should have working Traefik proxy running and should have valid certificate to serve HTTPS. (Actually it's not that easy, please refer to official docs for troubleshooting; or, leave comments below, if I happened to pickup, no guaranty.

I hope this article is inspiring for you, and maybe, assist you to learn, or setup, your own Traefik proxy. I found it quite a deep learn curve initially, as there aren't much example configs that suites my scenario.

Configure other projects to use this Traefik proxy:

As it is quite lengthly, please follow my second article for custom GitHub Actions workflow and docker-compose.yml.

Feedbacks: from colleague,

1. "They said" it was a hour of "labour" work to deploy a version of project before, but I made it done under 3 minutes, and automatically.

2. I cut cloud expenses by half.