ECS Anywhere & Traefik Proxy with ECS Compose-X

Original post can be found here along with the technical resources

TL;DR

Using ECS Compose-X, deploy Traefik Proxy on-premise with AWS ECS Anywhere with only a few changes from running on AWS EC2 or AWS Fargate.

Introduction

Our tools for today's lab

ECS Compose-X is an open-source project that allows you to use docker-compose services definitions, and render CFN templates (just like with AWS CDK, but without having to write code) to deploy your application service stacks.

Traefik Proxy is an open source project that will allow you to define ingress rules for your applications and will automatically route traffic to your backend services based on various rules. It is also capable of doing Service Discovery, and today we are going to look at the ECS & ECS Anywhere discovery providers.

AWS ECS Anywhere is an extension to AWS ECS, which is a managed container orchestration service, that now allows you to run your workloads in your datacenter/on-premise, and really just, anywhere!

The objective

When running on AWS, we have access to services such as AWS Certificates Manager (ACM), AWS Load Balancing (manages ELB, ALB, NLB and more), which can offload a lot of complexity and is very feature rich.

However, coming to on-premise environments, the costs for hardware that would give us the same functionalities (think F5 load-balancers, your expensive licensed VXLAN resources), are only affordable by a few. And typically for a "home-labber" such as myself, way out of my budget.

So I needed an alternative solution that would allow me to use AWS ECS services, route traffic to my services based on service discovery. It should also be able to deal with managing SSL certificates for me. And finally, I must be able to deal with non-persistent storage.

Welcome Traefik Proxy

For years, I have been an NGINX and/or HA Proxy user. They are very lightweight, very popular, great documentation and community support in general.

But, they aren't quite capable of doing Service discovery all by themselves.

I came across Traefik Proxy, and a whole new world of capabilities was now wide open. With service discovery providers, Traefik can scrape your services and based on labels/tags, identify instructions to perform. And AWS ECS is one of such providers.

Just a tiny little problem

When I first tried Traefik a little over a year ago for ECS Anywhere, it wouldn't work. That's because until then, Traefik only considered using Fargate or EC2 instances to run the containers. There was no implementation of discovering AWS ECS Anywhere on-prem instances.

This has been since addressed, and one can specifically enable the ECS Anywhere discovery in Traefik.

Traefik and Let's Encrypt SSL management

When you define routers with Let's Encrypt, you can define whether or not you want Traefik to provision certificates.

With Traefik, you can automatically get new certificates for yourself when you need them. There are different validation methods, my chosen one being with DNS validation.

For validation, given my DNS domain is managed in Route53, I simply indicate to Traefik to use that DNS method / zone for validation.

Why DNS validation works for me?

If I have internally exposed services (not available on the internet), but I still want to have SSL certificates provisioned for them, DNS is the only option for that. It will generally come down to your preference.

Deployment

Prerequisites

You will need

• An AWS Account
• Configured a local user with IAM permissions to deploy resources
• Have an existing ECS Cluster with a registered ECS Instance that runs on-premise.
• Installed ECS Compose-X (version 0.22 and above). See below.

Compose-X install

You can install it locally for your user

pip install pip -U;
pip install --user ecs-composex

or install it in a python virtual/isolated enviroment

python3 -m venv compose-x
source compose-x/bin/activate
pip install pip -U
pip install ecs-composex

Once you have installed it, run the following command that will ensure we have the necessary settings and resources to get started.

ecs-compose-x init

Clone the labs repo

Clone the repo, and head to the configuration files.

git clone https://github.com/compose-x/compose-x-labs.git
cd traefik/part_2/

In the current files, you will have to edit to change the domain name in-use.

You can either edit it with your preferred IDE, or simply run

sed -i 's/bdd-testing.compose-x.io/<your_domain_name.tld>/g'

If your domain is not maintained in AWS Route53, you will need to head over to the Let's Encrypt ACME documentation in order to use a different validation method.

Getting ready to deploy

The deployment to ECS Anywhere is only a command away

CLUSTER_NAME=MyExistingECSCLuster ecs-compose-x up \
-n traefik-anywhere \
-d templates \
-f docker-compose.yaml \
-f ecs-anywhere.yaml

Compose-X will render all of the CFN templates and store them in your local folder (under templates), as well as in AWS S3. It is required to be in S3 for CFN nested stacks.

After a few minutes, you should have running on your ECS Anywhere instances, Traefik.

Adding SSL Certificates backup.

Let's Encrypt "production" endpoint, has a rate limit in place for the number of certificates requested per domain.

So if you are new to this, we recommend to use the Let's Encrypt staging environment, which will allow not to hit the rate limit.

Sadly, it seems that the persistent storage of the file that holds the SSL certificates requested by Traefik to Let's Encrypt is not a feature that we might see coming in any time soon.

So instead, we are going to implement the backup-and restore ourselves.

Using 2 sidecars, one to restore the files prior to traefik starting, and another constantly watching for a change to automatically backup the file to AWS S3, we will ensure that we don't request certificates we already did provision before.

To deploy the solution, we added the backup.yaml file to our deployment command.

Note: the S3 bucket already exists for us, and if you want to use an existing one, you will need to adopt the Lookup Tags in order to use your own/the right bucket.

So now, we deploy our updated definition to AWS

CLUSTER_NAME=MyExistingECSCLuster ecs-compose-x up \
-n traefik-anywhere \
-d templates \
-f docker-compose.yaml \
-f ecs-anywhere.yaml
-f backup.yaml

Hint: the order of the files does matter.

And that's it! You now have successfully deployed Traefik to ECS Anywhere, with automated backup & restore for your certificates.

To add additional services you wish Traefik to route to, simply deploy them with the appropriate labels, just like we used in the demo for the whoami service