ECS Blue/Green deployment with CodeDeploy and Terraform

In this article you will find a helpful step by step guide on how to setup Blue/Green deployment with AWS CodeDeploy for ECS Fargate using Terraform and the ways of working around common challenges of doing this through the Terraform.

Note: if you want to skip and get to the solution then please follow this link to my repository

Assumptions:

1. You already know how to use Terraform
2. You already know what is AWS ECS and how to create ECS service in Fargate
3. You already know what is AWS ALB and Target Groups
4. You already have idea what is Blue/Green deployment strategy if no then check this page

So what are the challenges of doing this with Terraform? Before answering this question let me briefly go through the Blue/Green deployment process in ECS when it's performed by CodeDeploy.

This is what you should have in place before starting deployment:

1. ECS cluster, ECS task definition
2. In that cluster ECS service running behind Load Balancer
3. In that Load Balancer - ALB Listener rule (Production Listener for example with port 443) that forwards traffic to the Target group(Blue target group) where ECS service tasks are registered.
4. A second Listener (Test traffic listener for example with port 8443) that points to the Green Target group, there are no ECS service tasks registered in this group at the moment.
5. CodeDeploy app and deployment group - you can find my example here

Now to perform B/G deployment you need to:

1. Create new ECS task revision
2. Update appspec.yaml file with New Task arn and ALB information
3. Create a new deployment in CodeDeploy with provided appspec.yaml

This is what CodeDeploy will do

• Creates new tasks with the new version of the image, adds them into Green Target Group. Now you can access new version of the app through the Test Listener port

• Once Confirmed that new tasks are up and running (or all pre-hook tests are passed) CodeDeploy shifts traffic in LoadBalancer - Pointing Production Listener now to the Green Target Group

• Once traffic is shifted CodeDeploy will (after configured time) decommission tasks running the old version of the app

So the challenges of doing this through the terraform:

• CodeDeploy makes changes in ALB (shifts the traffic between target groups) outside of Terraform. If we run terraform apply it will rewrite all changes made my CodeDeploy

To workaround this add lifecycle { ignore_changes = ...} in ALB

resource "aws_lb_listener_rule" "example"{
...
 action {
  type = "forward"
  target_group_arn = aws_lb_target_group.example.arn
 }
# because CodeDeploy will switch target groups during the B/G deployment
 lifecycle {
   ignore_changes = [action] 
 }
}

• After deployment is done CodeDeploy changes task version in ECS service and updates target group information outside of Terraform

To workaround this add lifecycle { ignore_changes = ...} in ECS

resource "aws_ecs_service" "example"{
...
 task_definition = aws_ecs_task_definition.example.arn

 load_balancer {
  container_name = "example"
  container_port = 8080
  target_group_arn = aws_lb_target_group.example.arn
 }
# because CodeDeploy will handle task definition and alb changes outside of terraform
 lifecycle {
   ignore_changes = [load_balancer, task_definition]
 }
}

• To create deployment in CodeDeploy we need execute CLI
• To update the appspec.yaml file we need to know the ARN of the new task revision. On different blog posts I saw examples of creating new task revision using CLI - but what if we want to manage ECS task through Terraform

To do this through Terraform use local_file and local-exec to update appspec.yaml and execute CLI to create deployment in CodeDeploy

In the code below we are creating content of the appspec.yaml file and then executing CLI commands to start the deployment and wait until it's finished.

locals {

  # appspec file  
  appspec = {
    version = "0.0"
    Resources = [
      {
        TargetService = {
          Type = "AWS::ECS::Service"
          Properties = {
            TaskDefinition = var.ecs_task_def_arn
            LoadBalancerInfo = {
              ContainerName = var.container_name
              ContainerPort = var.container_port
            }
          }
        }
      }
    ]
  }
  appspec_content = replace(jsonencode(local.appspec), "\"", "\\\"")
  appspec_sha256  = sha256(jsonencode(local.appspec))

  # create deployment bash script
  script = <<EOF
#!/bin/bash

echo "creating deployment ..."
ID=$(aws deploy create-deployment \
    --application-name ${var.codedeploy_application_name} \
    --deployment-group-name ${var.deployment_group_name} \
    --revision '{"revisionType": "AppSpecContent", "appSpecContent": {"content": "${local.appspec_content}", "sha256": "${local.appspec_sha256}"}}' \
    --output text \
    --query '[deploymentId]')

echo "======================================================="
echo "waiting for deployment $deploymentId to finish ..."
STATUS=$(aws deploy get-deployment \
    --deployment-id $ID \
    --output text \
    --query '[deploymentInfo.status]')

while [[ $STATUS == "Created" || $STATUS == "InProgress" || $STATUS == "Pending" || $STATUS == "Queued" || $STATUS == "Ready" ]]; do
    echo "Status: $STATUS..."
    STATUS=$(aws deploy get-deployment \
        --deployment-id $ID \
        --output text \
        --query '[deploymentInfo.status]')

    SLEEP_TIME=30

    echo "Sleeping for: $SLEEP_TIME Seconds"
    sleep $SLEEP_TIME
done

if [[ $STATUS == "Succeeded" ]]; then
    echo "Deployment succeeded."
else
    echo "Deployment failed!"
    exit 1
fi

EOF

}

resource "local_file" "deploy_script" {
  filename             = "${path.module}/deploy_script.txt"
  directory_permission = "0755"
  file_permission      = "0644"
  content              = local.script

  depends_on = [ 
    aws_codedeploy_app.this,
    aws_codedeploy_deployment_group.this,
  ]
}

resource "null_resource" "start_deploy" {
  triggers = {
    appspec_sha256 = local.appspec_sha256 # run only if appspec file changed
  }

  provisioner "local-exec" {
    command     = local.script
    interpreter = ["/bin/bash", "-c"]
  }

  depends_on = [ 
    aws_codedeploy_app.this,
    aws_codedeploy_deployment_group.this,
  ]
}