Building an EKS Fargate cluster with Terraform

Fargate is service by AWS to run serverless workloads in Kubernetes. With Fargate you do not have to manage VMs as cluster nodes yourself as each of the pods are provisioned as nodes by Fargate itself. It is different from Lambda in the sense that you're still self-managing the Kubernetes cluster or the runtime for all the workloads you run in that cluster. Having said that, I believe it's more suitable for teams that are running containerised microservices and want to do away with managing Kubernetes infrastructure themselves.

While Lambda prices on a combination of requests, CPU and memory; Fargate pricing is just the CPU and Memory of the nodes running in the cluster in addition to a fixed monthly cost of the Fargate service itself. If you want to go serverless without Vendor lock-in, Fargate is a good option. Hence we at Digité prefer running our microservices in the Fargate model.

Managing such an infrastructure is certainly not a feasible manual job, hence we rely on IAC to manage and operate our Infrastructure. Here Terraform has been our tool of choice for various reasons from ease of learning to it's robust design. Terraform is an open source Infrastructure As Code tool by Hashicorp that lets you define AWS Infrastructure via a descriptive DSL and has been quite popular in the DevOps world since it's inception.

In this blog I'll share how we've used Terraform to Deploy an EKS Fargate cluster.

VPC

We'll start with deploying the Amazon VPC via Terraform. There are three recommended approaches for deploying a VPC to run EKS Fargate, let's look at each of them:

• Public and Private Subnets: the pods run in a private subnets while loadbalancers, both Application or Network are deployed in the Public subnets. One public and private subnet is deployed to each of the availability zones within the region for availability and fault tolerance, this is the deployment model we will follow fir this blog
• Public Subnets Only: both the pods (or nodes) and the loadbalancers are in public subnets, here three public subnets are deployed in three different availability zones within the region. All nodes have a public IP address and a security group blocks all inbound and outbound traffic to the nodes. To be honest I haven't ever figured out why would anyone need this :-)
• Private Subnets Only: both pods and loadbalancers run in private subnets only, three of which are created in each Availability zone of the region. Quite naturally we have to configure additional NAT Gateway, Egress Only Gateway, VPN or Direct Connect to be able to access the cluster. There's additional configuration on the kubectl side as well( which we will skip in this blog

VPC subnets should have certain tags which allow EKS Fargate to deploy internal loadbalancers to them and provision nodes; lets look at the tags first

• Key: kubernetes.io/cluster/cluster-name
• Value: Shared

The following tags allows EKS Fargate to decide where auto provisioned Elastic loadbalancers are deploy and also allows you to control where the application or network loadbalancers are configured

• Private Subnets:
  • Key: kubernetes.io/role/internal-elb
  • Value: 1
• Public Subnets:
  • Key: kubernetes.io/role/elb
  • Value: 1

The VPC configuration therefore is as follows, we'll use the AWS VPC Terraform module for this purpose as it provides easier configuration via declarative properties instead of having to write all the resources yourself.

module "vpc" {
  source                        = "terraform-aws-modules/vpc/aws"
  version                       = "3.4.0"
  name                          = "vpc-serverless"
  cidr                          = "176.24.0.0/16"
  azs                           = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets               = ["176.24.1.0/24","176.24.3.0/24","176.24.5.0/24"]
  public_subnets                = ["176.24.2.0/24","176.24.4.0/24","176.24.6.0/24"]
  enable_nat_gateway            = true
  single_nat_gateway            = true
  enable_dns_hostnames          = true
  manage_default_security_group = true
  default_security_group_name   = "vpc-serverless-security-group"

  public_subnet_tags = {
    "kubernetes.io/cluster/vpc-serverless" = "shared"
    "kubernetes.io/role/elb"               = "1"
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/vpc-serverless" = "shared"
    "kubernetes.io/role/internal-elb"      = "1"
  }

  tags = {
    "kubernetes.io/cluster/vpc-serverless" = "shared"
  }

}

There's a single NAT gateway for managing all traffic to the Nodes running in the Private subnet, we also have to ensure we keep the enable_dns_hostnames option set to true so that any ALBs that we configure in the future can be assigned hostnames for Canonical DNS mapping.

EKS Cluster

We'll use the AWS EKS Terraform module to deploy the EKS Fargate Cluster. A basic configuration for it is as follows

module "eks-cluster" {
  source                        = "terraform-aws-modules/eks/aws"
  version                       = "17.1.0"
  cluster_name                  = "eks-serverless"
  cluster_version               = "1.21"
  subnets                       = flatten([module.vpc.outputs.public_subnets, module.vpc.outputs.private_subnets])
  cluster_delete_timeout        = "30m"
  cluster_iam_role_name         = "eks-serverless-cluster-iam-role"
  cluster_enabled_log_types     = ["api", "audit", "authenticator", "controllerManager", "scheduler"]
  cluster_log_retention_in_days = 7

  vpc_id = module.vpc.outputs.vpc_id

  fargate_pod_execution_role_name = "eks-serverless-pod-execution-role"
  // Fargate profiles here
}

Fargate Profiles and CoreDNS

A basic configuration like the one above will deploy the EKS cluster, however you need to create Fargate Profiles that allow you to define which pods will run in Fargate. These Fargate profiles define selectors and a namespace to run the pods, along with optional tags, you also have to add a pod execution role name, for allowing the EKS infrastrcuture to make AWS API calls on the cluster owner's behalf. You can have up-to 5 selectors for a Fargate profile.

While Fargate takes care of provisioning nodes as pods for the EKS cluster, it still needs a component that can manage the networking within the cluster nodes, coreDNS is that plugin for EKS Fargate, and like any other workload, needs a Fargate profile to run. So we'll add both the plugin and profile configuration to our Terraform code.

First, let's update the profile configuration

fargate_profiles = {
    coredns-fargate-profile = {
      name = "coredns"
      selectors = [
        {
          namespace = "kube-system"
          labels = {
            k8s-app = "kube-dns"
          }
        },
        {
          namespace = "default"
        }
      ]
      subnets = flatten([module.vpc.outputs.private_subnets])
    }
  }

We're essentially saying, select the pods with label k8s-app to run in the kube-system namespace. Let's also add the CoreDNS plugin to the configuration

resource "aws_eks_addon" "coredns" {
  addon_name        = "coredns"
  addon_version     = "v1.8.4-eksbuild.1"
  cluster_name      = "eks-serve"
  resolve_conflicts = "OVERWRITE"
  depends_on        = [module.eks-cluster]
}

Conclusion

At this stage it's a simple module so you can bundle all of it into a single one. Here's how the file structure looks like.

cluster
├── main.tf
├── outputs.tf
├── providers.tf
├── terraform.tf
├── terraform.tfvars
├── variables.tf

The providers.tf file defined AWS provider along with AWS CLI credentials as variable that you can read from variables defined in the variables.tf file

terraform {
  required_version = "=1.0.2"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "3.49.0"
    }
  }
}

# Provider definition
provider "aws" {
  access_key = var.access_key
  secret_key = var.secret_key
  region     = var.region
  token      = var.session_token
}

if you're saving TF State in a remote backend you can define the configuration for it in the terraform.tf file.

terraform {

  backend "s3" {
    bucket         = "swiftalk-iac"
    dynamodb_table = "swiftalk-iac-locks"
    key            = "vpc/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
  }
}

That's it! Run terraform init to get an EKS Fargate Cluster up and running in minutes!