EC2 Charges — They are NAT a joke

#aws #cloud #tutorial

How often do you check your cloud provider bill and costs? Do you know what your biggest costs in the cloud are? Have you checked if all the costs are justified?

This story begins with me doing just that...

I was investigating a project that is running in AWS. One of the biggest costs is Elastic Compute Cloud (EC2), which accounts for nearly 70% of the total bill:

However, when I was having a look at these EC2 costs in more detail, there was an extra EC2-Other service which accounted for a hefty amount of the total EC2 costs:

EC2-Other — sorry, what? 😐

After some research into this EC2-Other service, it sounds almost like it is an expected cost, and it was just part and parcel of using EC2:

The EC2-Other category includes multiple service-related usage types, tracking costs associated [with] Amazon EBS volumes and snapshots, elastic IP addresses, NAT gateways, data transfer, and more.

– EC2 Cost Explorer

So, for a moment I stopped looking into these costs and just assumed they were normal.

EC2-Other — sorry, no. ✋

A few days had passed, and I was still investigating that same project. This time I was looking specifically at a monthly bill breakdown. This was when I noticed something odd:

Did you spot it?

52,664 GB (~53 TB) of data being processed through our NAT gateway — that is suspicious. We would expect to see some data being transferred through the NAT gateway but that amount is absurd.

My investigations began, and well it did not take long to find an answer.

I determined that every time a container in EC2 was pulling an image from ECR (Elastic Container Registry) it was being transferred through our NAT gateway. Every file that was being transferred from S3 to a container or vice versa, this was going through the gateway. Even our logs being sent from any container to CloudWatch, through the gateway…

Now the costs make a little more sense.

EC2-Other — sorry, bye 👋

Do not panic — there is a solution — enter VPC Endpoints, also known as AWS PrivateLink.

You can use AWS PrivateLink to connect the resources in your VPC to services using private IP addresses, as if those services were hosted directly in your VPC.

– AWS PrivateLink concepts — Amazon Virtual Private Cloud

If you use your console for changes to infrastructure then follow this tutorial on how to set these endpoints up for S3, ECR and logs: Create Private Links via Console or also this one provided by AWS: New VPC Endpoint for S3.

However, if you use CDK to deploy your infrastructure, like a hero, then here is how to set up your 3 new VPC endpoints. I am using Kotlin but if you use TypeScript or another language it shouldn’t be too hard to adjust.

CDK (In Kotlin)

Here are your imports for this setup:

import software.amazon.awscdk.services.ec2.GatewayVpcEndpoint
import software.amazon.awscdk.services.ec2.GatewayVpcEndpointProps
import software.amazon.awscdk.services.ec2.GatewayVpcEndpointAwsService
import software.amazon.awscdk.services.ec2.InterfaceVpcEndpoint
import software.amazon.awscdk.services.ec2.InterfaceVpcEndpointProps
import software.amazon.awscdk.services.ec2.InterfaceVpcEndpointService

You’ll need to have your VPC available in this stack.

Gateway Endpoint

Firstly here is your Gateway Endpoint for S3:

private val s3VpcEndpoint = GatewayVpcEndpoint(
  scope,
  "your-s3-endpoint",
  GatewayVpcEndpointProps.builder()
    .vpc(vpc)
    .service(GatewayVpcEndpointAwsService.S3)
    .build()
)

At the time of writing this, gateway endpoints were only available for S3 and DynamoDB — otherwise you must use an Interface Endpoint. Gateway endpoints for S3 are offered at no cost and the routes are managed through route tables.

Interface Endpoint

Interface endpoints are priced at $0.01/per AZ/per hour. Cost depends on the Region, check current pricing. Data transferred through the interface endpoint is charged at $0.01/per GB (depending on Region).

For the Interface Endpoint for ECR you need to have your security groups available:

It is important to include the correct security groups here or else you could have issues with your containers not being able to pull from ECR.

private val ecrVpcEndpoint = InterfaceVpcEndpoint(
  scope,
  "your-ecr-endpoint",
  InterfaceVpcEndpointProps.builder()
.service(InterfaceVpcEndpointService("com.amazonaws.$yourRegion.ecr.dkr"))
    .vpc(vpc)
    .privateDnsEnabled(true)
    .securityGroups(
      listOf(exampleSecurityGroup)
    )
    .build()
)

Then for the Interface Endpoint for Logs:

private val logsVpcEndpoint = InterfaceVpcEndpoint(
  scope,
  "your-logs-endpoint",
  InterfaceVpcEndpointProps.builder()
    .service(InterfaceVpcEndpointService("com.amazonaws.$yourRegion.logs"))
    .vpc(vpc)
    .privateDnsEnabled(true)
    .securityGroups(
      listOf(exampleSecurityGroup)
    )
    .build()
)

Sadly with CDK you cannot add a name tag to these endpoints (see issue here) so when you deploy you will see something that looks like this:

So, I pushed these changes to our production environment halfway through the day 23rd June, but I think you can see that quite clearly here:

So after leaving our system to run for a little while with these endpoints in place we started to see the EC2-Other costs drop significantly. For us, our costs dropped from over $2000 per month to just over $200!

I hope that you too can find a use for these VPC Endpoints to help lower your costs!

There is an increase, of course, in the costs for your VPC for these endpoints, but I am not too worried about the $1 increase there: