AWS PrivateLink supports Amazon S3! What makes you happy?

Introduction

With the update on 2/2/2021, you can now access Amazon S3 via AWS PrivateLink.

🚀 AWS PrivateLink for Amazon S3 is Now Generally Available

The following is a description of the advantages and points to note compared to the existing Gateway-type VPC endpoint.

Good points

Access from on-premises

I think this is what we have been waiting for.

In the case of the gateway endpoint, the S3 endpoint remains a global IP address. The route table configuration ensures that communication to S3 in the VPC is directed to the VPC endpoint.

Due to this specification, it was not possible to directly communicate with S3 from an on-premises environment connected using AWS Direct Connect or VPN.

How did we handle it so far?

It was necessary to build a proxy server on EC2. Naturally, there are operating and maintenance costs associated with this proxy server.

For Interface type endpoints with AWS PrivateLink, an ENI is created in the VPC. The private IP assigned to the ENI can be used to make a private connection directly from the on-premises environment to S3!

Access from another VPC or region

The concept is the same as on-premise.

Since it can be accessed via the private IP of the VPC, it can also be accessed by VPCs in other regions connected via VPC Peering.

It does not mean that a single endpoint can access S3 buckets in multiple regions.
As with the Gateway VPC endpoint, the S3 buckets to be accessed must reside in the same region. VPCs and VPC endpoints in the Tokyo region can only access S3 buckets that reside in the Tokyo region.

Use Gateway and Interface endpoints together

S3 Gateway endpoint and Interface endpoint can use together. For example, an application on a VPC can continue to use the Gateway endpoint, while an application on-premises can use the Interface endpoint.

As discussed in more detail below, this configuration has advantages because Interface endpoints must take into account data transfer charges and changes to the endpoint URL.

Points to consider

Usage fee

Gateway VPC endpoints are free of charge, but Interface VPC endpoints are charged per endpoint (USD/hour) and per GB of processed data .

Pricing in Tokyo Region

If you have a redundant configuration with 3 AZs, you will have to pay for 3 endpoints.

Pricing per GB data processed is also not very expensive, but if you are exchanging large amounts of data, you may need to consider the price beforehand.

Specifying endpoints

The S3 Interface endpoint does not support the private DNS feature. The checkbox is grayed out when creating the endpoint.

The Private DNS feature resolves the default DNS name of the service (e.g. ec2.ap-northeast-1.amazonaws.com) to the private IP address assigned to the VPC endpoint.

This allows VPCs using Route 53 Resolver (formerly Amazon Provided DNS) and on-premises environments using Route 53 Resolver for Hybrid Clouds to use the default DNS hostname to send requests to VPC endpoints.

Since this feature is not available, you will need to access S3 using your VPC endpoint's unique DNS name.

AWS CLI Example

$ aws s3 ls s3://my-bucket/ --endpoint-url https://bucket.vpce-1a2b3c4d-5e6f.s3.ap-northeast-1.vpce.amazonaws.com

AWS SDK (boto3) example

import boto3

s3_client = boto3.client(
    's3',
    endpoint_url = 'https://bucket.vpce-1a2b3c4d-5e6f.s3.ap-northeast-1.vpce.amazonaws.com'
)

You need to change the subdomain depending on the API you want to operate. Specify ”bucket" for the subdomain if you want to perform API operations related to the S3 bucket. It's a "bucket", not a bucket name.

For example, if you directly specify the private IP of the endpoint, SSL validation failed will occur.

$ aws s3 ls --endpoint-url https://10.0.0.9
SSL validation failed for https://10.0.0.9/ ("hostname 'xxx.xxx.xxx.xxx' doesn't match either of 's3.ap-northeast-1.amazonaws.com', 
'*.accesspoint.vpce-1a2b3c4d-5e6f-ap-northeast-1d.s3.ap-northeast-1.vpce.amazonaws.com', 
'*.control.vpce-1a2b3c4d-5e6f-ap-northeast-1a.s3.ap-northeast-1.vpce.amazonaws.com', 
'*.accesspoint.vpce-1a2b3c4d-5e6f.s3.ap-northeast-1.vpce.amazonaws.com', 'bucket.vpce-1a2b3c4d-5e6f-ap-northeast-1a.s3.ap-northeast-1.vpce.amazonaws.com', 
'*.s3-control.ap-northeast-1.amazonaws.com', '*.control.vpce-1a2b3c4d-5e6f-ap-northeast-1d.s3.ap-northeast-1.vpce.amazonaws.com', 
'*.s3.ap-northeast-1.amazonaws.com', '*.s3-accesspoint.ap-northeast-1.amazonaws.com', '*.control.vpce-1a2b3c4d-5e6f.s3.ap-northeast-1.vpce.amazonaws.com', '*.bucket.vpce-1a2b3c4d-5e6f-ap-northeast-1d.s3.ap-northeast-1.vpce.amazonaws.com', 
'bucket.vpce-1a2b3c4d-5e6f-ap-northeast-1c.s3.ap-northeast-1.vpce.amazonaws.com', 'bucket.vpce-1a2b3c4d-5e6f.s3.ap-northeast-1.vpce.amazonaws.com',
'*.control.vpce-1a2b3c4d-5e6f-ap-northeast-1c.s3.ap-northeast-1.vpce.amazonaws.com', '*.bucket.vpce-1a2b3c4d-5e6f.s3.ap-northeast-1.vpce.amazonaws.com', '*.accesspoint.vpce-1a2b3c4d-5e6f-ap-northeast-1c.s3.ap-northeast-1.vpce.amazonaws.com', 
'*.bucket.vpce-1a2b3c4d-5e6f-ap-northeast-1c.s3.ap-northeast-1.vpce.amazonaws.com', 
'bucket.vpce-1a2b3c4d-5e6f-ap-northeast-1d.s3.ap-northeast-1.vpce.amazonaws.com', 
'*.accesspoint.vpce-1a2b3c4d-5e6f-ap-northeast-1a.s3.ap-northeast-1.vpce.amazonaws.com', 
'*.bucket.vpce-1a2b3c4d-5e6f-ap-northeast-1a.s3.ap-northeast-1.vpce.amazonaws.com'",)

Experiment

What if my application doesn't support endpoint URL changes?

I tried setting the private IP address of the interface VPC endpoint locally to the hosts file to force it to override the default DNS name locally. Then, I was able to perform the bucket-level operations. However, this kind of usage is not documented and will most likely not be supported by AWS.

10.0.0.9 s3.ap-northeast-1.amazonaws.com
10.0.0.9 <bucket-name>.s3.ap-northeast-1.amazonaws.com

$ aws s3 ls
$ aws s3 ls s3://my-bucket/

Please note that there are many disadvantages to using a hosts file, so please use it only as a reference.

e.g.

• No redundant configuration
• Need to add a DNS name for each bucket.

Official Document

Amazon S3 and interface VPC endpoints (AWS PrivateLink)
https://docs.aws.amazon.com/AmazonS3/latest/userguide/privatelink-interface-endpoints.html

That's all.
Happy storing!