As infrastructure architects, we need to manage cloud resources with cost in mind, so understanding features like Requester Pays for AWS S3 is useful for cost optimization.
The Requester Pays feature of AWS S3 buckets is a good option to explore in "some cases" because it can potentially help with optimizing the cost of AWS S3 buckets.
Note: Please keep in mind that I used the term "some cases" because different use cases will likely require different resources and configurations.
Examples of such practical cases include:
Large Data Sets: Companies provide large datasets used for training machine learning (ML) models.
Benefit from enabling Requester Pays because they can reduce the high costs associated with accessing training datasets for AI and ML.
Commercial Distribution: Platforms that offer video editing directly in the cloud, where users stream large video files during the editing process (an example of such an app mostly likely already installed on your phone :) )
Benefit: Users will pay for the data they consume, and the platform will provide high-performance editing tools, saving on costs with usage for scalable service offerings.
Cross-Account Access: This is when users from other AWS accounts frequently access S3 objects.
Benefit: Users will pay the cross-account data transfer costs
How to enable the Requester Pays feature
Using Terraform, we can create the aws_s3_bucket_request_payment_configuration resource for an AWS S3 bucket. For the payer attribute (described later in this blog), we can select BucketOwner or Requester.
BucketOwner is the default setting for all new S3 buckets if no specific request payment configuration is applied.
Simple terraform code snippet ( of course, not production-friendly):
provider "aws" {
region = "us-east-1"
}
variable "request_payer" {
description = <<-EOD
(Optional) Specifies who should bear the cost of Amazon S3 data transfer.
It can be either BucketOwner or Requester. By default, the owner of the S3
bucket would incur the costs of any data transfer. See Requester Pays
Buckets developer guide for more information.
EOD
type = string
default = "Requester"
}
resource "aws_s3_bucket" "this" {
bucket = "in-n-out-editme"
force_destroy = false
tags = {
foo = "bar"
}
}
resource "aws_s3_bucket_request_payment_configuration" "this" {
bucket = aws_s3_bucket.this.id
payer = var.request_payer
}
output "bucket_name" {
value = aws_s3_bucket.this.bucket
}
output "requester_pays_status" {
value = aws_s3_bucket_request_payment_configuration.this.payer
}
Note: aws_s3_bucket_request_payment_configuration resource cannot be used with S3 directory buckets.
View in the AWS Console UI
The result of the above code (Enabled feature) can be found in the AWS console under the S3 Bucket overview and navigating to the Requester Pays UI view (refer to the screenshot below)
Understanding the Cost breakdown
Charges are handled differently in AWS S3 when using the Requester Pays feature compared to the standard S3 pricing model.
Normally, the bucket owner pays for all data transfer and request charges associated with the bucket. However, in the Requester Pays model, these costs are shifted to the person or service accessing the data (requester).
Bucket Owner Costs:
Storage: the data stored in the S3 bucket regardless of access patterns.
Requester Costs:
Data Transfer Out: When data is transferred from the S3 bucket to the internet or another AWS region, the requester pays for the data transfer costs.
S3 Requests: for initiated operations, such as PUT, GET, POST, and LIST.
Including when request authentication fails and the request is anonymous, both resulting in an HTTP 403 error.
How are Requesters billed?
When accessing the Requester Pays S3 buckets, requesters must include billing details: x-amz-request-payer: requester in the request header. This indicates that they agree to pay for the data transfer and request costs. AWS utilizes this header to ensure that costs are billed to the requester.
This setup prevents unauthorized requesters from being charged without their consent.
Requesters also need IAM permissions to ensure that only authorized users can access the data.
How to View Payment Split in AWS Console
Here are the steps to check how the payment is split for Requester Pays S3 buckets:
*Billing and Cost Management Dashboard
*
- Navigate to the Billing and Cost Management Dashboard.
- Use the Cost Explorer to get detailed information about S3.
- Go to Cost Explorer: filter your data by service ( which in this case would be Amazon S3)
Detailed Billing Reports:
Enable Detailed Billing Reports with Resources and Tags to see detailed information about your S3 usage. If you have Requester Pays enabled, these reports will include information on requester-initiated transactions.
Reports can be set to show which bucket incurred which costs so you can see the split between what you pay as the bucket owner (for storage and owner-initiated data transfers) and what requesters pay (for their data transfers).
Tagging:
Add cost allocation tags on your S3 buckets. Once tagged, activate these tags in the Cost Allocation Tag section of the Billing Dashboard to include them in your reports.
Visualizing with AWS QuickSight Integration:
It is possible to download Detailed Billing Reports reports from the S3 bucket specified and analyze them using tools like Amazon QuickSight.
Summary
When implementing the Requester Pays model on an S3 bucket, it's important to ensure that potential data users are aware that they will incur charges for their data access.
It is important to Monitor Access to your S3 buckets at all times. Use S3 access logs or AWS CloudTrail to monitor who is accessing your S3 buckets to manage data transfer and associated costs (maybe one day I will write a blog about this setup)
Understanding the Requester Pays model allows all of us, AWS users, to:
deploy cost-effective data-sharing solutions in AWS
manage costs associated with data stored in AWS S3
help to build cost-effective services aligned with organizational or project goals.
Official documentation link:
Top comments (0)