Optimizing S3 Bucket Management and Lifecycle with Terraform

Recently, I managed to bring an existing manually-created S3 bucket under Terraform control and set up a lifecycle policy for it. Here are the notes from that experience.

Procedures

Step 1: Define the Terraform Resource

First, define the existing S3 bucket as a Terraform resource. Below is the configuration for a development bucket named example-dev.

resource "aws_s3_bucket" "example_dev" {
  bucket = "example-dev"

  tags = {
    Environment = "dev"
  }
}

Step 2: Set Up the Lifecycle Policy

Next, set up a lifecycle policy for the bucket's data. This policy transitions the data to the STANDARD_IA storage class after 30 days, and schedules it for deletion after 90 days.

resource "aws_s3_bucket_lifecycle_configuration" "example_dev_lifecycle" {
  bucket = aws_s3_bucket.example_dev.id

  rule {
    id     = "ManageLifecycleAndDelete"
    status = "Enabled"

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    expiration {
      days = 90
    }
  }
}

Step 3: Import the Resources into Terraform

To ensure Terraform recognizes the existing bucket, use the following commands to import the bucket and its lifecycle policy:

terraform import module.s3.aws_s3_bucket.example_dev example-dev
terraform import module.s3.aws_s3_bucket_lifecycle_configuration.example_dev_lifecycle example-dev

Step 4: Verification

Use the AWS CLI to verify that the bucket's settings have been correctly applied.

aws s3api head-object --bucket example-dev --key [object key]

Supplement: Cost-Efficient Lifecycle Policy Settings

When designing the lifecycle policy, I paid close attention to the following cost-related points:

Minimum Storage Duration

The STANDARD_IA storage class requires a minimum of 30 days of storage. If the deletion period set in the lifecycle policy is less than 30 days, charges for the unelapsed period will occur. Therefore, ensure the deletion policy is set for at least 30 days after transitioning to STANDARD_IA.

Lifecycle Transition Charges

Moving data from the STANDARD class to the STANDARD_IA class incurs lifecycle transition fees. These costs are calculated based not only on the amount of data but also on the number of data transition requests. If you're dealing with a large number of small objects, these charges can be significant, so plan accordingly.

References

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket