How to automatically create S3 lifecycles using AWS CLI and AWS SDK

Co-author: Le Hai Dang @natsu08122

1. Introduction
S3 lifecycle is a feature in Amazon S3 that allows you to automatically manage the storage of objects throughout their lifecycle. The main purpose of S3 lifecycle is help AWS user to optimize storage costs and automate data management. There are two main actions:

• Transition: Move objects between storage classes.

• Expiration: Delete objects after a specified time.

S3 lifecycle does support these storage classes: Standard, Intelligent-Tiering, Standard-IA (Infrequent Access), One Zone-IA, Glacier Instant Retrieval, Glacier Flexible Retrieval, Glacier Deep Archive. One important thing to note: S3 Policy cannot block S3 Lifecycle Rules. Here is the official AWS docs.

2. Problem statement
One day, we have a bunch of S3 buckets with the below folder structure: //

The retention directory of an object is defined by its expiration time. So, we need a solution to make a appropriate lifecycle of all S3 buckets in the corresponding folders. By utilizing S3 lifecycle management feature, we will create the rules using two main information:

• Prefix: /
• Object Tag: "isFile:True" (It means that all objects should be tagged with : that is "isFile:True"

We have several methods to do it using AWS management console, IaC, CDK, AWS CLI, AWS SDK. This blog will cover using AWS CLI and SDK to accomplish this task.

3. Solutions
3.1. Using AWS CLI
Prerequisites:

• AWS CLI version 2
• AWS Credentials In order to create S3 lifecycle rule, we use the following command:

aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --lifecycle-configuration file://config.json

However, there is one point that we should pay our attention to:

All old rules will be overwritten by the new rule. In other words, all old rules will be deleted, new rules will be created when running the command.

This is an example of config.json file:

{
  "Rules": [
      {
          "Expiration": {
              "Days": 7,
          },
          "Filter": {
              "And":
                  {
                  "Prefix": "7days/",
                  "Tags": [{
                      "Key": "isFile",
                      "Value": "true"
                  }],
              }
          },
          "ID": "7days-retention-policy",
          "Status": "Enabled",
      },
  ],
}

Note: If we want to have multiple conditions in "Filter" block, just use "And". Otherwise, we will get an error "MalformedXML".

3.2. Using AWS SDK
Prerequisites: AWS SAM latest version

How to do:

• Step 1: Clone this code into your environment.

• Step 2: Follow README.md in above code.

• Step 3: Run the command to setup AWS environment:
  

sam deploy --guided

After deployment is complete, the S3 lifecycle rules generation code will run on Lambda.

4. Solutions

This blog shows how to automatically create S3 lifecycles using AWS CLI and AWS SDK. I hope it will be useful for you. Thank you for your reading and feel free to leave your comment!