DEV Community

AWS Community Builders profile image Dickson
Dickson for AWS Community Builders

Posted on • Edited on

AWS Config Auto Remediation for Configuring S3 Lifecycle Rule

#aws #config #ssm #s3

Amazon Simple Storage Service (Amazon S3) is a popular storage solution offering high availability, scalability and durability. Enterprises around the world have put petabytes of data on S3 for various use cases, ranging from big data analysis to critical data backup. With the increasing number of objects stored in S3 buckets, storage cost becomes a major concern.

To ensure the objects are stored cost-effectively, S3 lifecycle configuration can be configured in every bucket so that all objects are automatically transitioned, archived or expired after a certain time. In this article, we will go through a solution that leverages AWS Config and AWS Systems Manager (SSM). This guarantees all S3 buckets to have at least one active lifecycle rule.

Note: Lifecycle configuration cannot be set on buckets with MFA delete enabled.

Architecture

The Config rule evaluates the lifecycle rule configuration of all S3 buckets. If a bucket is found non-compliant, AWS Config applies the remediation using a SSM Automation document.

Creating an AWS Config rule

  1. In the AWS Config console, select Rules in the navigation pane and click Add rule. If you cannot find the button, set up the AWS Config first.
    Add rule

  2. Among the AWS managed rules, choose the s3-lifecycle-policy-check rule and click Next.
    Specify rule type

    Note: There is another similar rule called s3-version-lifecycle-policy-check which checks on versioned S3 buckets.

  3. Leave all the values as default and click Next. Optionally, you can specify the parameters such as targetTransitionDays and targetTransitionStorageClass.
    Configure rule

  4. Review all the configurations and click Add rule.
    Review and create rule

The rule is now created and shows up on the console. If there is any noncompliant resource (i.e. S3 bucket with no lifecycle policy configured), you should see the count on the compliance column.
Rules

Creating a custom SSM Automation document

Since there is no existing automation for configuring S3 lifecycle rule, we have to write our own document. The automation will execute both the PutBucketLifecycleConfiguration and GetBucketLifecycleConfiguration APIs.



description: |
  ### Document Name - ConfigureS3BucketLifecycleRule

  ## What does this document do?
  This document is used to create or modify the lifecycle rule configuration for an Amazon S3 bucket.

  ## Input Parameters
  * BucketName: (Required) Name of the S3 bucket (not the ARN).
  * TransitionDays: (Optional) Number of days after creation when objects are transitioned to the specified storage class.
    * Default: 90
  * TransitionStorageClass: (Optional) Storage class to which the object is transitioned.
    * Default: "INTELLIGENT_TIERING"
  * NoncurrentTransitionDays: (Optional) Number of days after becoming noncurrent when objects are transitioned to the specified storage class.
    * Default: 30
  * NoncurrentTransitionStorageClass: (Optional) Storage class to which the noncurrent object is transitioned.
    * Default: "INTELLIGENT_TIERING"
  * AutomationAssumeRole: (Required) ARN of the role that allows Automation to perform the actions.

  ## Output Parameters
  * GetBucketLifecycleConfiguration.Output - JSON formatted response from the GetBucketLifecycleConfiguration API call
schemaVersion: "0.3"
assumeRole: "{{ AutomationAssumeRole }}"
outputs:
  - GetBucketLifecycleConfiguration.Output
parameters:
  BucketName:
    type: String
    description: (Required) Name of the S3 bucket (not the ARN).
    allowedPattern: (?=^.{3,63}$)(?!^(\d+\.)+\d+$)(^(([a-z0-9]|[a-z0-9][a-z0-9\-]*[a-z0-9])\.)*([a-z0-9]|[a-z0-9][a-z0-9\-]*[a-z0-9])$)
  TransitionDays:
    type: Integer
    description: (Optional) Number of days after creation when objects are transitioned to the specified storage class.
    default: 90
  TransitionStorageClass:
    type: String
    description: (Optional) Storage class to which the object is transitioned.
    default: INTELLIGENT_TIERING
    allowedValues:
      - STANDARD_IA
      - INTELLIGENT_TIERING
      - ONEZONE_IA
      - GLACIER
      - GLACIER_IR
      - DEEP_ARCHIVE
  NoncurrentTransitionDays:
    type: Integer
    description: (Optional) Number of days after becoming noncurrent when objects are transitioned to the specified storage class.
    default: 30
  NoncurrentTransitionStorageClass:
    type: String
    description: (Optional) Storage class to which the noncurrent object is transitioned.
    default: INTELLIGENT_TIERING
    allowedValues:
      - STANDARD_IA
      - INTELLIGENT_TIERING
      - ONEZONE_IA
      - GLACIER
      - GLACIER_IR
      - DEEP_ARCHIVE
  AutomationAssumeRole:
    type: String
    description: (Required) ARN of the role that allows Automation to perform the actions.
    allowedPattern: ^arn:(aws[a-zA-Z-]*)?:iam::\d{12}:role/[\w+=,.@-]+
mainSteps:
  - name: PutBucketLifecycleConfiguration
    action: "aws:executeAwsApi"
    description: |
      ## PutBucketLifecycleConfiguration
      Creates or modifies the lifecycle configuration for a S3 Bucket.
    isEnd: false
    inputs:
      Service: s3
      Api: PutBucketLifecycleConfiguration
      Bucket: "{{BucketName}}"
      LifecycleConfiguration:
        Rules:
          - Filter:
              Prefix: ""
            ID: "Default Lifecycle Rule"
            Status: Enabled
            Transitions:
              - Days: "{{ TransitionDays }}"
                StorageClass: "{{ TransitionStorageClass }}"
            NoncurrentVersionTransitions:
              - NoncurrentDays: "{{ NoncurrentTransitionDays }}"
                StorageClass: "{{ NoncurrentTransitionStorageClass }}"
    isCritical: true
    maxAttempts: 2
    timeoutSeconds: 600
  - name: GetBucketLifecycleConfiguration
    action: "aws:executeScript"
    description: |
      ## GetBucketLifecycleConfiguration
      Retrieves the S3 lifecycle configuration for a S3 Bucket.
      ## Outputs
      * Output: JSON formatted response from the GetBucketLifecycleConfiguration API call.
    timeoutSeconds: 600
    isCritical: true
    isEnd: true
    inputs:
      Runtime: python3.6
      Handler: validate_s3_bucket_lifecycle_configuration
      InputPayload:
        Bucket: "{{BucketName}}"
        TransitionDays: "{{ TransitionDays }}"
        TransitionStorageClass: "{{ TransitionStorageClass }}"
        NoncurrentTransitionDays: "{{ NoncurrentTransitionDays }}"
        NoncurrentTransitionStorageClass: "{{ NoncurrentTransitionStorageClass }}"
      Script: |-
        import boto3

        def validate_s3_bucket_lifecycle_configuration(event, context):
            s3_client = boto3.client("s3")
            bucket = event["Bucket"]
            transition_days = event["TransitionDays"]
            transition_storage_class = event["TransitionStorageClass"]
            noncurrent_transition_days = event["NoncurrentTransitionDays"]
            noncurrent_transition_storage_class = event["NoncurrentTransitionStorageClass"]

            output = s3_client.get_bucket_lifecycle_configuration(Bucket=bucket)
            updated_rules = output["Rules"]

            if any(
                any(updated_transition["Days"] == transition_days
                    and updated_transition["StorageClass"] == transition_storage_class
                    for updated_transition in updated_rule["Transitions"])
                and any(updated_noncurrent_transition["NoncurrentDays"] == noncurrent_transition_days
                        and updated_noncurrent_transition["StorageClass"] == noncurrent_transition_storage_class
                        for updated_noncurrent_transition in updated_rule["NoncurrentVersionTransitions"])
                and updated_rule["Status"] == "Enabled"
                for updated_rule in updated_rules
            ):
                return {
                    "output":
                    {
                        "message": "Bucket lifecycle configuration successfully set.",
                        "configuration": updated_rules
                    }
                }
            else:
                info = "CONFIGURATION VALUES DO NOT MATCH WITH PARAMETERS PROVIDED VALUES TransitionDays: {}, TransitionStorageClass: {}, NoncurrentTransitionDays: {}, NoncurrentTransitionStorageClass: {}".format(
                    transition_days,
                    transition_storage_class,
                    noncurrent_transition_days,
                    noncurrent_transition_storage_class
                )
                raise Exception(info)
    outputs:
      - Name: Output
        Selector: $.Payload.output
        Type: StringMap
Enter fullscreen mode Exit fullscreen mode

The same document can also be found in https://github.com/tiksangng/aws-community-resources/blob/main/config-remediation/s3-lifecycle-rule/ConfigureS3BucketLifecycleRule.yaml.

  1. In the AWS Systems Manager console, select Documents in the navigation pane, click Create document and choose Automation.
    Create document

  2. For Name, enter ConfigureS3BucketLifecycleRule. Switch to the Editor tab, click Edit and replace the content with the above document. Click Create automation.
    Create automationCreate automation

Creating an IAM role for the SSM automation

  1. In the IAM console, select Roles in the navigation pane and click Create role.
    Create role

  2. For Use case, choose Systems Manager. Click Next.
    Select trusted entity

  3. Choose AmazonSSMAutomationRole and click Next.
    Add permissions

  4. For Role name, enter s3-configure-bucket-lifecycle-rule. Click Create role.
    Name, review and createName, review and create

  5. Click View role.
    View role

  6. Click Add permissions and choose Create inline policy.
    s3-configure-bucket-lifecycle-rule

  7. Switch to the JSON tab, replace the content with the below document. Click Review policy.```json

{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:PutLifecycleConfiguration",
"s3:GetLifecycleConfiguration"
],
"Resource": "*",
"Effect": "Allow"
}
]
}


```Create policy

  1. For Name, enter s3-configure-bucket-lifecycle-rule. Click Create policy. Create policy

Setting up an automatic remediation for the AWS Config rule

  1. On the Rules page of the AWS Config console, choose the s3-lifecycle-policy-check rule and click View details.
    View details

  2. Click Actions and choose Manage remediation.
    Manage remediation

  3. For Select remediation method, choose Automatic remediation. For Remediation action details, choose ConfigureS3BucketLifecycleRule. For Resource ID parameter, choose BucketName. For AutomationAssumeRole, enter arn:aws:iam::<<aws-account-id>>:role/s3-configure-bucket-lifecycle-rule. Leave other values as default and click Save changes. Optionally, you can specify the parameters such as TransitionDays and TransitionStorageClass.
    Edit: Remediation actionEdit: Remediation action

The remediation is now configured. If there was any noncompliant resource before, it will become compliant after some time. In the future, all new S3 buckets will also have the lifecycle rule configured upon creation.
Resources in scope

Advanced usage

The entire solution can be deployed with an AWS CloudFormation template, which can be found in https://github.com/tiksangng/aws-community-resources/blob/main/config-remediation/s3-lifecycle-rule/main.yaml.

For enterprise use, this template can be deployed into multiple accounts easily using an AWS CloudFormation StackSet.

Note: If the solution is deployed in multiple regions, it is advised to separate the IAM role creation from the template as IAM resources are global constructs.

References

Top comments (0)

Read next

suin profile image

Securing external-dns: Encrypting TXT Registry Records

suin -

garciadiazjaime profile image

React + AWS Cognito: Email Authentication Setup Guide (First Part)

Jaime -

deekshagunde profile image

Cross-Project Dependencies Handling with DBT in AWS MWAA

Deeksha Gunde -

yyarmoshyk profile image

How to enable AWS S3 replication between Global AWS Region and AWS China

Yaroslav Yarmoshyk -