Optimise AWS Costs: Automate Unused EBS Snapshot Cleanup with Lambda

A Step-by-Step Guide to Building and Deploying a Cost-Saving Lambda Function for EBS Snapshot Management

💡 Introduction

Welcome to the world of DevOps! Today, we’ll dive into an exciting cost optimisation project that tackles a common challenge faced by AWS users—managing unnecessary EBS snapshots. In this project, we’ll create a Python program that automatically cleans up:

• Snapshots unattached to any volume.
• Snapshots of instances that are no longer in a running state.

This hands-on project is not only a great way to reduce your AWS storage costs but also an opportunity to strengthen your understanding of resource utilisation in AWS. Along the way, we’ll explore boto3, the AWS SDK for Python, to interact with AWS resources and manipulate them to suit our needs.

Let’s get started on this journey to automate cost-saving measures and enhance your DevOps skills!

💡 Pre-Requisites

Before we jump into building our cost optimization project, let’s make sure you have the necessary tools and knowledge in place. Here’s what you’ll need:

• Python: Ensure Python is installed on your system. Version 3.7 or above is recommended.
• Basic Python Knowledge: Familiarity with Python fundamentals will help you understand and implement the script effectively.
• AWS Account: You’ll need access to an AWS account to interact with resources like EC2 and EBS snapshots.
• Basic Understanding of EC2: Since we’ll be working with the EC2 dashboard and snapshots, having a foundational understanding of EC2 instances is crucial.
• AWS CLI: Install and configure the AWS CLI on your system using the command:

aws configure

This step will set up your access keys, region, and output format to enable seamless communication with AWS services.

Once you’ve ticked off these prerequisites, you’ll be ready to proceed with the project!

💡 Starting the Project

The source code for this project is available in my GitHub repository:

🔗 GitHub Repo: AWS Cloud Cost Optimization

Follow these steps to create and configure your Lambda function:

Create a Lambda Function:

• Navigate to the Lambda Console in your AWS account.
• Click on Create Function and select the option Author from scratch.
• Provide the function name, e.g., ebs-volume-check, and select Python 3.12 as the runtime.

Add the Code:

• In the code editor of the AWS console, open the lambda_function.py file.
• Copy and paste the content of script.py from the GitHub repository into the editor. Here’s the code snippet for reference:

    import boto3

    def lambda_handler(event, context):
        ec2 = boto3.client('ec2')

        # Get all EBS snapshots
        response = ec2.describe_snapshots(OwnerIds=['self'])

        # Get all active EC2 instance IDs
        instances_response = ec2.describe_instances(Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
        active_instance_ids = set()

        for reservation in instances_response['Reservations']:
            for instance in reservation['Instances']:
                active_instance_ids.add(instance['InstanceId'])

        # Iterate through each snapshot and delete if it's not attached to any volume or the volume is not attached to a running instance
        for snapshot in response['Snapshots']:
            snapshot_id = snapshot['SnapshotId']
            volume_id = snapshot.get('VolumeId')

            if not volume_id:
                # Delete the snapshot if it's not attached to any volume
                ec2.delete_snapshot(SnapshotId=snapshot_id)
                print(f"Deleted EBS snapshot {snapshot_id} as it was not attached to any volume.")
            else:
                # Check if the volume still exists
                try:
                    volume_response = ec2.describe_volumes(VolumeIds=[volume_id])
                    if not volume_response['Volumes'][0]['Attachments']:
                        ec2.delete_snapshot(SnapshotId=snapshot_id)
                        print(f"Deleted EBS snapshot {snapshot_id} as it was taken from a volume not attached to any running instance.")
                except ec2.exceptions.ClientError as e:
                    if e.response['Error']['Code'] == 'InvalidVolume.NotFound':
                        # The volume associated with the snapshot is not found (it might have been deleted)
                        ec2.delete_snapshot(SnapshotId=snapshot_id)
                        print(f"Deleted EBS snapshot {snapshot_id} as its associated volume was not found.")

Deploy the Code:

• Click the Deploy button to save and apply the changes.

Increase Timeout:

• By default, the timeout for a Lambda function is 3 seconds, which is insufficient for this operation.
• Go to the Configuration tab, select General Configuration, and increase the timeout to 10 seconds.

Test the Function:

• Create a test event with dummy data and invoke the function.
• You will encounter an UnauthorizedOperation error during the test.

This error occurs because the Lambda function lacks the required permissions, which we will address in the next step.

💡 Adding Required Permissions

To enable the Lambda function to interact with EC2 resources, we need to add specific permissions to its service role. Instead of granting full access, we’ll follow the principle of least privilege to ensure the function has only the permissions it requires. Follow these steps:

Locate the Service Role:

• Navigate to the Configuration tab of the Lambda function.
• Under the Permissions section, find the service role associated with the function. The role name will be displayed there.

Open the Service Role in IAM:

• Click on the service role name, and it will redirect you to the IAM Console.

Create an Inline Policy:

• In the IAM Console, under the Permissions tab, click on Add Permissions and select Create Inline Policy.
• Configure the policy as follows:
• Service: Select EC2.
• Actions: Choose only the following permissions:
• DescribeSnapshots
• DescribeInstances
• DescribeVolumes
• DeleteSnapshots
• Resources: Set the permissions to apply to all resources for simplicity. If you need stricter control, specify the resource ARNs.

Review and Assign the Policy:

• Click Next and review the policy details.
• Assign a name to the policy, e.g., ebs-permissions.
• Click Create Policy to apply it to the service role.

The ebs-permissions policy will look like this:

Once this inline policy is added, the Lambda function will have all the permissions it needs to describe and delete snapshots, volumes, and instances effectively.

With the permissions in place, let’s test the function again in the next step!

💡 Testing the Lambda Function

With the permissions in place, it’s time to test our Lambda function using real-world scenarios. Follow these steps to see how the function performs:

1. First Use-Case: Cleaning Snapshots of Deleted Instances

Create an EC2 Instance:

• Launch a t2.micro instance and name it test-ebs.
• Use a default AMI and ensure the instance has a root volume.

Create a Snapshot:

• Go to the EC2 Dashboard and create a snapshot from the volume attached to test-ebs.
• Wait for the snapshot creation to complete.

Delete the Instance:
Once the snapshot is ready, terminate the test-ebs instance. This leaves behind a snapshot of a non-existent instance, which is now redundant.

Run the Lambda Function:

• Navigate to your Lambda function in the AWS Console and test the function.
• The function will detect that the snapshot is not attached to a running instance and delete it.

Verify the deletion in the Snapshots section of the EC2 Dashboard.

2. Second Use-Case: Cleaning Snapshots of Deleted Volumes

Create an EBS Volume:

• Create a 1 GB gp3 EBS volume and name it ebs-volume.

Create a Snapshot:

• Navigate to the EBS Volumes section and create a snapshot from ebs-volume.
• Wait for the snapshot to be created.

Delete the Volume:
Once the snapshot is ready, delete the ebs-volume. This leaves behind a snapshot not attached to any volume.

Run the Lambda Function:

• Test the Lambda function again.
• The function will identify that the snapshot is not linked to any volume and delete it.
• Confirm the deletion in the Snapshots section of the EC2 Dashboard.

By successfully handling these two scenarios, the Lambda function demonstrates its ability to identify and clean up redundant EBS snapshots, saving costs and streamlining resource management.

💡 Conclusion

Congratulations! 🎉 You’ve just built a cost-optimization solution on AWS using Python and Lambda. In this project, we explored how to identify and clean up redundant EBS snapshots that no longer serve a purpose, helping you save on cloud costs while improving resource management.

Here’s a quick recap of what we achieved:

• Learned how to create and configure a Lambda function from scratch. Utilized AWS’s boto3 library to interact with EC2 resources programmatically.
• Followed the principle of least privilege to ensure secure access to AWS services.
• Tested real-world scenarios to validate our function’s efficiency in cleaning up unattached snapshots.

This project highlights the importance of resource optimization in a cloud environment and equips you with hands-on experience in automating AWS tasks using Python.

Feel free to customize and enhance this function further. For example, you could set up a CloudWatch rule to trigger the function periodically or extend the logic to handle additional cleanup scenarios.

Thank you for following along, and stay tuned for more exciting DevOps and cloud projects! 🚀

🚀 For more informative blog, Follow me on Hashnode, X(Twitter) and LinkedIn.

Till then, Happy learning! 😊