Tagging Made Easy: Automating Resource Labeling in AWS with Lambda and Resource Explorer

Tags and labels are one of the most important elements for resource inventory, compliance and cost savings.

Managing tags in a growing AWS infrastructure can be cumbersome and time-consuming. Manually tagging resources often leads to inconsistencies and inaccurate data, hindering cost optimization, resource management, and compliance efforts. Adding new tags across all existing resources in a specific account, especially across multiple regions, can be a daunting task.

All Cloud providers recommend to always tag/label your resources during their creation time, however, there are situations in which an organization might decide to add new tags to their existing resources.

Solution:

Here’s a Python script that leverages the power of AWS’ “Resource Explorer” - a powerful search and discovery service - and Boto3 - AWS’ SDK for Python - to automate the process of adding missing tags while saving time and ensuring consistency. This solution could be easily implemented by SRE's, DevOps and/or Cloud Engineers scoping to improve resource organization, inventory and cost management.

Scenario:



In this fictional example, I’m imagining that a University is restructuring its infrastructure and its ‘tagging strategy.’ The University's CTO decided to dedicate/re-tag an entire already existing AWS account to the “College of Liberal Arts, and the Philosophy Department.” This Python script is searching for two tags:

‘philosophy’ and ‘liberal-arts’

in the two AWS Zones in which the College of Liberal Arts, and the Philosophy Department resources are located:
’us-east-1’ and ‘us-east-2’

Currently, this script is written for an AWS Lambda function, with the Lambda Handler set up, yet, it could be easily refactored to work as a simple Boto3 script to be executed from a local machine.

Code Breakdown:

This section describes the dependencies and functions that make up the AWS Auto Tagging Solution.

Dependencies:
These are the necessary dependencies for the Lambda Function:

import boto3
from botocore.exceptions  import ClientError
from botocore.config import Config
import logging
import json

Logger setup:
As a good habit, I added a Logger for possible debugging, either in testing or production stages. This also provides data for further analysis of the script's performance:

logger = logging.getLogger()
logger.setLevel(logging.INFO)
logging.getLogger("boto3").setLevel(logging.WARNING)
logging.getLogger("botocore").setLevel(logging.WARNING)

Lambda Handler:
The core of this script lies on the Resource Explorer Boto3 Client, which is added to the Lambda Handler . ‘Resource Explorer’ searches for all the resources in the AWS account and looks for its tags.
If the service does not find the ‘philosophy’ and ‘liberal-arts’ key tags in the ’us-east-1’ and ‘us-east-2’ regions, then it will automatically add them with the apply_tags function - called inside the def lambda_handler:

def lambda_handler(event, context):
    logger.debug('Incoming Event')
    logger.debug(event)

    resource_explorer_client = boto3.client(
        'resource-explorer-2',
    )

    missing_philosophy_tag = get_resources_missing_tag(resource_explorer_client, 'philosophy')
    missing_liberal_arts_tag = get_resources_missing_tag(resource_explorer_client, 'liberal-arts')

    logger.info(f"# of Resources Missing 'philosophy' {missing_philosophy_tag['Count']['TotalResources']} - Complete List? {missing_philosophy_tag['Count']['Complete']}")
    logger.info(f"# of Resources Missing 'liberal-arts' {missing_liberal_arts_tag['Count']['TotalResources']} - Complete List? {missing_liberal_arts_tag['Count']['Complete']}")

    map_philosophy_arns=[]
    for this_resource in missing_philosophy_tag['Resources']:
        map_philosophy_arns.append(this_resource['Arn'])
    logger.info(f"The Map Philosophy ARN:{map_philosophy_arns}")

    map_liberal_arts_arns=[]
    for this_resource in missing_liberal_arts_tag['Resources']:
        map_liberal_arts_arns.append(this_resource['Arn'])
    logger.info(f"The Map Liberal Arts ARN:{map_liberal_arts_arns}")

    apply_tags(map_philosophy_arns, {'philosophy': 'phil-dept-server'})

    apply_tags(map_liberal_arts_arns, {'liberal-arts': 'la-dept-server'})

Missing Tags:
'Resource Explorer' looks for the specified tags in all the resources available in the current account in which the solution is being deployed. Since 'Resource Explorer' will only provide 100 results (resources) at a time, I added a paginator:

def get_resources_missing_tag(client, tag_name): 
    return (
        client.get_paginator('search')
            .paginate(QueryString=f"-tag.key:{tag_name}")
            .build_full_result()
    )

Applying Tags:
The ResourceGroupsTaggingAPI efficiently applies the identified missing tags, ensuring consistency and optimizing resource management:

def apply_tags(list_of_resources, tag_map):
    resources_by_region = return_resources_by_region(list_of_resources)
    counter = 0
    for this_resource in list_of_resources:
        counter += 1
        logger.info(f"{counter}) Add tag '{tag_map.keys()}' to '{this_resource}'")
    # iterates over regions:
    regions = ['us-east-1', 'us-east-2']
    for region in regions: 
        tagging_client = boto3.client('resourcegroupstaggingapi', region_name=region)

Returning a list of tagged resources:
The final portion of the script returns a dictionary with a list of all the newly tagged resources, separated by their respective regions - 'us-east-1', 'us-east-2' - and writes these results into a JSON file. For enhanced tracking and auditing capabilities, a future implementation could leverage DynamoDB to store the JSON file, providing a detailed history of tagging changes:

def return_resources_by_region(resources_for_all_regions):
    resources_by_region = dict()
    regions = ['us-east-1', 'us-east-2']
    for region_name in regions:
        resources_by_region[region_name] = [arn for arn in resources_for_all_regions if region_name in arn]

        logger.info(f"The {region_name} resources are: \n {resources_by_region[region_name]} \n") 

    return resources_by_region

def format_in_json(response):
    return json.dumps(response, indent=4, sort_keys=True, default=str)

This simple Lambda function can be scheduled once a week, as a simple auto-tagging solution, or be triggered by an event, such as the creation of a new resource.

Tag enforcement:
A long-term approach to an account-level 'Tagging' requirement would be to - once the new tags are applied to the current infrastructure by this Lambda function - implement an SCP (Service Control Policy). This SCP would deny the creation of any new resources that didn't contain the specified tags required by the Organization - in this example: ‘philosophy’ and ‘liberal-arts’ .

Click here for the full script available on GitHub, plus installation instructions for the required Boto3 and Botocore packages.