Send notification based on CloudWatch logs filter patterns

Multiple resources are deployed and we want a mechanism to get notified when any unusual things start to appear in logs, which as a developer we do not want to see. To get notified, we need to parse the logs to track certain Keywords, Errors, and Critical information which we think are red flags. And in this article, I will share how to send an email notification when certain Keywords, Errors appear in the CloudWatch logs.

For the purpose of this tutorial, we will consider a lambda function that we want to track and get notified of any errors or keywords. To track the logs, we will use CloudWatch Log Subscription Filter. CloudWatch Subscription Filter will analyze logs for certain errors or keywords using filter patterns (we will define them in the latter part of this article) and based on that it will invoke the destination resource (in our case, it’s the Error Handler lambda function).

The above is the architecture that we will implement. Here, we have a number of lambda functions for which we want to get notified in case of any errors. The lambda functions will push logs to CloudWatch and on the log group, we will have a subscription filter along with filter patterns. It will look for keywords like ERROR, INFO in the logs and if any of the keywords exist then it will invoke the Error Handler lambda function. The Error Handler lambda function will decode, decompress & parse the log data and will publish a message to the SNS topic, and the SNS topic will further publish the message to its subscribers (In our case it will be an email subscriber)

Here, we will start with the creation of the SNS Topic along with the email subscription (To which the notification will be sent). After that, create a dummy lambda function, which will act as an error, keyword-producing function. As a next step, create the Error Handler lambda function followed by the configuration of the CloudWatch Event trigger (where we will define the filter patterns). Finally, update the code base and add the necessary permission to publish a message from the lambda function.

Hands-on

SNS

Create the standard SNS topic

Open the SNS topic that you have created and create Email subscription.

After creating the subscription, you will receive the confirmation email. So, make sure to click on confirm subscription to receive email notifications.

Lambda functions

Error/Keyword producer lambda function

Create the error/keyword producer lambda function with the appropriate name & Python3.9 as runtime. The purpose of this lambda function will be to publish certain keywords in the logs, which we want to track to validate the end-to-end flow. Post creation, update the code of the same with the below snippet.

import json
import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
def lambda_handler(event, context):
    logger.info("Sample INFO log")
    logger.error("Sample ERROR log")
    logger.critical("High LATENCY")

Test it once to check if the lambda function is running without errors.

Error Handler lambda function

Create another lambda function with the relevant name & Python3.9 as runtime. The purpose of this lambda function is to parse the error logs and publish the message to SNS.

As a next step, update the source code of the function from here and deploy: https://github.com/srcecde/aws-tutorial-code/blob/master/lambda/lambda_proces_cw_error_notification.py

After deployment, add the environment variable SNS_TOPIC_ARN and the value as the ARN of the SNS topic that is created earlier.

Post adding the environment variable, we need to provide the permission to lambda function to publish a message to SNS. So, add the permission to the IAM role (Which can be found by clicking on Configuration → Permissions → Role name) of the lambda function to publish the message to the SNS topic. Create a new policy within IAM (IAM → Click Policies → Create policy → JSON) and paste the below policy.

Note: Please update the policy with region & account id.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "sns:Publish",
            "Resource": "arn:aws:sns:<region>:<account_id>:publish-notification"
        }
    ]
}

Attach the policy with the IAM role of the lambda function (IAM → Roles → Open Role → Add permissions → Attach policies → select the policy that you created above → Attach policies)

CloudWatch Trigger | Filter Pattern

Open the Error Handler lambda function, click on Add Trigger, and select CloudWatch Logs from the drop-down.

• Under Log Group, select the CloudWatch log group of the error/keyword producer lambda function.
• Enter the Filter name
• Under Filter pattern, enter the filter value pattern that you want to track. For ex: I want to track the logs that contain keywords like ERROR & LATENCY, so I will enter ?ERROR ?LATENCY

Similarly, you can define different patterns as per your requirement and to learn more about how you can define patterns, please refer to this material

Post configuration, click on Add

Testing

We are all set to test this setup. Go ahead to the error/keyword producer lambda function and click on Test. And in a while, you will receive an email notification, which will look something like this because the logs contain the keyword ERROR & LATENCY

For a detailed step-by-step tutorial and implementation of the above setup, you can refer to the below video.

Finally, I also want to provide the CDK code for this use case and for that please refer to this repository: https://github.com/srcecde/aws-cdk-code/tree/main/lambda-cloudwatch-pattern-notification

If you have any questions, comments, or feedback then please leave them below. Subscribe to my channel for more.