💥 AWS CDK 101 - 🐣 Event source mapping with Cfn property override

🔰 Beginners new to AWS CDK, please do look at my previous articles one by one in this series.

If in case missed my previous article, do find it with the below links.

🔁 Original previous post at 🔗 Dev Post

🔁 Reposted the previous post at 🔗 dev to @aravindvcyber

In this article, we will be making a simple change in our CDK stack which eventually helps us with more optimization and control in processing streams. The high-level configuration which we have made to connect dynamodb streams event source with lambda in the last two articles will be optimized using event source mapping L2 construct with simple override with its cfn L1 construct as well.

Why we need this refactoring 🪲

In our last article, you can find that we are using dynamodb streams from stgMessages to delete objects from S3. The current configuration does not filter or distinguish the event based on the event name at the source, instead, we invoke the lambda unnecessary even for events other than REMOVE like INSERT and we end up, making unwanted batch invocations, which could have otherwise been avoided.

Benefits we achieve 🌋

• In a previous couple of articles we directly interfaced with our dynamodb streams without any filter and our lambda is processing every single one of them and optionally choosing business logic based on the CRUD operation which leads to more number of stream records processed by the lambda eventually.
• The above scenario can be avoided by filtering using the eventName in the dynamodb stream record and selectively using only the most appropriate record needed to be processed in our specific lambda handler.
• Here we not only get a chance to filter with CRUD operation, but also you can filter with the other fields which are part of the record object, this will help us filter these messages to the fullest.
• One more thing we achieve here would be that we can offload the filtering from inside the lambda to the invocation source mapping itself and hence time and cost are saved.

Necessary imports in the stack 📙

Here we will be making use of the below imports in our stack file.

import {
  CfnEventSourceMapping,
  EventSourceMapping,
  StartingPosition,
} from "aws-cdk-lib/aws-lambda";

Previous event source mappings 🌷

Earlier we have used something like the below to configure the source to the lambda. This does not have the filtering enabled and all the streams are used to invoke the lambda, which leads to unwanted executions.

The below L2 construct does not yet provide a provision to add the filter and hopefully, we can expect this soon.

const stgTableSourceProps: DynamoEventSourceProps = {
    startingPosition: lambda.StartingPosition.LATEST,
    batchSize:100,
    maxBatchingWindow: Duration.seconds(60),
}
stgMessageStreamFunc.addEventSource(new DynamoEventSource(stgMessages, stgTableSourceProps));

Generic event source mapping construct 🌰

Due to the shortcomings discussed above, we are going to try another generic L2 construct.

We will replace this with a more generic event source mapping construct as shown below for both the lambda handlers used in the previous 2 articles.
Here we have added a few changes besides filtering.

• startingPosition is changed to StartingPosition.TRIM_HORIZON to get the oldest item first.
• bisectBatchOnError will split the record chunk into half and retry each half so that we can identify and isolate the problematic record from the other records while processing them inside the handler.

const msgDeleteEventSourceMap = new EventSourceMapping(this, "msgDeleteEventSourceMap", {
      target: messageStreamFunc,
      eventSourceArn: messages.tableStreamArn,
      batchSize: 100,
      maxBatchingWindow: Duration.seconds(60),
      startingPosition: StartingPosition.TRIM_HORIZON,
      bisectBatchOnError: true, 
});

msgDeleteEventSourceMap.applyRemovalPolicy(RemovalPolicy.DESTROY);

const stgMsgDeleteEventSourceMap = new EventSourceMapping(this, "stgMsgDeleteEventSourceMap", {
      target: stgMessageStreamFunc,
      eventSourceArn: stgMessages.tableStreamArn,
      batchSize: 100,
      maxBatchingWindow: Duration.seconds(60),
      startingPosition: StartingPosition.TRIM_HORIZON,
      bisectBatchOnError: true, 
});

stgMsgDeleteEventSourceMap.applyRemovalPolicy(RemovalPolicy.DESTROY);

Granting stream read to handler functions 🍄

In the last two articles, we have not performed this explicitly, since the dynamodb event source addition to the lambda construct, automatically applied for the necessary permissions. Whereas in this case, we may have to grant the privileges as shown below.

messages.grantStreamRead(messageStreamFunc);

stgMessages.grantStreamRead(stgMessageStreamFunc);

Cloudformation property override 🌼

Currently, we could not add the filter criteria as this EventSourceMapping L2 construct directly, so here we have to access its L1 construct and add a property override as shown below for both the lambda handlers.

Once do note that there is much scope in the filtering pattern used here where we could include more fields to achieve the desired effect and it is not only limited to eventName only.

const cfnMsgDeleteEventSourceMap = msgDeleteEventSourceMap.node.defaultChild as CfnEventSourceMapping;

cfnMsgDeleteEventSourceMap.addPropertyOverride("FilterCriteria", {
      Filters: [
        {
          Pattern: JSON.stringify({
            eventName: ["INSERT"],
          }),
        },
      ],
});

const cfnStgMsgDeleteEventSourceMap = stgMsgDeleteEventSourceMap.node.defaultChild as CfnEventSourceMapping;

cfnStgMsgDeleteEventSourceMap.addPropertyOverride("FilterCriteria", {
      Filters: [
        {
          Pattern: JSON.stringify({
            eventName: ["REMOVE"],
          }),
        },
      ],
});

Issue while deploying the stack 🍭

You may occasionally get an error failure to deploy the stack, by then it is necessary for us to manually delete the event source mapping in the lambda either with the AWS console or using the AWS CLI as shown below.

The stack named CommonEventStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "The event source arn (" arn:aws:dynamodb:ap-south-1:*****:table/stgMessagesTable/stream/2022-05-21T03:39:54.488 ") and function (" CommonEventStack-stgMessageStreamFuncF95CB7BF-877BwMFlctoc ") provided mapping already exists. Please update or delete the existing mapping with UUID 43eccb79-7a6f-4fd0-82f1-c176cff317da

AWS console to delete the mapping 🎏

AWS cli to delete mapping 🍣

Verify the event source mapping if it is existing as shown below

aws lambda delete-event-source-mapping --uuid ********

Then we can delete it by executing the below command

aws lambda delete-event-source-mapping --uuid ********

Conclusion 🍤

Here we have demonstrated two things as shown below.

• Ability to filter streams inside the generic event source mapping which is not only limited to dynamodb stream, this has a lot of other applications to be used similarly for kinesis streams, SQS source, and a lot more sources.
• Additionally we have learned how we can use the L1 construct for an L2 resource and override certain properties, this will have a lot of use cases beyond this example.

We will be adding more connections to our stack and making it more usable in the upcoming articles by creating new constructs, so do consider following and subscribing to my newsletter.

⏭ We have our next article in serverless, do check out

https://dev.to/aravindvcyber/aws-cdk-101-graphql-using-appsync-with-dynamodb-16dd

🎉 Thanks for supporting! 🙏

Would be great if you like to ☕ Buy Me a Coffee, to help boost my efforts 😍.

🔁 Original post at 🔗 Dev Post

🔁 Reposted at 🔗 dev to @aravindvcyber

🏄 AWS CDK 101 - 🐲 GraphQL using AppSync with dynamodb
{ by @Aravind_V7 }

Checkout more such in my pagehttps://t.co/CuYxnKr0Ighttps://t.co/phzOKFlXXO#aws #typescript #graphql #dynamodb #awscdk https://t.co/yeqeLZQt8A

— Aravind V (@Aravind_V7) May 30, 2022