The AWS SNS (Simple Notification Service) is a fully managed Pub/Sub messaging service that provides high-throughput, durable, push-based notification service supporting both A2A (Application to Application) and A2P (Application to Person) communication.
The A2A architecture provides messaging capability between the distributed services, EDA (Event-Driven Architecture), and microservices. In contrast, the A2P architecture provides messaging capability to the end users and customers over SMS, Email, and push notifications on the mobile apps.
Overview
The AWS SNS has wide integration support with other services of AWS like AWS Kinesis Data Firehose, AWS Lambda, and of course our very own AWS SQS (Simple Queuing Service). It also provides a loosely coupled architecture in our system.
The AWS SNS in conjunction with other services like AWS SQS usually allows our system to fan out messages coming from a publisher to different subscribers via the SNS Topic.
A typical fan-out architecture looks like below where a Message, M1 arrives in both the SQS Queues for the parallel processing of jobs without any message filter:-
Adding Message Filtering Strategy in SNS Messages
The AWS SNS offers two types of message filter strategies:-
- Attribute-based message filtering
- Payload(or Message body)-based message filtering
Attribute-based message filtering is not a new thing in SNS and has been in the game since the beginning. It is the most basic version of the message-filtering strategy that AWS SNS offers and is completely free. Here a message is filtered using the message attribute itself that we need to configure in the SNS Topic subscription.
Payload (or Message body)-based message filtering is the most advanced and will be looked at in detail in this blog post.
The latest enhancement in SNS introduces a payload-based message filtering feature that grants subscribers the ability to articulate their subscription filter policies based on the message contents. This added functionality is particularly advantageous for event-driven architectures (EDA) and cross-account workloads, where subscribers may lack influence over a publisher's event attributes. With payload-based message filtering, you now have an uncomplicated, no-code alternative to proactively prevent undesired data from reaching subscriber systems, streamlining their code and diminishing costs linked to downstream compute infrastructure. This innovative message filtering option applies to both SNS Standard and SNS FIFO topics, specifically designed for JSON message payloads.
The architecture of a payload-based message filtering using AWS SNS
In the above example, the loan applications are processed by a bank through their digital platform where every document first arrives in the S3 bucket which then triggers the ObjectCreated:Put event of S3 sending the notification to AWS SNS. Because of the payload-based message filter policy configuration in AWS SNS, the message is then forwarded to the appropriate SQS queue for the downstream application to process the loan applications.
Wondering how every message arrives in the right queue?
(A) When the data arrives in the S3 bucket then S3's ObjectCreated event during the Put operation is triggered to SNS
{
"Records": [{
"eventVersion": "2.1",
"eventSource": "aws:s3",
"awsRegion": "us-east-1",
"eventTime": "2024-01-21T10:40:23.395Z",
"eventName": "ObjectCreated:Put",
"userIdentity": {
"principalId": "AWS:JKHKJHKSD44T44646:vinod-admin"
},
"requestParameters": {
"sourceIPAddress": "192.456.564.19"
},
"responseElements": {
"x-amz-request-id": "SHDJKDDK43894K",
"x-amz-id-2": "DSk0+DKLFSKLFFSF94KLTJEJTLJKLDGDGDGD/...jIr3"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "mydigitalbank-loan-processing",
"bucket": {
"name": "mydigitalbank2024",
"ownerIdentity": {
"principalId": "11313131313"
},
"arn": "arn:aws:s3:::mydigitalbank2024"
},
"object": {
"key": "home-loan-20240121.xml",
"size": 17,
"eTag": "9842424kjdskjfsfk94824844hj",
"sequencer": "87535jk35hkjhjk"
}
}
}]
}
(B) We have the Message filter policies configured in the SNS Subscription for both Home Loan Applications & Vehicle Loan Applications as shown below. The filter policy for home loan SQS queue subscription is as below:-
{
"Records": {
"s3": {
"object": {
"key": [{
"prefix": "home-"
}]
}
},
"eventName": [{
"prefix": "ObjectCreated:"
}]
}
}
For vehicle loan SQS queue subscription, the filter policy is:-
{
"Records": {
"s3": {
"object": {
"key": [{
"prefix": "vehicle-"
}]
}
},
"eventName": [{
"prefix": "ObjectCreated:"
}]
}
}
Every Put object event received in the SNS Topic is then scanned based on the filter policy, e.g. - "prefix": "home-" will forward the message to the home loan queue and anything with the prefix in the file name as vehicle- to the vehicle loan queue.
The AWS SNS is intelligent enough and will not forward any event to these queues if the operation is not Put.
Estimated Costs
Let's look at the estimated cost now. The provisioning of SNS Topic itself is free however there are some other costs associated with it like the number of the SNS API request, the size of the payload, etc. Unlike, the attribute-based filter policy, the payload-based filter policy is not free and costs about $0.09 per GB of scanned payload data.
I have assumed 1 million requests per month, 100KB payload size, 2 subscriptions, and 5GB of outbound payload data scanned per month (As shared earlier, you will be charged for the amount of data scanned for filtering if you are using payload based filtering). The cost would be around 2.67 USD per month.
Conclusion
In this blog post, we have seen how payload-based message filtering works for AWS SNS providing event routing to different SQS queues based on the filter policies applied.
Top comments (0)