DynamoDB Streams

While working on a backend service, I came across this problem where I had to retrieve a set of data of a specific type, but that type wasn’t having any index and the data was also updating frequently.

Since the data was constantly changing, the first thing that came to mind was to attach a trigger. However, when looking at the documentation on how to add triggers to DynamoDB in AWS, I opted to go with another approach :p

By the time I had time to consider, there were three viable options that were obvious without lengthy consideration

→ Complete scan of the table and filter the results
From the very first moment it entered my head, I knew this is a very inefficient solution

→ Add an Index for that type

There were a couple of reasons I didn’t go along with this idea

• That wasn't a very frequently queried data
• Already had so many indexes so didn't want to add one more just yet before trying other solutions

→ Run a cron-job
A cron-job that will perform a table scan, store the result in a cache, or store it in another table.

A Great man once said, If you can't think of anything else, run a cron-job.

Back in my mind, I wasn’t quite convinced with any of the above solutions and so I gave Dynamo db triggers try. I was aware of Dynamo db streams but never tried them.

DynamoDB Streams are basically triggers like that we have in Relational DBs but the only difference is that it generates a stream of events when there is a data change and you can easily pipe that stream into Kinesis or a Lambda (In my case it was lambda).

AWS made the things look difficult but the process was not complex at all.

All I had to do was the following -

• Setup a stream on the dynamodb table which was just a one click action.
• Create a new lambda function and attach it to the dynamodb stream which was also fairly easy
• Rest was basic stuff of listening the data then processing it and finally saving it to the new table

DynamoDB streams are much more than what I have covered in my use case. You can read more about it here : -https://aws.amazon.com/blogs/database/dynamodb-streams-use-cases-and-design-patterns/