Challenge 2: Add SQS + S3 Event Triggers for Lambda Apps

Quick Recap

In the previous post, we have created a simple loyalty application. It can create and display loyalty cards.

Your next challenge

The loyalty application you have made was a big hit! New user's enrollment is now at an all-time high. However, your old users are complaining. They still are using the old paper-based system and they refused to enroll as new users unless the points from their old loyalty card can be migrated to the new loyalty card system that you created.

The restaurants have a backlog of 100,000 paper loyalty cards to be migrated to the new system. With a booming business, they can only do 100 a day and your loyal customers are already complaining!

To help them, your boss volunteered you (ugh, I know right) to create a new module for the application. The staff will just encode the paper loyalty cards in a CSV file. And you will just collect the CSV file and upload it to S3. The new module will read the contents and create a unique loyalty card for each one, and still maintain the points.

Specification

In order to do this, you will have to:

1. Create an S3 bucket in your AWS Account
2. Create a new lambda function (L4). Lambda functions are off by default and are triggered by events. In the previous challenge, it was triggered by an API call. For this function, it is triggered by a file upload to an S3 bucket.
3. Once the file is uploaded in your S3 bucket, a Lambda function is triggered. In the event object of the Lambda function, the S3 bucket and the file_key of the file uploaded is given. Use this information to download and read the file line by line.
4. Each line is created one job in an SQS queue. An SQS queue is sort of like a line in a bank. If we have a file with 10,000 rows, we would have 10,000 jobs lined up for processing in the SQS queue. Each job is represented by a JSON object sent to SQS. The contents of the JSON object is the values of the row. A sample job would look like {"card_number": "3000-4000-5000-6000", "first_name": "Raphael", "last_name": "Jambalos", "points": 500}
5. At the end of the line is a "teller" processing the requests one after the other. For this, we create another Lambda function (L5). L5 takes a request from SQS. It receives a JSON object that has information about the job. With this information, create a loyalty card in your DynamoDB database.

Hints

1. For L4, you can begin by connecting the L4 to the S3 bucket by setting up its event property. For an example, start here
2. For L5, you can begin by configuring the L5 function to connect to the SQS queue. For an example, you can start here. This example automatically triggers a Lambda function when a request arrives in SQS.

Resources

• Serverless Example for SQS (L5)
  • NodeJS: https://github.com/serverless/examples/tree/master/aws-node-sqs-worker
  • Python: https://github.com/serverless/examples/tree/master/aws-python-sqs-worker
• Serverless Example for S3 (L4)
  • NodeJS: https://github.com/serverless/examples/tree/master/aws-node-s3-file-replicator

Show off your work!

Comment a screencap of your work below. Or better yet, create a blog post here in dev.to explaining how you did it.

If you have any questions or are stuck somewhere, comment below or send me a pm, and I'd be happy to help you.

Photo by Tim Gouw on Unsplash