AWS Simple Queue Service (SQS) 101 with a demo using AWS CLI

Table of Contents

1. Introduction
2. Standard vs FIFO Queue
3. Queue Configuration
4. Create a SQS queue
5. Create a Cloud9 instance
6. Challenge
7. Summary
8. Referrals

Introduction

In this blog we are going to create a SQS queue using the AWS Console and then interact with it using AWS CLI, bash and jq.

What is SQS?

Amazon Simple Queue Service (SQS) lets you send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.

Some of the use cases of SQS

• Increase application reliability and scale.
• Decouple microservices and process event-driven applications.
• Ensure work is completed cost-effectively and on time.
• Maintain message ordering with deduplication

Standard vs FIFO Queue

SQS offers two types of queues – Standard & FIFO queues
Some of the major differences between the standard and the FIFO queues are:

https://dev.to/aws-builders/sagemaker-and-amazon-stack-2jaa

Message Order

• Standard queues provide best-effort ordering which ensures that messages are generally delivered in the same order as they are sent. Occasionally (because of the highly-distributed architecture that allows high throughput), more than one copy of a message might be delivered out of order.
• FIFO queues offer first-in-first-out delivery and exactly-once processing: the order in which messages are sent and received is strictly preserved.

Delivery

• Standard queues guarantees that a message is delivered at least once and duplicates can be introduced into the queue.
• FIFO queues ensure a message is delivered exactly once and remains available until a consumer processes and deletes it. The duplicates are not introduced into the queue.

Transactions Per Second (TPS)

• Standard queues allow nearly-unlimited number of transactions per second.
• FIFO queues are limited to 300 transactions per second per API action. However It can be increased to 3000 by using batching.

Queue Configuration

Note: The following is from AWS Console

Visibility timeout: Should be between 0 seconds and 12 hours (default: 30 seconds).

• Visibility timeout sets the length of time that a message received from a queue (by one consumer) will not be visible to the other message consumers.
• The visibility timeout begins when Amazon SQS returns a message. If the consumer fails to process and delete the message before the visibility timeout expires, the message becomes visible to other consumers. If a message must be received only once, your consumer must delete it within the duration of the visibility timeout.
• The default visibility timeout setting is 30 seconds. This setting applies to all messages in the queue. Typically, you should set the visibility timeout to the maximum time that it takes your application to process and delete a message from the queue.

Message retention period: Should be between 1 minute and 14 days (default: 4 days).

• The message retention period is the amount of time that Amazon SQS retains a message that does not get deleted. Amazon SQS automatically deletes messages that have been in a queue for more than the maximum message retention period. The default retention period is 4 days. The retention period has a range of 60 seconds to 1,209,600 seconds (14 days).
• The expiration of a message is always based on its original enqueue timestamp. When a message is moved to a dead-letter queue, the enqueue timestamp remains unchanged. For example, if a message spends 1 day in the original queue before being moved to a dead-letter queue, and the retention period of the dead-letter queue is set to 4 days, the message is deleted from the dead-letter queue after 3 days. For this reason, we recommend that you always set the retention period of a dead-letter queue to be longer than the retention period of the original queue.

Delivery delay: Should be between 0 seconds and 15 minutes (default: 0 seconds).

• If your consumers need additional time to process messages, you can delay each new message coming to the queue. The delivery delay is the amount of time to delay the first delivery of each message added to the queue. Any messages that you send to the queue remain invisible to consumers for the duration of the delay period. The default (minimum) delay for a queue is 0 seconds. The maximum is 15 minutes.
• For standard queues, the per-queue delay setting is not retroactive; changing the setting doesn't affect the delay of messages already in the queue.
• For FIFO queues, the per-queue delay setting is retroactive; changing the setting affects the delay of messages already in the queue.

Maximum message size: Should be between 1 KB and 256 KB (default: 256 KB).

• You can set the maximum message size for your queue. The smallest supported message size is 1 byte (1 character). The largest size is 262,144 bytes (256 KB). To send messages larger than 256 KB, you can use the Amazon SQS Extended Client Library for Java (https://github.com/awslabs/amazon-sqs-java-extended-client-lib). This library allows you to send an Amazon SQS message that contains a reference to a message payload in Amazon S3. The maximum payload size is 2 GB.

Receive message wait time: Should be between 0 and 20 seconds (default: 0 seconds).

• The receive message wait time is the maximum amount of time that polling will wait for messages to become available to receive. The minimum value is zero seconds and the maximum value is 20 seconds.
• Long polling helps reduce the cost of using Amazon SQS by eliminating the number of empty responses (when there are no messages available for a ReceiveMessage request) and false empty responses (when messages are available but aren't included in a response). If a receive request collects the maximum number of messages, it returns immediately. It does not wait for the polling to time out.
• If you set the receive message wait time to zero, the receive requests use short polling.

Demo

Let's get started with the demo.

Create a SQS queue

1. Navigate to Simple Queue Service and click Create queue
2. Enter Name as MyMessages, leave the rest as defaults and click Create queue

SQS_1

SQS_2

SQS_3

SQS_4

Cloud9

1. Navigate to Cloud9 > Create Environment
2. Enter Name as SQS and click Create
3. Click Open under AWS Cloud9 > Environments

Let's check aws cli version.

Sri:~/environment $ aws --version
aws-cli/1.19.112 Python/2.7.18 Linux/4.14.301-224.520.amzn2.x86_64 botocore/1.20.112

jq is not installed by default on Cloud9.

Sri:~/environment $ jq
bash: jq: command not found

Install jq by executing the following command.

sudo yum install jq -y

How to check if jq is installed.

Sri:~/environment $ rpm -qa|grep jq
jq-1.5-1.amzn2.0.2.x86_64

jq -help

Sri:~/environment $ jq
jq - commandline JSON processor [version 1.5]
Usage: jq [options] <jq filter> [file...]

        jq is a tool for processing JSON inputs, applying the
        given filter to its JSON text inputs and producing the
        filter's results as JSON on standard output.
        The simplest filter is ., which is the identity filter,
        copying jq's input to its output unmodified (except for
        formatting).
        For more advanced filters see the jq(1) manpage ("man jq")
        and/or https://stedolan.github.io/jq

        Some of the options include:
         -c             compact instead of pretty-printed output;
         -n             use `null` as the single input value;
         -e             set the exit status code based on the output;
         -s             read (slurp) all inputs into an array; apply filter to it;
         -r             output raw strings, not JSON texts;
         -R             read raw strings, not JSON texts;
         -C             colorize JSON;
         -M             monochrome (don't colorize JSON);
         -S             sort keys of objects on output;
         --tab  use tabs for indentation;
         --arg a v      set variable $a to value <v>;
         --argjson a v  set variable $a to JSON value <v>;
         --slurpfile a f        set variable $a to an array of JSON texts read from <f>;
        See the manpage for more options.

Send a SQS message using AWS CLI

In order to send a message to SQS, we need queue-url.
We could either hard code the queue-url or retreive the queue-url and then pass it to send-message, receive-message or delete-message

# queueurl is required to send or receive messages from SQS

# Note: AWS CLI and boto3 need to use legacy endpoint
# If we use the AWS CLI or SDK for Python, we need to use the legacy endpoints from https://docs.aws.amazon.com/general/latest/gr/sqs-service.html.
# Otherwise, you will get the following error
# botocore.exceptions.ClientError: An error occurred (InvalidAddress) when calling the ReceiveMessage operation: The address https://eu-central-1.queue.amazonaws.com/ is not valid for this endpoint.

export queueurl=$(aws sqs get-queue-url --queue-name "MyMessages"|jq ".QueueUrl"|sed "s/sqs.us-east-1.amazonaws.com/queue.amazonaws.com/g"|sed 's/"//g')
echo "queueurl:" $queueurl

Sri:~/environment $ aws sqs send-message --queue-url $queueurl --message-body Message --region us-east-1
{
    "MD5OfMessageBody": "4c2a8fe7eaf24721cc7a9f0175115bd4", 
    "MessageId": "614ec45a-82b8-4788-aed6-6f3d80051780"
}

Sri:~/environment $ aws sqs receive-message --queue-url $queueurl
{
    "Messages": [
        {
            "Body": "Message", 
            "ReceiptHandle": "AQEBxxUZvU5S1OP2VFBIY5bpg267ZQvSkX3n7jfV505VumHa2JCHo5BR/trRSCKOlhcLSyvJzBUBhq9TPHlQ4F4nQu839HruurRMMYCaaZBiiEdFnADm3EwEi8+Hj9YsmmGf0YL4ftlTyMHnJFjL9l06e7qZZyV8lWKv+q/mnEU8v69AumtvcS7OhmgAtUzQUqG+4/enJgt3WrN1eJiJRlmiUbDpV3VCGr7wQbU5C+GAmYjVNs+vNrMSzOceKFs3XQH6vwhNnnp1d3dJZzn+JEjpfx8g6INrq1LYShgg6cTUVygc2jXuQNBoSHP4Jx6szscdLm2P4gUD515UJ948nX/Zlsu90GZtfyGY5S4wzgl9oYpJcPLRV7z6ISnte0WKuK9cHPCEzmyGeERS1yC7YKXRNA==", 
            "MD5OfBody": "4c2a8fe7eaf24721cc7a9f0175115bd4", 
            "MessageId": "614ec45a-82b8-4788-aed6-6f3d80051780"
        }
    ]
}

👉 If you run the command again with in the Visibility timeout of 30 secs, SQS will not send the message again.
However if the message is not processed and deleted by the consumer, the message becomes visible to other consumers. If a message must be received only once, the consumer must delete it within the duration of the visibility timeout.

Sri:~/environment $ aws sqs receive-message --queue-url $queueurl
Sri:~/environment $ 

So in order to delete the message, we need to retreive the receipt-handle and then pass the --receipt-handle to sqs delete-message
Since the output message is a JSON message and we only need receipt-handle, we can use jq, which is like sed for JSON data - you can use it to slice, filter, map and transform structured data in the same way that you would with sed, awk, grep and other linux tools.

What does get-queue-url do?
It retreives QueueUrl and then replaces the endpoint with a legacy endpoint and also removes double quotes from the QueueUrl.

The following code snippet retrieves and deletes the message.

export queueurl=$(aws sqs get-queue-url --queue-name "MyMessages"|jq ".QueueUrl"|sed "s/sqs.us-east-1.amazonaws.com/queue.amazonaws.com/g"|sed 's/"//g')
echo "queueurl:" $queueurl

# receiptHandle is required to delete the message from SQS after the message is retreived
export receiptHandle=$(aws sqs receive-message --queue-url $queueurl|jq .Messages[0].ReceiptHandle|sed 's/"//g')
echo "receiptHandle:" $receiptHandle

#echo -e "aws sqs delete-message --queue-url $queueurl --receipt-handle $receiptHandle --region us-east-1"
aws sqs delete-message --queue-url $queueurl --receipt-handle $receiptHandle --region us-east-1;

Let's send a 100 messages to the SQS queue, retrieve them and then delete the messages using bash scripts.
Open up two terminal windows on Cloud9 and use one to execute ./sendMessages.sh and the other for ./receiveMessages.sh.

sendMessages.sh

#!/bin/bash

# queueurl is required to send or receive messages from SQS

# Note: AWS CLI and boto3 need to use legacy endpoint
# If we use the AWS CLI or SDK for Python, we need to use the legacy endpoints from https://docs.aws.amazon.com/general/latest/gr/sqs-service.html.
# Otherwise, you will get the following error
# botocore.exceptions.ClientError: An error occurred (InvalidAddress) when calling the ReceiveMessage operation: The address https://eu-central-1.queue.amazonaws.com/ is not valid for this endpoint.

queueurl=$(aws sqs get-queue-url --queue-name "MyMessages"|jq ".QueueUrl"|sed "s/sqs.us-east-1.amazonaws.com/queue.amazonaws.com/g"|sed 's/"//g')
echo "queueurl:" $queueurl

echo "aws sqs send-message --queue-url $queueurl --message-body Message --region us-east-1"
for i in {1..100} ; do aws sqs send-message --queue-url $queueurl --message-body "Message-$i" --region us-east-1 ; done;

receiveMessages.sh

#!/bin/bash

# queueurl is required to send or receive messages from SQS

# Note: AWS CLI and boto3 need to use legacy endpoint
# If we use the AWS CLI or SDK for Python, we need to use the legacy endpoints from https://docs.aws.amazon.com/general/latest/gr/sqs-service.html.
# Otherwise, you will get the following error
# botocore.exceptions.ClientError: An error occurred (InvalidAddress) when calling the ReceiveMessage operation: The address https://eu-central-1.queue.amazonaws.com/ is not valid for this endpoint.

queueurl=$(aws sqs get-queue-url --queue-name "MyMessages"|jq ".QueueUrl"|sed "s/sqs.us-east-1.amazonaws.com/queue.amazonaws.com/g"|sed 's/"//g')
echo "queueurl:" $queueurl

for i in {1..100};
  do
        # receiptHandle is required to delete the message from SQS after the message is retreived
        receiptHandle=$(aws sqs receive-message --queue-url $queueurl|jq .Messages[0].ReceiptHandle|sed 's/"//g')
        #echo "receiptHandle:" $receiptHandle

        #echo -e "aws sqs delete-message --queue-url $queueurl --receipt-handle $receiptHandle --region us-east-1"
        aws sqs delete-message --queue-url $queueurl --receipt-handle $receiptHandle --region us-east-1;

        echo "Sleep for 1 second..."
        sleep 1
done;

Challenge

There are some improvements which we can make to the above. I will leave this as a challenge for now.
Please let me know in the comments if you have managed to solve it.
I will provide an update in a comment at some point.

1. We can use --max-number-of-messages (integer).

   The maximum number of messages to return. Amazon SQS never returns more messages than this value (however, fewer messages might be returned). Valid values: 1 to 10. Default: 1.

2. jq query in the command also need to be updated to retrieve an array of --receipt-handle.

3. bash script also need to store the returned array and loop through the array to delete all the messages returned by --max-number-of-messages.

Clean Up

1. Delete the MyMessages Queue.
2. Terminate Cloud9 EC2 instance.

Summary

• We learned about SQS.
• We also learned about bash scripting and jq.

See you next time 👋

Referrals

• Amazon SQS API Reference
• botot3 with sqs: the address 'QueueUrl' is not valid for this endpoint
• Amazon Simple Queue Service endpoints and quotas