Debugging SQS subscription issues to topics

When we want to subscribe an SQS queue to an SNS topic, it must have the relevant permissions to send messages to the queue. How we add it as a subscriber to the topic via the AWS Console matters because the SQS access policy doesn't always contain the correct permission statement.

1. The scenario

Say we want to build a fanout serverless architecture.

The key service that publishes messages to its subscribers is Simple Notification Service (SNS). We can add multiple different subscribers to a topic, for example, email addresses, phone numbers, or SQS queues.

One use case of queues is for buffering purposes. When the consumer can process the messages slower than the producer sends them, queues can temporarily store them until the consumer eventually processes them.

In AWS, everything is about permissions. Even AWS services can't call other services if the relevant permissions are missing.

When we use CDK to create our infrastructure, the framework will take care of the permissions.

But what if we need to put together a quick demo in the Console? It should work, too, right?

2. Ways to solve the task

This blog post will not explain how we create a topic or a queue. Please see the references section at the end if you need help.

The task is to subscribe the queue to the topic. To continue with the serverless momentum, assume that a Lambda function polls the queue and does something with the message.

We seem to have two options.

2.1. Subscribe to the topic from the SNS page

We select the topic and choose Create subscription in the Subscriptions section. Select Amazon SQS for Protocol, specify the ARN of the queue, and we should be good to go.

Well, not really. Chances are that it will not work. At the end of a long day, one can spend precious minutes (hours) debugging why the Lambda function is not running after we have published a message on the topic.

The reason is the SNS will not add the relevant permissions to the SQS queue's access policy. The access policy is a resource-based policy, and it must contain the principal element that specifies who or which service can send messages to the queue. SNS is not an identity, so we should use the access policy here.

By default, the access policy of the queue test-queue looks like this:

{
  "Version": "2012-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__owner_statement",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:root"
      },
      "Action": "SQS:*",
      "Resource": "arn:aws:sqs:eu-central-1:123456789012:test-queue"
    }
  ]
}

We might first think that the policy should suffice because it gives the account full permissions on the queue. (Account and root user are the same when specified in the Principal element.) But this policy only delegates authority to the account, and will not give permission to the SNS topic. We still need to ensure that the SQS access policy explicitly allows the topic to send messages to the queue.

The problem is that the Console will create the subscription as expected. We can find it under the Subscriptions section, but the SQS access policy will not receive permission. If we forget about the mandatory explicit permission requirements in AWS, we can scratch our heads and stare at the monitor thinking why our simple architecture is not working.

I haven't seen this issue with other protocols, and it works well with a direct Lambda or email subscription.

2.2. Try from the SQS side

Luckily the Console allows us to subscribe test-queue to the topic from the SQS service page, too.

The process is very similar. We click on test-queue, and then SNS subscriptions.

When we click Subscribe to Amazon SNS topic, we will select the topic from the dropdown, and done!

It will work, because the queue's access policy will contain the following statement:

{
  "Sid": "topic-subscription-arn:aws:sns:eu-central-1:123456789012:test-sns-topic",
  "Effect": "Allow",
  "Principal": {
    "AWS": "*"
  },
  "Action": "SQS:SendMessage",
  "Resource": "arn:aws:sqs:eu-central-1:123456789012:test-queue",
  "Condition": {
    "ArnLike": {
      "aws:SourceArn": "arn:aws:sns:eu-central-1:123456789012:test-sns-topic"
    }
  }
}

This statement allows test-sns-topic to SendMessages to `test-queue. This permission was missing from the access policy when we tried to add the subscription from the SNS service page.

3. Summary

Everything in AWS is about permissions.

When we want to add an SQS queue as a subscriber to an SNS topic in the Console, we can do it in one of two ways. The subscription process will not always add the necessary permissions to the access policy of the queue. This way, it will not receive the messages we publish on the topic.

But we can perform the same action from the SQS service page. The permission will be there, and the flow will work as expected.

4. Further reading

Creating an Amazon SNS topic - Step-by-step guide to create an SNS topic

Create a queue (console) - Same for SQS queues