Dynamic AWS S3 Notification Configuration using Terraform

It is the simple stuff that you expect to be, well, simple. Until they aren’t.

So the situation is like this:

• I have an aws s3 bucket
• I have multiple dynamically created consumers for objectCreated event from this bucket
• Each consumer subscribes to specific prefix on this bucket. i.e. service 1 will listen to new files on srv1/ path only. service2 will listen to new files on srv2/ path only and so on.

Since one SNS cannot differentiate between s3 prefixes, I need to have multiple SNS topics — one for each service

Doing it manually — it is super easy, just create another pair of sns/sqs and I’m ready to go. But manual process belongs to the dark ages of 1995, I need to do it automatically using IaaC, Terraform in this case.

Easy, let’s do it!

So there I went, knowing exactly what I need to do. Create a terraform file to manage the s3 bucket, sns topics and their policies, and attach each sns topic to the bucket with correct prefix (/srv1, /srv2, etc…)

Like any good programmer — I wanted to copy paste stuff just to get it going, so I googled “aws s3 sns notification terrafom” and clicked the first link (which super convinently was terraform documentations). This is the link:

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_notification

A simple note on that page caught my eyes and broke my heart:

What does that all mean??

Not so easy…

well, it means that I can attach one aws_s3_bucket_notification to the bucket. It also means, I have to know in advance all topics I’m going to have since I have to write them as part of the terraform code.

But I said before, I have a dynamic list of consumers, I can’t know all of them in advance, am I stuck?

Creativity time

So after thinking about it for a while, I tried playing with the aws api, fetching configuration and updating it. So if I can do it using the API and get what I want, I surely can automate it!

It took few trials and errors and I wrote a python script which accepts few parameters — bucket name, sns arn, and the relevant prefix. The scripts get the current configuration from aws API, check if the subscription already exists and adds it if it doesn’t.

In case someone needs the python file, I’d be happy to share it

Now I just need to run the script as part of Terraform.

Terraform Null resource and local exec provisioner

The null_resource resource implements the standard resource lifecycle but takes no further action.

The triggers argument allows specifying an arbitrary set of values that, when changed, will cause the resource to be replaced.

Basically null_resource can do anything, it has no backing resource in the cloud, it has a trigger and a provisioner which runs once the trigger changed.

So I’ve added a null resource wherever I have an SNS topic, pass the bucket name, the sns arn and the prefix, and that’s it!

provider "null" {}

resource "null_resource" "s3_object_created_subscription" {

  triggers = {
     prefix = var.prefix
  }

  provisioner "local-exec" {
     command = "python ${var.bucket_name} ${var.sns_arn} ${var.prefix}"
  }
}

This little piece of code can run from many terraform state files and attach an sns to a bucket with whatever prefix and how many prefixes I need. Like any terraform resource it has state, so it will run only once, unless you change the prefix.

Conclusion

I know that’s a hack. It might break since s3 API is eventually consistent. But it does not bother me since I don’t have a massive amount of parallel changes.

The reasoning behind it, vs using multiple buckets, is the bucket limit I tackled. Amazon currently allows only 1000 buckets per account. That’s actually a lot of buckets. But once you have multiple services, let’s say 30, each one requires at least 1 bucket and you have dynamic test environments, you can easily see how you reach this limit.