AWS Exception: PolicyLengthExceededException

The other day we'd an issue at work, where we were getting a PolicyLengthExceededException for our lambda permissions, which we were attaching programmatically during the runtime, every-time we were creating a resource. This wasn't a recurring issue but rather an intermittent one.

Upon further debugging, I came to know that the resource based policy for a lambda function is limited at 20KB.
The policy we were creating was overshooting the limit by 20 bytes. It's not much, but we've to abide by the limits set.

To my astonishment, the parameters being passed on to the policy were more or less same as compared to those from the other policies that we'd earlier attached to the same lambda.

So I'd now 2 genuine concerns:

• If there was really any limit breach, this wouldn't have been an intermittent issue.

• How did the earlier policies get added?

Upon some further R&D, found out that the policy size limit wasn't something that was being checked very rigorously. Instead the same was only being checked once you'd a large chunk of policies with similar names, performing similar jobs and which could have otherwise been clubbed into one using wildcards.

Solution:

Added an aws_lambda_permission component to our terraform script which creates the corresponding permission every-time the lambda function is created.

Note that I've generalised the source_arn using wildcard so that we don't have to create a policy every-time we create a resource and thus solving the problem of too many duplicate/similar policies.

resource "aws_lambda_permission" "allow_event_bridge_rule_permission" {
  statement_id  = "AllowLambdaInvocationFromEventBridge"
  action        = "lambda:InvokeFunction"
  function_name = "${var.func_name}"
  principal     = "events.amazonaws.com"
  source_arn    = "arn:aws:events:${var.aws_region}:${var.aws_account_id}:rule/model_*_name"
}

Note: Even after adding the above code to the tf script, still we were getting the same exception while executing the script. I removed all the existing duplicate policies and re-ran the script and it went fine.

Even if you're manually creating the policies from the console/CLI its the best practice to use wildcards in the source_arn so as to minimise the number of policies attached to a resource.

Hope this helps you!