Context
My friend puts the following Error Message on a Slack Channel that our Team monitors and provides assistance in
coolLambdaPermission | UPDATE_FAILED | Resource handler returned message: "The provided principal was invalid. Please check the principal and trying again" (Service: Lambda, Status Code: 400 ...)
He also provides some details on the Jenkins Job that showed this error message, and wanted to get some help. I was free so I jumped into resolving it.
Issue
The AWS::Lambda::Permission
CloudFormation Resource would fail to update within the CloudFormation Stack, due to the Principal provided being invalid (based on the error message), even though the Principal provided within the CloudFormation Template was verified as valid.
Debugging
The first step I took was checking the AWS CloudFormation Events Tab to see the error message, and as seen the AWS::Lambda::Permission
CloudFormation Resource failed to update.
Looking into the CloudFormation Template that initiated the Stack Update, I verified the the AWS::IAM::Role
used within the AWS::Lambda::Permission
Principal attribute existed and was valid.
I checked the Policy Permissions and Trust Relationship for the given IAM Role, and compared it with similar CloudFormation Stacks that use the same IAM Role for the AWS::Lambda::Permission
resource, but things looked fine.
Quick Google-Fu searches later, I didn't really get anywhere. My Manager took a quick look and mentioned that the Lambda Permission has an IAM Role associated, which does not exist since it was deleted, which might be causing issues for CloudFormation Stack Updates 🤷
🤔 Since the CloudFormation Template changes involved the following
# serverless.yml
coolLambdaPermission:
Type: "AWS::Lambda::Permission"
Properties:
FunctionName: ...
Action: ...
- Principal: ${self:custom.deletedRole}
+ Principal: ${self:custom.validRole}
I would have expected the new Valid Role
to replace the old Deleted Role
when the CloudFormation Stack is being updated, which it's exactly trying to do.
For a Sanity check, I went into the Lambda Resource, associated with the Lambda Permission, available under Lambda -> Configuration Tab -> Permissions Tab -> Resource-based policy statements
and verified the Deleted Role
is present in the Principal, but the actual IAM Role is deleted.
I concluded that it must be the AWS::Lambda::Permission
Physical ID
that might be causing the error and hence suggested my friend to create a PR that updates that resource
# serverless.yml
- coolLambdaPermission:
+ coolLambdaPermissions:
Type: "AWS::Lambda::Permission"
Properties:
FunctionName: ...
Action: ...
- Principal: ${self:custom.deletedRole}
+ Principal: ${self:custom.validRole}
This would mean that CloudFormation would create a new AWS::Lambda::Permission
Resource with a new Physical ID
, and the update would go through. Nope ❌, that did not work, It failed with the same error message.
The provided principal was invalid. Please check the principal and trying again
❓ My next question was, Can I manually add in the Valid Role to the Lambda Permission?
So I went into the Lambda Console, and attempted to manually add the Valid Role
, and it failed with the same error. This is actually great.
Next step was to manually remove the Deleted Role
Principal within the Lambda Console, and try to add the Valid Role
Principal again, and that worked! 🎉
This seems like the Lambda wants all the Principals provided to be Valid, and only then would it allow for more Principals to be added in.
I suggested my friend to do the following:
- Create a PR that removes the
AWS::Lambda::Permission
altogether
# serverless.yml
- coolLambdaPermission:
- Type: "AWS::Lambda::Permission"
- Properties:
- FunctionName: ...
- Action: ...
- Principal: ${self:custom.deletedRole}
- Create a follow up PR that adds the
AWS::Lambda::Permission
Resource back
# serverless.yml
+ coolLambdaPermission:
+ Type: "AWS::Lambda::Permission"
+ Properties:
+ FunctionName: ...
+ Action: ...
+ Principal: ${self:custom.validRole}
✅ That worked and the CloudFormation Stack was updated successfully
Solution
If you have the AWS::Lambda::Permission
CloudFormation Resource, which contains a Principal that does not exist anymore, and if you want to update the Principal attribute with a Valid Role
, you'll need to remove the AWS::Lambda::Permission
CloudFormation Resource first, and then add it back in with the Valid Role
i.e.
❌ Replacing the Principal within the CloudFormation Template and deploying it would not work(atleast for me)
# serverless.yml
- coolLambdaPermission:
+ coolLambdaPermissions:
Type: "AWS::Lambda::Permission"
Properties:
FunctionName: ...
Action: ...
- Principal: ${self:custom.deletedRole}
+ Principal: ${self:custom.validRole}
Rather,
- ✅ Removing the CloudFormation Resource and deploying the CloudFormation Template
# serverless.yml
- coolLambdaPermission:
- Type: "AWS::Lambda::Permission"
- Properties:
- FunctionName: ...
- Action: ...
- Principal: ${self:custom.deletedRole}
- ✅ Adding back the CloudFormation Resource with the new Principal and then deploying the CloudFormation Template
# serverless.yml
+ coolLambdaPermission:
+ Type: "AWS::Lambda::Permission"
+ Properties:
+ FunctionName: ...
+ Action: ...
+ Principal: ${self:custom.validRole}
Notes
- I would have assumed that changing the Principal Value within the
AWS::Lambda::Permission
Resource would have replaced it with aValid Role
, but the oddities of AWS are a mystery. - A question you might ponder is why was the
Deleted Role
, deleted? It seems like there are Resources that use theDeleted Role
? It was because of a migration process that involved removing thatDeleted Role
since it should not be used anymore. Who started the migration process, ummm me 😬
Top comments (0)