It’s a good practice to use CloudFormation’s DeletionPolicy
to protect stateful resources such as DynamoDB tables or RDS databases from accidental deletions. Such as when someone accidentally deletes a CloudFormation stack!
As I discussed previously [1], this is a much better way to guard against these accidental data losses than separating stateful and stateless resources into different stacks. After all, how we think about infrastructure vs. code needs to evolve in the serverless era [2]. That separation is no longer so clear-cut.
The problem with DeletionPolicy
But using DeletionPolicy
on all your stateful resources causes friction with another common practice in serverless development – the use of ephemeral (or temporary) environments [3].
If you set DeletionPolicy
to retain
on all your stateful resources, then they will all linger when you delete the temporary environment.
In fact, you only need to use DeletionPolicy
in production. That’s the only environment where it’s necessary.
So, how do we do that?
The CDK solution
CDK gives you the full power of a general-purpose programming language (even though it’s often needed [4]). So it’s trivial to do this in a CDK application:
- Pass the environment name as either a CloudFormation parameter or context variable.
- Set the
removalPolicy
inside anif
block like this:
if (environment === 'prod') {
myTable.removalPolicy = cdk.RemovalPolicy.RETAIN
}
The Serverless framework solution
The easiest way to do this with the Serverless framework is to use the serverless-plugin-ifelse
plugin.
Once you have installed the plugin as a dev dependency, you can use it to remove the DeletionPolicy
on stateful resources like this:
plugins:
- serverless-plugin-ifelse
custom:
serverlessIfElse:
- If: '"${sls:stage}" != "prod"'
Exclude:
- resources.Resources.MyTable.DeletionPolicy
resources:
Resources:
MyTable:
Type: AWS::DynamoDB::Table
DeletionPolicy: Retain
Properties:
...
Notice that DeletionPolicy
is enabled by default and is removed if the current stage (i.e. environment name) is not prod
?
This is defensive coding. So that if I get anything wrong, the worst that can happen is I have to manually delete the table from temporary environments.
With the reverse approach (i.e. add DeletionPolicy
only for the prod
stage), the worst-case scenario is far worse.
If you want to see this solution in action, then check out this repo [5].
SAM / CloudFormation solution
For SAM or CloudFormation, this is the best solution I have found:
- Add an
Environment
CloudFormation Parameter to your template. - Add a map of
DeletionPolicy
value to use for each environment. - Use the
Fn:FindInMap
intrinsic function to find theDeletionPolicy
value to use for the current environment.
Something like this:
Parameters:
Environment:
Type: String
Default: dev
Description: Environment name, e.g. dev, test, prod
Mappings:
DeletionPolicyMap:
dev:
value: Delete
prod:
value: Retain
Resources:
MyTable:
Type: AWS::DynamoDB::Table
DeletionPolicy: !FindInMap [ DeletionPolicyMap, !Ref Environment, value ]
Properties:
...
This solution is not ideal because it requires the environment names to be explicitly configured. You have to update the DeletionPolicyMap
every time you create a new environment.
As such, it doesn’t work well with ephemeral environments where environment names are usually dynamic. If you know of a better way to do this with SAM and CloudFormation, please let me know!
DynamoDB DeletionProtectionEnabled attribute
I have used DynamoDB in the examples above. But it’s worth pointing out that DeletionPolicy
is actually not the most effective protection against data loss for DynamoDB.
For starters, it doesn’t guard against someone deleting a DynamoDB table in the console or programmatically through the AWS CLI or SDK.
Fortunately, DynamoDB also has the DeletionProtectionEnabled
[6] attribute. When enabled, it protects the table from accidental deletion by any user or process.
Additionally, DynamoDB offers other protection against data loss, such as point-in-time recovery and the ability to export data to S3.
Having said that, DeletionPolicy
is still a very useful tool for other services such as S3, EventBridge, SQS, SNS and so on.
Shout out to Hala
I want to give a shout-out to Hala Al Aali for asking me about this in the Testing Serverless Architectures [7] course’s forum.
It’s a common problem that many students have come across. I hope this article has helped you. If you want to learn more about building serverless applications for the real world, check out my upcoming workshops [8].
Links
[1] This is why you should keep stateful and stateless resources together
[2] Are we getting infrastructure all wrong in the Serverless era?
[3] Serverless Ephemeral (Temporary) Environments Explained
[4] Are You Ready for This? Top 5 Earth-Shattering Pros and Cons of AWS CDK
[5] github.com/theburningmonk/conditional-deletionpolicy-demo
[6] DynamoDB’s DeletionProtectionEnabled attribute
[7] Testing Serverless Architectures course
[8] Production-Ready Serverless workshop
The post Help! How do I set DeletionPolicy to Retain for production only? appeared first on theburningmonk.com.
Top comments (0)