Issue :
Sometimes You would see that You've created a DLM policy but It is in "Error State" or Not Creating Snapshots.
===== Some useful Tips =====
➜ Explaining the cause why you are not able to see the snapshots being created immediately:
- It is being said that when you are crating a new policy , It doesn't creates snapshots immediately. Normally it is being take in consideration that "first snapshot begin to be created by a policy within one hour following the specified start time" and for more info you can refer public documentation on this:
[+] Automating the Amazon EBS Snapshot Lifecycle - (Considerations) :https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot-lifecycle.html#dlm-considerations
➜ Hence, At this point, I will suggest you to please wait for an hour following the specified start time in policy and then check the snapshots if created in Snapshot page in EC2 Console.
➜ Common Scenario #1- "Duplicate tag key 'Name' specified." - While you have Policy in Error State
Have a look here -
[+] https://stackoverflow.com/questions/52048012/where-to-read-aws-dlm-data-lifecycle-management-error-details
Event Name will look like :
CreateSnapshot: CreateSnapshot @2018-12-24T20:25:58.000Z UTC "errorCode": "Client.InvalidParameterValue", "errorMessage": "Duplicate tag key 'Name' specified.", "requestParameters": { "volumeId": "vol-xxxxxxxxxxxx", "description": "Created for policy: policy-xxxxschedule: First Schedule",
Now, You need to review the resource tagging and update your DLM Policy to update the Duplicate Key entry field under Tag Created snapshots option and try again.
Below are the steps to follow: ( Example)
a). Login to your EC2 console.
b). Go to “Lifecycle Manager” on the left hand side.
c). Go to your DLM policy and “Right click” -- Click “Modify Snapshot LifeclyclePolicy”.
d). Move to the bottom of the policy and you can find “Tag created snapshots” option.
e). Change the Key from tag name for example “Name” to something Like “NameDLMPolicyTest”
[It should be unique than your resource i.e. volume key tags]
f). Click on “Update” and the policy will be updated.
➜ Common Scenario #2 - "errorCode": "Client.InvalidParameterCombination", "errorMessage": "Tag (Name) is already defined in resource id vol-xxxxx" for example
You might see some CloudTrail Event (CreateSnapshots) such as :
CreateSnapshots Error : "errorCode": "Client.InvalidParameterCombination", "errorMessage": "Tag (Name) is already defined in resource id vol-xxxxxxxxxxxx.", --------------------------------------------------------------------------------------{ "eventVersion": "1.05", "userIdentity": { "type": "AssumedRole", "eventTime": "2020-01-xxxxxxxx", "eventSource": "ec2.amazonaws.com", "eventName": "CreateSnapshots", "awsRegion": "us-east-1", "sourceIPAddress": "dlm.amazonaws.com", "userAgent": "dlm.amazonaws.com", "errorCode": "Client.InvalidParameterCombination", "errorMessage": "Tag (Name) is already defined in resource id vol-xxxxxxxx.", "requestParameters": { "requestParameters": { "CreateSnapshotsRequest": { "Description": "Created for policy: policy-xxxxxxxschedule: Default Schedule", "InstanceSpecification": { "ExcludeBootVolume": false, "InstanceId": "i-xxxxxxx" }, --------------------------------------------------------
Now, To solve this
The recommendation is to please review your existing policies and tags being used to make sure you're following the requirements / considerations listed here :
[+] DLM - Considerations for Amazon Data Lifecycle Manager : https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot-lifecycle.html#dlm-considerations
➜ There are some basic references to check If the policy doesn't trigger any Snapshots after that time as expected :
Please make sure the policy state is (Enabled), You can check it on EC2 Console - On Left side , Click on "Lifecycle Manager" and see the state
Console quick link : -
[+] http://console.aws.amazon.com/ec2/v2/home?Lifecycle:sort=PolicyId#Lifecycle:sort=PolicyId ( Make sure you are in correct region)And if you found your policy in "Error" State, Then you need to look for Your resource tags, IAM permissions, trust relationships (if you are using a custom IAM role) and many other factors to troubleshoot to make sure your configurations are all fine. You can refer docs for reference.
[+] https://forums.aws.amazon.com/thread.jspa?messageID=863017&tstart=0
[+] https://stackoverflow.com/questions/52048012/where-to-read-aws-dlm-data-lifecycle-management-error-details
[+] AWS DLM Limits : https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot-lifecycle.html#dlm-limits
➜ About the IAm role & Trust Relationships Cause :
Basically, Amazon DLM uses an IAM role to get the permissions that are required to manage snapshots on your behalf. Amazon DLM creates the "AWSDataLifecycleManagerDefaultRole" role the first time that you create a lifecycle policy using the AWS Management Console. For additional information on configuring Data Lifecycle Manager using a Custom IAM Role and about Permissions and trust relationship for Amazon DLM , please refer:
[+] Permissions & Trust relationship for Amazon DLM: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot-lifecycle.html#dlm-permissions
➜ Tracking the CloudTrail to see the cause :
Also, You can trace events for DLM Policy State Change — A DLM event emitted when a lifecycle policy enters an "ERROR" state. The event contains a description of what caused the error as well as the state of the policy. To know more, please refer:
[+] DLM - Using CloudWatch Events : https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot-lifecycle.html#monitor-cloudwatch-events
➜ Also, basic reference on the DLM Event Details present in CloudTrail which is useful:
Explaining a brief on DLM events in cloudtrail which will be logged using Event Name - "CreateSnapshot".
In general - DLM POLICY related Events
- CreateLifecyclePolicy - New Policy
- UpdateLifecyclePolicy - For any update in the policy
- CreateSnapshot - DLM Creates Snapshots ( Agent Name would be assigned as - "userAgent": "dlm.amazonaws.com") For more Info in this, please refer:
[+] DLM events references : https://docs.aws.amazon.com/cli/latest/reference/dlm/index.html
Other, Some basic info on a fresh setup, You can refer:
[+] Data Lifecycle Manager - Setup New Policy : https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot-lifecycle.html#dlm-lifecycle-policies
[+] Blog - Data Lifecycle Manager : https://aws.amazon.com/blogs/aws/new-lifecycle-management-for-amazon-ebs-snapshots/
Top comments (0)