Spot Instance Scenarios

Scenario

• Instance stopped by AWS due to Insufficient Capacity but not started automatically by AWS when Capacity is available again.
• No issue when user start the instance manually.

Reason

• Service role is not able to access the KMS key that is cross account and the instance is using this KMS for their volume.

Troubleshooting

• Look at the configuration changes and you will see “Client error on launch”.

• Check CloudTrail logs and you can see the Access Denied error on KMS.

Solution

On the account whereby the instance is launched, run the following command to grant the KMS permission to service role.

aws kms create-grant — region <region> –key-id <arn of the KMS> — grantee-principal <arn of the Spot Service Role> — operations “Decrypt” “Encrypt” “GenerateDataKey” “GenerateDataKeyWithoutPlaintext” “CreateGrant” “DescribeKey” “ReEncryptFrom” “ReEncryptTo”

Example:

aws kms create-grant — region ap-southeast-1 — key-id arn:aws:kms:ap-southeast-1:123456789:key/479d6414-e442–4873–9b10-123dwdas343 — grantee-principal arn:aws:iam::987654321:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot — operations “Decrypt” “Encrypt” “GenerateDataKey” “GenerateDataKeyWithoutPlaintext” “CreateGrant” “DescribeKey” “ReEncryptFrom” “ReEncryptTo”

Result:

Note:
Monitor the situation to ensure that instance starts up whenever spot instance is reclaimed by AWS due to insufficient capacity.