Incident management
Excited to share my latest AWS project focusing on automating an incident response following what I found to be a really engaging AWS made workshop.
The project involves setting up a core configuration of 3 EC2 instances with its corresponding security groups and a VPC in us-east-1 through a CloudFormation template.
To enhance security and ensure prompt incident response, a pipeline has been implemented. This pipeline starts with GuardDuty constantly monitoring the environment. When an anomaly is detected, an Amazon EventBridge Rule triggers a Lambda function.
The Lambda function plays a crucial role in securing the EC2 instances by restricting access to only ports 3389 and 22. Additionally, it takes an EC2 snapshot and stores it in an S3 bucket to prevent any data loss. The architecture is as follows:
To initiate the scenario and create the infrastructure for the automated incident response, I followed these steps to deploy the CloudFormation template:
Deploy the CloudFormation template
- Review the CloudFormation Template
- Before deploying, you can review the template to understand its components and configurations.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Security Automations Workshop template. Sets up VPC, EC2 instances, turns on GuardDuty and GuardDuty-Tester",
"Metadata": {
"AWS::CloudFormation::Interface": {
"ParameterGroups": [
{
"Label" : {"default": "Workshop Service Configuration"},
"Parameters": ["EnableGuardDuty"]
},
{
"Label" : {"default": "Workshop Parameters"},
"Parameters": ["LatestAMZNLinuxAMI", "LatestAMZNLinux2AMI", "LatestWindows2016AMI"]
}
],
"ParameterLabels": {
"EnableGuardDuty": {"default" : "Automatically enable GuardDuty?"}
}
}
},
"Parameters": {
"LatestAMZNLinuxAMI": {
"Description": "DO NOT CHANGE: The latest AMI ID for Amazon Linux",
"Type": "AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>",
"Default": "/aws/service/ami-amazon-linux-latest/amzn-ami-hvm-x86_64-gp2"
},
"LatestAMZNLinux2AMI": {
"Description": "DO NOT CHANGE: The latest AMI ID for Amazon Linux2",
"Type": "AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>",
"Default": "/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2"
},
"LatestWindows2016AMI": {
"Description": "DO NOT CHANGE: The latest AMI ID for Windows 2016",
"Type": "AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>",
"Default": "/aws/service/ami-windows-latest/EC2LaunchV2-Windows_Server-2016-English-Full-Base"
},
"EnableGuardDuty": {
"Description": "Choose Yes if GuardDuty is not yet enabled in the account and region this template will be deployed to, otherwise choose No.",
"Type": "String",
"AllowedValues": ["Yes-Enable GuardDuty", "No-GuardDuty is already enabled"],
"Default": "Yes-Enable GuardDuty"
}
},
"Mappings": {
"AWSRegionAMIMap": {
"ap-south-1": {"HVM64": "ami-b46f48db"},
"eu-west-3": {"HVM64": "ami-cae150b7"},
"eu-west-2": {"HVM64": "ami-c12dcda6"},
"eu-west-1": {"HVM64": "ami-9cbe9be5"},
"ap-northeast-3": {"HVM64": "ami-68c1cf15"},
"ap-northeast-2": {"HVM64": "ami-efaf0181"},
"ap-northeast-1": {"HVM64": "ami-28ddc154"},
"sa-east-1": {"HVM64": "ami-f09dcc9c"},
"ca-central-1": {"HVM64": "ami-2f39bf4b"},
"ap-southeast-1": {"HVM64": "ami-64260718"},
"ap-southeast-2": {"HVM64": "ami-60a26a02"},
"eu-central-1": {"HVM64": "ami-1b316af0"},
"us-east-1": {"HVM64": "ami-467ca739"},
"us-east-2": {"HVM64": "ami-976152f2"},
"us-west-1": {"HVM64": "ami-46e1f226"},
"us-west-2": {"HVM64": "ami-6b8cef13"}
}
},
"Conditions": {
"EnableGuardDuty": {"Fn::Equals": [{"Ref": "EnableGuardDuty"}, "Yes-Enable GuardDuty"]}
},
"Resources": {
"SSMInstanceRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"ssm.amazonaws.com",
"ec2.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
},
"Policies": [
{
"PolicyName": "S3andSSMAccess",
"PolicyDocument": {
"Statement": [
{
"Effect": "Allow",
"Action": [
"ssm:DescribeAssociation",
"ssm:GetDeployablePatchSnapshotForInstance",
"ssm:GetDocument",
"ssm:DescribeDocument",
"ssm:GetManifest",
"ssm:GetParameters",
"ssm:GetParameter",
"ssm:ListAssociations",
"ssm:ListInstanceAssociations",
"ssm:PutInventory",
"ssm:PutComplianceItems",
"ssm:PutConfigurePackageResult",
"ssm:UpdateAssociationStatus",
"ssm:UpdateInstanceAssociationStatus",
"ssm:UpdateInstanceInformation"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ssmmessages:CreateControlChannel",
"ssmmessages:CreateDataChannel",
"ssmmessages:OpenControlChannel",
"ssmmessages:OpenDataChannel"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2messages:AcknowledgeMessage",
"ec2messages:DeleteMessage",
"ec2messages:FailMessage",
"ec2messages:GetEndpoint",
"ec2messages:GetMessages",
"ec2messages:SendReply"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"cloudwatch:PutMetricData"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2:DescribeInstanceStatus"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ds:CreateComputer",
"ds:DescribeDirectories"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"logs:PutLogEvents"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:PutObject",
"s3:GetObject",
"s3:GetEncryptionConfiguration",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts",
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Resource": "*"
},
{
"Sid": "S3ListBuckets",
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets"
],
"Resource": "arn:aws:s3:::*"
},
{
"Sid": "S3GetObjects",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:GetObject"
],
"Resource": {
"Fn::Join": [
"",
[
"arn:aws:s3:::",
"agentbucket-",
{
"Ref": "AWS::AccountId"
},
"/*"
]
]
}
}
]
}
}
],
"Path": "/"
}
},
"VPC": {
"Type": "AWS::EC2::VPC",
"Properties": {
"CidrBlock": "10.0.0.0/16",
"EnableDnsSupport": "true",
"EnableDnsHostnames": "true",
"Tags": [
{
"Key": "Application",
"Value": {
"Ref": "AWS::StackName"
}
},
{
"Key": "Name",
"Value": {
"Fn::Join": [
"-",
[
"VPC",
{
"Ref": "AWS::StackName"
}
]
]
}
}
]
}
},
"PublicSubnet": {
"Type": "AWS::EC2::Subnet",
"Properties": {
"VpcId": {
"Ref": "VPC"
},
"CidrBlock": "10.0.1.0/24",
"MapPublicIpOnLaunch": "true",
"AvailabilityZone": {
"Fn::Select": [
"0",
{
"Fn::GetAZs": {
"Ref": "AWS::Region"
}
}
]
},
"Tags": [
{
"Key": "Name",
"Value": {
"Fn::Join": [
"-",
[
"Pub1",
{
"Ref": "AWS::StackName"
}
]
]
}
}
]
}
},
"IGW": {
"Type": "AWS::EC2::InternetGateway",
"Properties": {
"Tags": [
{
"Key": "Application",
"Value": {
"Ref": "AWS::StackName"
}
}
]
}
},
"AttachGateway": {
"Type": "AWS::EC2::VPCGatewayAttachment",
"Properties": {
"VpcId": {
"Ref": "VPC"
},
"InternetGatewayId": {
"Ref": "IGW"
}
}
},
"PublicRouteTable": {
"Type": "AWS::EC2::RouteTable",
"Properties": {
"VpcId": {
"Ref": "VPC"
},
"Tags": [
{
"Key": "Application",
"Value": {
"Ref": "AWS::StackName"
}
},
{
"Key": "Network",
"Value": "Public"
}
]
}
},
"PublicRoute": {
"Type": "AWS::EC2::Route",
"DependsOn": [
"AttachGateway"
],
"Properties": {
"RouteTableId": {
"Ref": "PublicRouteTable"
},
"DestinationCidrBlock": "0.0.0.0/0",
"GatewayId": {
"Ref": "IGW"
}
}
},
"PublicSubnetRouteAssociation": {
"Type": "AWS::EC2::SubnetRouteTableAssociation",
"Properties": {
"SubnetId": {
"Ref": "PublicSubnet"
},
"RouteTableId": {
"Ref": "PublicRouteTable"
}
}
},
"GDdetector": {
"Type": "AWS::GuardDuty::Detector",
"Condition": "EnableGuardDuty",
"Properties": {
"Enable": true,
"FindingPublishingFrequency": "FIFTEEN_MINUTES"
}
},
"GuardDutyTesterTemplate": {
"Type": "AWS::CloudFormation::Stack",
"Properties": {
"TemplateURL":{
"Fn::Join": [
"",
[
"https://sa-security-specialist-workshops-",
{
"Ref": "AWS::Region"
},
".s3.",
{
"Ref": "AWS::Region"
},
".amazonaws.com/security-hub-workshop/templates/guardduty-tester-template.json"
]
]
},
"Parameters": {
"InstanceSubnetId": {
"Ref": "PublicSubnet"
},
"DeployVPC": {
"Ref": "VPC"
},
"DeployVPCCidr": {
"Fn::GetAtt": [
"VPC",
"CidrBlock"
]
},
"LatestWindows2012R2AMI": "/aws/service/ami-windows-latest/EC2LaunchV2-Windows_Server-2016-English-Full-Base"
}
}
}
}
}
-
Deploy the Template
- Select aregion, for instance, I will be using us-east-1 (N. Virginia).
-
Specify Stack Details
- Enter the following parameters:
- Stack name: AutomatedIncidentResponseWorkshop
- Enable GuardDuty: Yes
- After filling in the parameters, click Next.
- Enter the following parameters:
-
Configure Stack Options
- Click Next again on the following page, leaving all options at their default values.
-
Acknowledge and Create Stack
- Scroll down to the bottom of the page, check the box acknowledging that the template will create IAM roles, and click Create stack.
Setting up a security group
- Create Security Group
- Create a new security group named ForensicsSG.
- Remove all outbound rules and set the following inbound rules:
- RDP: Protocol TCP, Port 3389, Source (your IP), Description: RDP for IR team
- SSH: Protocol TCP, Port 22, Source (your IP), Description: SSH for IR team
Creating and Attaching Policies
-
Create a New IAM Policy
- Create a policy called ec2instance-containment-with-forensics-policy with the following JSON to deny termination of isolated instances: {
- "Version": "2012-10-17",
- "Statement": [
- {
- "Sid": "VisualEditor0",
- "Effect": "Deny",
- "Action": [
- "ec2:TerminateInstances",
- "ec2:DeleteTags",
- "ec2:CreateTags"
- ],
- "Resource": "*",
- "Condition": {
- "StringEquals": {
- "aws:ResourceTag/status": "isolated"
- }
- }
- }
- ]
- }
Create the Execution role for the Lambda function:
Create a role called ec2instance-containment-with-forensics-role with Lambda as a trusted entity in Trust Relationships-
Create a User Group
- Create a group named ec2-users.
-
Attach Policies to the Group
- Attach the following policies to the ec2-users group:
- AmazonEC2FullAccess (AWS Managed Policy)
- Deny-termination-of-isolated-instances (Custom policy created above).
- Attach the following policies to the ec2-users group:
-
Create a New User
- Create an IAM user named testuser
- Add this user to the ec2-users group.
Configuring the Lambda Function
-
Create IAM Policy for Lambda
- Create an IAM policy and attach it to the IAM role that the Lambda function will assume for automated responses.
-
Create the Lambda Function
- Develop and deploy the Lambda function that will handle automated incident responses. Change the timeout to 15 minutes and select ec2instance-containment-with-forensics-role as the execution role. Select Python as runtime.
- Add the following environmental values:
- Key: ForensicsSG
- Value: sg-...(the ID of your Forensics SG)
- Include the following code: import boto3 import time from datetime import date from botocore.exceptions import ClientError import os
def lambda_handler(event, context):
# Copyright 2022 - Amazon Web Services
# Permission is hereby granted, free of charge, to any person obtaining a copy of this
# software and associated documentation files (the "Software"), to deal in the Software
# without restriction, including without limitation the rights to use, copy, modify,
# merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so.
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
# INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
# PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
# print('## ENVIRONMENT VARIABLES')
# print(os.environ)
# print('## EVENT')
# print(event)
response = 'Error remediating the security finding.'
try:
# Gather Instance ID from CloudWatch event
instanceID = event['detail']['resource']['instanceDetails']['instanceId']
print('## INSTANCE ID: %s' % (instanceID))
# Get instance details
client = boto3.client('ec2')
ec2 = boto3.resource('ec2')
instance = ec2.Instance(instanceID)
instance_description = client.describe_instances(InstanceIds=[instanceID])
print('## INSTANCE DESCRIPTION: %s' % (instance_description))
#-------------------------------------------------------------------
# Protect instance from termination
#-------------------------------------------------------------------
ec2.Instance(instanceID).modify_attribute(
DisableApiTermination={
'Value': True
})
ec2.Instance(instanceID).modify_attribute(
InstanceInitiatedShutdownBehavior={
'Value': 'stop'
})
#-------------------------------------------------------------------
# Create tags to avoid accidental deletion of forensics evidence
#-------------------------------------------------------------------
ec2.create_tags(Resources=[instanceID], Tags=[{'Key':'status', 'Value':'isolated'}])
print('## INSTANCE TAGS: %s' % (instance.tags))
#------------------------------------
## Isolate Instance
#------------------------------------
print('quarantining instance -- %s, %s' % (instance.id, instance.instance_type))
# Change instance Security Group attribute to terminate connections and allow Forensics Team's access
instance.modify_attribute(Groups=[os.environ['ForensicsSG']])
print('Instance ready for root cause analysis -- %s, %s' % (instance.id, instance.security_groups))
#------------------------------------
## Create snapshots of EBS volumes
#------------------------------------
description= 'Isolated Instance:' + instance.id + ' on account: ' + event['detail']['accountId'] + ' on ' + date.today().strftime("%Y-%m-%d %H:%M:%S")
SnapShotDetails = client.create_snapshots(
Description=description,
InstanceSpecification = {
'InstanceId': instanceID,
'ExcludeBootVolume': False
}
)
print('Snapshot Created -- %s' % (SnapShotDetails))
response = 'Instance ' + instance.id + ' auto-remediated'
except ClientError as e:
print(e)
return response
- Test the Lambda Function
- Perform tests to ensure the Lambda function works as expected. You can use the following code (remember to change some of the variables): import boto3, json import time from datetime import date from botocore.exceptions import ClientError import os
def lambda_handler(event, context):
# Copyright 2022 - Amazon Web Services
# Permission is hereby granted, free of charge, to any person obtaining a copy of this
# software and associated documentation files (the "Software"), to deal in the Software
# without restriction, including without limitation the rights to use, copy, modify,
# merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so.
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
# INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
# PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
# print('## ENVIRONMENT VARIABLES')
# print(os.environ)
# print('## EVENT')
# print(event)
response = 'Error remediating the security finding.'
try:
# Gather Instance ID from CloudWatch event
instanceID = event['detail']['resource']['instanceDetails']['instanceId']
print('## INSTANCE ID: %s' % (instanceID))
# Get instance details
client = boto3.client('ec2')
ec2 = boto3.resource('ec2')
instance = ec2.Instance(instanceID)
instance_description = client.describe_instances(InstanceIds=[instanceID])
print('## INSTANCE DESCRIPTION: %s' % (instance_description))
#-------------------------------------------------------------------
# Protect instance from termination
#-------------------------------------------------------------------
ec2.Instance(instanceID).modify_attribute(
DisableApiTermination={
'Value': True
})
ec2.Instance(instanceID).modify_attribute(
InstanceInitiatedShutdownBehavior={
'Value': 'stop'
})
#-------------------------------------------------------------------
# Create tags to avoid accidental deletion of forensics evidence
#-------------------------------------------------------------------
ec2.create_tags(Resources=[instanceID], Tags=[{'Key':'status', 'Value':'isolated'}])
print('## INSTANCE TAGS: %s' % (instance.tags))
#------------------------------------
## Isolate Instance
#------------------------------------
print('quarantining instance -- %s, %s' % (instance.id, instance.instance_type))
# Change instance Security Group attribute to terminate connections and allow Forensics Team's access
instance.modify_attribute(Groups=[os.environ['ForensicsSG']])
print('Instance ready for root cause analysis -- %s, %s' % (instance.id, instance.security_groups))
#------------------------------------
## Create snapshots of EBS volumes
#------------------------------------
description= 'Isolated Instance:' + instance.id + ' on account: ' + event['detail']['accountId'] + ' on ' + date.today().strftime("%Y-%m-%d %H:%M:%S")
SnapShotDetails = client.create_snapshots(
Description=description,
InstanceSpecification = {
'InstanceId': instanceID,
'ExcludeBootVolume': False
}
)
print('Snapshot Created -- %s' % (SnapShotDetails))
response = 'Instance ' + instance.id + ' auto-remediated'
except ClientError as e:
print(e)
return response
**IMPORTANT: **Verify status after execution: check on the EC2 console what is the current status of the instance "BasicLinuxTarget”. You will see that the security group has changed to the one we configured only for the IR team. You will also see new snapshots have been created. You are now seeing our automated response in live action!
Before testing, the security group in the instance was:
After testing, note the change in security group. Our IR security group took over.
*You can check the GuardDuty Dashboard as well to see the threats it detected and the lambda logs, as well as the snapshots created. *
- Create EventBridge Rule
- Create a rule in EventBridge that triggers the Lambda function based on findings from GuardDuty. On creation method, select custom pattern and use the following code:
{
"source": ["aws.guardduty"],
"detail": {
"type": ["UnauthorizedAccess:EC2/TorClient", "Backdoor:EC2/C&CActivity.B!DNS", "Trojan:EC2/DNSDataExfiltration", "CryptoCurrency:EC2/BitcoinTool.B", "CryptoCurrency:EC2/BitcoinTool.B!DNS"]
}
}
As target, select the lambda you previously created. Click on create.
This is one way to manage incident responses automatically in the cloud.
Personal learnings from this project:
•To perform automated basic incident response tasks for containment and gathering data for analysing cyber threats.
•To understand possible actions to take and how to prevent such threats affecting a production environment.
How could this be improved?
As a reflective professional, I like to spend some time after finishing projects thinking how they can be further improved.
For future enhancements, I plan to integrate an SNS topic to notify the incident response team. This will enable manual checks in case of any damage and facilitate reverting to the original state when the situation is under control.
There is also the downside of a 15 minute maximum time for the lambda function. If needed, some environments will probably benefit more of using a different architecture, one that relies on step functions to avoid the time restriction.
Excited to continue optimising this project for enhanced incident response capabilities. Thanks for reading and happy deploying if you want to give this a go!
Top comments (0)