Table of Contents
- Introduction
- Create an EC2 Instance Role
- Create a Key pair
- Create a Security Group
- Create an S3 Bucket
- Create an SQS queue
- Create a Cloud9 instance
- Create CloudWatch Alarms
- Create a Launch Template
- Create an Auto Scaling Group
- Verification and Monitoring
- Clean Up
- Summary
- Referrals
Introduction
In this blog we are going to set up auto scaling of EC2 instances using the SQS ApproximateNumberOfMessagesVisible metric.
AWS Auto Scaling monitors your applications and automatically adjusts capacity to maintain steady, predictable performance at the lowest possible cost. Using AWS Auto Scaling, it’s easy to set up application scaling for multiple resources across multiple services in minutes.
Benefits
- Set up Scaling quickly.
- Make SMART scaling decisions.
- Automatically maintain performance.
- Pay only for what you need.
Demo
Let's get started with the demo.
Step 1. Create an EC2 Instance Role
- Navigate to IAM > Roles > Click on Create role.
- Select EC2 under Common use case and Click Next.
- Select AmazonS3ReadOnlyAccess and AmazonSQSFullAccess and Click Next.
- Enter Role name as EC2InstanceRoleForSQS and Click Create role.
Note: you may create a custom policy for SQS with the required permissions instead of using the managed policy AmazonSQSFullAccess to provide the least amount of privileges.
AmazonSQSFullAccess
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"sqs:*"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
AmazonS3ReadOnlyAccess
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*",
"s3-object-lambda:Get*",
"s3-object-lambda:List*"
],
"Resource": "*"
}
]
}
Trusted entities
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Step 2. Create a key pair
- Navigate to EC2 > Key Pairs (under Network & Security).
- Click Create key pair.
- Enter Name as
auto-scale
. - Navigate to the folder where the key pair is downloaded and run.
chmod 400 <key_pair_name>.pem
Auto_Scale_Keypair
Step 3. Create a Security Group
- We are going to create a Security Group for SSH.
- Navigate to EC2 > Security Groups > Create a new security group for your ALB, and set the following values:
- Name:
MyIPSSH-SG
. - Add an Inbound rule to allow
SSH (TCP 22)
traffic fromMy IP
.
- Name:
- Click Create security group.
MyIPSSH-SG
Step 4. Create an S3 Bucket
- Navigate to EC2 > Click Create Bucket
- Enter the Bucket name as
autoscalescripts
, add some random numbers to make the bucket name unique. - Leave the rest of the settings as default and click Create Bucket
- Download
sendMessages.sh
andreceiveMessages.sh
from the github and upload them to the S3 bucket.
Step 5. Create an SQS queue
- Navigate to Simple Queue Service and click Create queue
- Enter Name as
MyMessages
, leave the rest as defaults and click Create queue
SQS_1
SQS_2
SQS_3
SQS_4
Step 6. Create a Cloud9 instance
- Navigate to Cloud9 > Create Environment.
- Enter Name as
awscli
and click Create. - Click Open under AWS Cloud9 > Environments.
- When Cloud9 is ready, Click File > Upload Local Files... and then upload
sendMessages.sh
. -
Let's check the AWS CLI version.
Sri:~/environment $ aws --version aws-cli/1.19.112 Python/2.7.18 Linux/4.14.301-224.520.amzn2.x86_64 botocore/1.20.112
The script uses
jq
, so we need to installjq
.Open a Terminal and execute
sudo yum install jq -y
.Execute
chmod +x sendMessages.sh
to provide the execute permissions.Now run the script to send 2000 messages to
MyMessages
queue.
Note: Alternatively you could use AWS CLI on your personal computer to send the messages.
Step 7. Create CloudWatch Alarms
We are going to create a ScaleOut Alarm to launch new instances when the ApproximateNumberOfMessagesVisible is greater than 500.
- Navigate to CloudWatch > Alarms > Click Create alarm.
- Click Select Metric, search for SQS, select SQS > Queue Metrics.
- Select MyMessages > ApproximateNumberOfMessagesVisible and Click Select Metric.
- Change Statistic to Sum, Period to 1 minute.
- Threshold type: Static.
- Whenever ApproximateNumberOfMessagesVisible is...: Greater.
- Define the threshold value: 500.
- Click Next.
- Click Remove under Notification and Click Next.
- Enter Alarm name as ScaleOut and Click Next.
- Click Create Alarm.
Similar to ScaleOut, we also need to create a ScaleIn Alarm to launch new instances when the ApproximateNumberOfMessagesVisible is lesser than 300.
- Select ScaleOut from CloudWatch > Alarms, Click on Actions and Copy.
- Change the following:
- Whenever ApproximateNumberOfMessagesVisible is...: Lower.
- Define the threshold value: 300.
- Click Next.
- Click Remove under Notification and Click Next.
- Enter Alarm name as ScaleIn and Click Next.
- Click Create Alarm.
👉 It is important to not have the same value for scale-in and scale-out thresholds. We should leave a gap between them to prevent oscillation.
For example: let's say we have 3 instances, and the CPU goes to 60%, which triggers the +1 step scaling policy. If the load stays constant, it will now be distributed to all 4 instances and the average CPU will drop to around 45% and the scale-in alarm will go off. This will then keep happening in a loop until the load goes up or down sufficiently for one of the alarms to stay in the alarm state and the ASG to reache the min or max.
ScaleOut_CreateAlarm1
ScaleOut_CreateAlarm2
ScaleOut_CreateAlarm3
ScaleOut_CreateAlarm4
ScaleOut_CreateAlarm5
ScaleOut_CreateAlarm6
ScaleOut_CreateAlarm7
ScaleOut_CreateAlarm7.1
ScaleIn_CreateAlarm1
ScaleIn_CreateAlarm2
ScaleIn_CreateAlarm3
ScaleIn_CreateAlarm4
ScaleIn_CreateAlarm4.1
Step 8. Create a Launch Template
We can use a Launch Template or Launch Configurations. Launch Template is preferred over Launch Configurations as we can have different versions of the template. Also we can't modify a Launch Configuration after we have created it.
Create a Launch Template that will be used by the Auto Scaling group. The Launch Template defines what the instances are and how they are created.
- Navigate to EC2 > Instances > Launch Templates.
- Create a new template, and name it
AutoScale-SQS
. - Search for
AMI
, and pick theAmazon Linux
. - Set the instance type as
t2.micro
. - Select
key pair
you created earlier. - Select the
MyIPSSH-SG
security group you created earlier. - Expand Advanced Details, and select
EC2InstanceRoleForSQS
Role under IAM instance profile. - Paste the following script under User data.
- Note: These are commands to install jq, AWS CLI, copy scripts from the S3 bucket and to execute receiveMessages.sh.
- Click Create Launch Template.
- Click View Launch templates.
User data
Note: Please update the bucket name in the script.
#!/bin/bash
# install jq
sudo yum install jq -y
# Update aws cli version
cd /home/ec2-user
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
sudo ./aws/install --bin-dir /usr/local/bin --install-dir /usr/local/aws-cli --update
which aws
ls -l /usr/local/bin/aws
aws --version
# copy sendMessages.sh and receiveMessages.sh from S3
aws s3 cp s3://sqssri/ . --recursive --exclude "*" --include "*.sh"
sudo chmod +x /home/ec2-user/*.sh
nohup ./receiveMessages.sh &
👉 As Launch Templates with User data are slow, it is recommended to create an AMI with the required software to improve the speed of instance initialisation.
LaunchTemplate1
LaunchTemplate1.1
LaunchTemplate1.2
LaunchTemplate1.3
LaunchTemplate1.4
Step 9. Create an Auto Scaling Group
- Navigate to EC2 > Auto Scaling > Auto Scaling Groups
- Click Create Auto Scaling group.
- Call the group
ASG-SQS
. - Select Launch Template, and choose the template named
AutoScale-SQS
. - We are using
default VPC
, which will be selected, so selectus-east-1a
as subnet. - Click Next.
- Leave the default for Health checks, which is
EC2
. - Leave the default for Configure advanced options and Click Next.
- For Group Size, enter the following values:
- Desired Capacity:
1
- Minimum Capacity:
1
- Maximum Capacity:
4
- Desired Capacity:
- We will not be adding Scaling policies here, so leave the default
None
and Click Next. - Click Next at
Add Notifications
. - Click Next at
Add tags
. - Click Create
Auto Scaling Group
. - Navigate to EC2 > Auto Scaling > click
ASG-SQS
and then clickAutomatic scaling
- We are going to add two dynamic scaling policies, one for scale out and another one for scale in. Click Create dynamic scaling policy and enter the following values and then Click Create:
- Policy type:
Simple Scaling
- Scaling policy name:
ScaleOut
- CloudWatch alarm:
ScaleOut
- Take the action:
Add
with1
capacity units - And then wait
60
seconds before allowing another scaling activity
- Policy type:
- Click Create dynamic scaling policy and enter the following values and then Click Create:
- Policy type:
Simple Scaling
- Scaling policy name:
ScaleIn
- CloudWatch alarm:
ScaleIn
- Take the action:
Remove
with1
capacity units - And then wait
120
seconds before allowing another scaling activity
- Policy type:
👉 It is best practice to scale up fast and scale down slow. Hence we used 60 seconds to scale out and 120 seconds to scale in.
ASG1
ASG2
ASG3
ASG4
ASG5
ASG6
ASG7
ASG7.1
ASG7.2
ASG8
ASG8.1_ScaleOut
ASG8.2_ScaleIn
ASG8.3_Dynamic_Scaling
Step 10. Verification and Monitoring
We have populated the queue with 2000 messages and have an auto scaling group launch with a maximum of 4 instances. We will now be verifying it using Cloud Watch Alarms and Auto Scaling group's Activity.
- Navigate to CloudWatch > Alarms > ScaleOut, the state of ScaleOut Alarm status will be
In alarm
and the ScaleIn Alarm status will beOK
. - After a period of 10 mins or so, you will notice 4 EC2 instances launched under EC2 > Auto Scaling groups > ASG-SQS > Activity.
- Monitor the messages in
MyMessages
queue, the number of messages will reduce as they are processed by the EC2 instances. - Navigate to CloudWatch > Alarms > ScaleIn, the state of ScaleIn Alarm status will be
In alarm
and the ScaleOut Alarm status will beOK
. - You will notice 3 EC2 instances terminated under EC2 > Auto Scaling groups > ASG-SQS > Activity when the messages are less than 300.
How to verify that the SQS messages are being processed
Log on to the EC2 Instance and then switch to the root user.
[ec2-user@ip-172-31-47-21 ~]$ sudo su -
Execute the following command, which will show the log file name.
ps xf
[root@ip-172-31-47-21 ~]# tail -f /var/log/cloud-init-output.log
Sleep for 1 second...
Sleep for 1 second...
Sleep for 1 second...
Sleep for 1 second...
^C
Messages_in_the_queue
ScaleOut_Alarm
ScaleOut_Activity
Processing_Messages
ScaleIn_Alarm
ScaleIn_Activity
Queue_is_Empty
👉 Auto Scaling Group Tip: When you do not want to have instances running or for Disaster Recovery purposes or to save costs, you may set the following inputs to Zero.
- Desired Capacity:
0
- Minimum Capacity:
0
- Maximum Capacity:
0
ASG_Tip
Clean Up
- Terminate
Cloud9
EC2 instance. - Delete
ASG-SQS
underAuto Scaling groups
. - Delete
AutoScale-SQS
underLaunch Templates
- Delete
auto-scale
underKey Pairs
- Delete Security Group
MyIPSSH-SG
. - Delete
EC2InstanceRoleForSQS
Role under IAM. - Delete S3 bucket
autoscalescripts
. - Delete SQS queue
MyMessages
.
Summary
👉 It is important to not have the same value for scale-in and scale-out thresholds. We should leave a gap between them to prevent oscillation.
👉 The best practice is to scale up fast, and scale down slow.
👉 Launching an EC2 instance might be slow if we have to install software, configure..etc during the scale out. One way to speed up the process is by creating an AMI with all the required software and then use that AMI in the Launch template.
Hope you learnt something new from the above demo.
See you next time 👋
Top comments (0)