DEV Community

Cover image for Create Instance Scheduler on Serverless by using Lambda, DynamoDB, API Gateway, Cognito,S3 and CloudFront
AWS Community Builders profile image Mohamed Radwan
Mohamed Radwan for AWS Community Builders

Posted on • Updated on

Create Instance Scheduler on Serverless by using Lambda, DynamoDB, API Gateway, Cognito,S3 and CloudFront

#serverless #lambda #aws #opensource

By using an instance scheduler you can start/stop or scheduler EC2/RDS instances to save costs.

The instance scheduler is open source and based on AWS serverless architecture.

Why do I use serverless?
Because:

  • No servers to manage.
  • Never pay for idler resources, only pay per event.
  • Scale with resilience and flexibility.
  • Fast development.

The App Architecture

instance scheduler architecture

In the application architecture:
Frontend
The user requests content from CloudFront service to get the HTML, CSS, and javascript. CloudFront caches the static files from the S3.
Backend
Once the Frontend is loaded locally on the user's internet browser, then the user SignUp or SignIn to the Cognito service, and it responded with a JWT token to the user.

The user can perform HTTP methods GET/POST/PUT/DELETE to the API gateway with the JWT token, API Gateway checks if the JWT token is valid from the Cognito service and after that invoke the lambda function with the request.

The lambda checks the events and extracts the user's email from the JWT token and add/update/delete the user's data from/to dynamoDB.
Also, lambda interacts with Key Management Service(KMS) to encrypt/decrypt sensitive data from/into dynamoDB.
In the case of email notification, lambda uses Simple Email Service (SES) to send emails to the users.

Database
The data is saved in dynamoDB.
The user's email is the partition key, and the sort key is created, always email is unique and the sort key can be changed with [ date or account information or aws keys].
Also used the attribute named dtime as Global Secondary Indexes GSI to get all users by specific date and time.

Scheduler
CloudWatch is used as a cron job to run each hour, it invokes the lambda function to get all attributes from Global Secondary Indexes GSI (dtime) of dynamoDB with specific dates and time, and it performs which EC2/RDS instances need to start/stop, after that will notify the user by email or webhook.

Steps to deploy the App:
You need to have aws cli and configured aws access key and secret keys.

Clone the app locally by

git clone https://github.com/maradwan/instance-scheduler.git
Enter fullscreen mode Exit fullscreen mode

1- Create a dynamoDB table

aws dynamodb create-table \
    --table-name instance-scheduler \
    --attribute-definitions AttributeName=email,AttributeType=S AttributeName=created,AttributeType=S AttributeName=dtime,AttributeType=S\
    --key-schema AttributeName=email,KeyType=HASH AttributeName=created,KeyType=RANGE \
    --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1 \
    --global-secondary-indexes "IndexName=dtime-index,\
    KeySchema=[{AttributeName=dtime,KeyType=HASH}],\
    Projection={ProjectionType=ALL},\
    ProvisionedThroughput={ReadCapacityUnits=1,WriteCapacityUnits=1}"
Enter fullscreen mode Exit fullscreen mode

2- Create Cognito service

aws cognito-idp create-user-pool --pool-name instance-scheduler --schema Name=email,Required=true --auto-verified-attributes email --verification-message-template DefaultEmailOption=CONFIRM_WITH_LINK

aws cognito-idp list-user-pools --max-results 20
Enter fullscreen mode Exit fullscreen mode

Get the id of UserPools of instance-scheduler and create the user pool client

aws cognito-idp create-user-pool-client --client-name my-user-pool-client --user-pool-id eu-west-1_2z7VlI6nq
Enter fullscreen mode Exit fullscreen mode

Take a note of the UserPoolId, ClientId

{
    "UserPoolClient": {
        "UserPoolId": "eu-west-1_2z7VlI6nq",
        "ClientName": "my-user-pool-client",
        "ClientId": "gqmnvsr3e8jmbg6ptaul30v9g",
        "LastModifiedDate": 1653128931.722,
        "CreationDate": 1653128931.722,
        "RefreshTokenValidity": 30,
        "AllowedOAuthFlowsUserPoolClient": false
    }
}
Enter fullscreen mode Exit fullscreen mode

Take a note arn of the Cognito service

aws cognito-idp describe-user-pool --user-pool-id eu-west-1_2z7VlI6nq
Enter fullscreen mode Exit fullscreen mode

Create custom domain name, should be unique name

aws cognito-idp create-user-pool-domain --domain instance-scheduler --user-pool-id eu-west-1_2z7VlI6nq
Enter fullscreen mode Exit fullscreen mode

3- Verify your email from Simple Email Service (SES) to used for sending notifications

aws ses verify-email-identity --email-address your_email_address@example.com --region eu-west-1
Enter fullscreen mode Exit fullscreen mode

You will receive an email from "Amazon Web Services" you need to click on the link to verify your email.

Check your email if has been verified by

aws ses list-identities
Enter fullscreen mode Exit fullscreen mode 
{
    "Identities": [
        "your_email_address@example.com"
    ]
}
Enter fullscreen mode Exit fullscreen mode

4- Create hourly Scheduler Lambda, KMS and Cloudwatch event by using terraform.
edit this file scheduler-lambda/scheduler-lambda.tf and change the following:

  • "SOURCE_EMAIL" to your verified email in step 4
  • "REGION_NAME" to which region you are using after that apply the following commands and take a note of kms key_id: 
cd scheduler-lambda
terraform init
terraform apply
Enter fullscreen mode Exit fullscreen mode

5- Create API Gateway and Lambda by using Zappa

edit this file app/zappa_settings.json and change the following:

  • "provider_arns" to your Cognito ARN in the step 2
  • "USERPOOLID" and "COGNITOR_CLIENTID" to your Cognito UserPoolId and ClientId in step 2
  • "SOURCE_EMAIL" to your verified email in step 3
  • "s3_bucket" bucket named should be unique globally
  • "REGION_NAME" to which region you are using
  • "KMS_KEY_ID" to your KMS id in the step 4

after that apply the following commands:

python3 -m venv env3
source env3/bin/activate
pip install -r app/requirements.txt
cd app
zappa deploy
Enter fullscreen mode Exit fullscreen mode

Take a note of your API Gateway looks like this

Your updated Zappa deployment is live!: https://74628jl936.execute-api.eu-west-1.amazonaws.com/staging
Enter fullscreen mode Exit fullscreen mode

6- Update the KMS to allow the lambda in step 5 to encrypt/decrypt

Add the following role to scheduler-lambda/kms.tf , "Sid" of Allow use of the key and Allow attachment of persistent resources

["arn:aws:iam::${local.account_id}:role/instance-scheduler-staging-ZappaLambdaExecutionRole", "${aws_iam_role.lambda_role.arn}"]
Enter fullscreen mode Exit fullscreen mode

After that apply terraform again

cd scheduler-lambda
terraform apply
Enter fullscreen mode Exit fullscreen mode

7- Create CloudFront and S3
The bucket should be a unique name, edit the cloudfront-s3-sync.sh and change the bucket name and region name. The script creates S3 and CloudFront then sync html folder to the bucket.
Edit this file html/js/config.js and change the following:

  • userPoolId, "userPoolClientId" and "region" to your UserPoolId, ClientId in step 2 and which region you are using
  • invokeUrl to your API Gateway in step 5 It looks like this: 
window._config = {
    cognito: {
        userPoolId: 'eu-west-1_XXX',
        userPoolClientId: 'XXX',
        region: 'eu-west-1'
    },
    api: {
         invokeUrl: 'https://api.example.com'

    }
};
Enter fullscreen mode Exit fullscreen mode

Then run the script

sh cloudfront-s3-sync.sh
Enter fullscreen mode Exit fullscreen mode

Wait 5 minutes until CloudFront be deployed and then login to the URL.

Test the APP
Create a new IAM user only for programmatic access and assign the the following policy.
replace REGION_NAME, ACCOUNT_NUMBER and INSTANCE_ID

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:StartInstances",
                "ec2:StopInstances"
            ],
            "Resource": "arn:aws:ec2:REGION_NAME:ACCOUNT_NUMBER:instance/INSTANCE_ID"

        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

For RDS policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "RDS:StopDBInstance",
                "RDS:StartDBInstance"
            ],
            "Resource": [
                "arn:aws:rds:REGION_NAME:ACCOUNT_NUMBER:db:DB_NAME"
            ]
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

  • Create a key in the app
    access-keys

  • Add the EC2 and RDS instances

instance-scheduler

Top comments (1)

Collapse
 
mgaberh profile image
Mohamed Gaber •

really amazing implementation, I will try to apply it as a POC to learn more.

Read next

mrcaption49 profile image

EXECUTE IMMEDIATE with BULK COLLECT

Pranav Bakare -

karthiksakthiveltechie profile image

Amazon EC2 now supports to Optimize CPUs post instance launch

Karthik Sakthivel -

camilacodes profile image

DynamoDB: Query x Scan! Para de torrar dinheiro usando Scan em produção

Camila Figueira -

lucasmelo profile image

How to deploy affected NX projects to AWS S3 using Github Actions

Lucas Melo -