Jinwook Baek

Posted on Oct 10, 2020

Migrate python async worker to asynchrounous Lambda

#aws #serverless #lambda #django

Intro

When you build a web service, there is a time that you need to implement simple async worker for background process. I am big fan of Django and whenever I need a async worker, I used celery or rq for convenience. celery or other async framework provides a lot of cool features but most of the time all I need is a simple background process that would not interrupt user experiences.

Overtime, as paradigm shifts to cloud, I have been thinking lambda as new async worker solution due to deployment complexity, scaling and cost constraint. Here are pros and cons for using lambda instead of celery on cloud environment(non-local environment).

Advantage

FasS advantage
- scaling benefit - cost, ops, etc
- Resiliency
- faster development
No need to maintain Broker (e.g elastic search redis)
Smaller side effect on code change
Fully-managed

Disadvantage

FasS disadvantage
- limited state
Hard to reuse app code
- Django ORM or settings module
- You can still use them with django package installed but inefficient
- can't reuse code from django app
- Django package is too heavy to use on lambda (still usable but too complex to setup
Latency on startup
Json formatted Invocation parameter
- If you need to pass binary or big size paras, need to consider S3 or other medium

Limitation

Most of early stage limitation are lifted as lambda is in mature state (e.g conccurent executions, layers, etc)

AWS Lambda quotas

timeout - 15min
- If your async worker take more than 5min, you need to break up codes in to multiple lambda functions or use other distributed system such as EMR, etc.
Invocation payload
- asynchronous - 256KB

Consideration

VPC, Subnet, Security Group
- If you are accessing Resources in private VPC, you need to consider placing lambda in same subnet.
- If you are lambda needs to talk to outside world, private subnets needs NAT gateway
Maintenance

Setup

AWS recently came up with SAM(Serverless Application Model) which utilize cloudformation for easier deployment of lambda application. SAM provides functionality such as API gateway and statemachine, but we will only be using AWS::Serverless::Function resource for our purpose. You can consider using Zappa for other option.

Github

refer to this repo for details

kokospapa8/async-lambda-sample-app

Prerequisite

You need following AWS resource created in order to invoke asynchronous lambda.

DLQ

In order use asynchronous lambda, you need to provide DLQ in case lambda fails and retry. Click link for details. I will create SNS topic in this post.

Asynchronous invocation

SECRET

Let's assume that you have mysql db in public VPC or s3 and need to access them in order to process some data, you can add secrets to lambda env, but it is not a good idea to put secret in plaintext for env variable. You can use aws secret manger , ssm parameterstore or vault . I will use ssm parameterstore for this post. Refer to previous post for creating secrets.(parameter store section)

How to deploy django app to ECS Fargate part3

arn:aws:ssm:<region>:<account_id>:parameter/SAMPLE/BUCKET

IAM ROLE

You need lambda execution role with following permission and polcies attached.



Policies
- AWSLambdaExecute
Permission 
- SNS – sns:Publish
- kms - kms:Decrypt
- ssm - ssm:GetParameters, ssm:DescribeParameters, ssm:GetParameter
You need following permission if you want to place your lambda function in a designated VPC
- ec2 - ec2:CreateNetworkInterface, ec2:DeleteNetworkInterface, ec2:DescribeSecurityGroups

SAM

What is the AWS Serverless Application Model (AWS SAM)?

Install (on macOS)

Installing the AWS SAM CLI on macOS



$ aws configure
AWS Access Key ID [None]: your_access_key_id
AWS Secret Access Key [None]: your_secret_access_key
Default region name [None]: 
Default output format [None]:

$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
$ brew tap aws/tap
$ brew install aws-sam-cli

Development



$ sam init

This code will create following structure



sam-app/
   ├── README.md
   ├── events/
   │   └── event.json
   ├── hello_world/
   │   ├── __init__.py
   │   ├── app.py            #Contains your AWS Lambda handler logic.
   │   └── requirements.txt  #Contains any Python dependencies the application requires, used for sam build
   ├── template.yaml         #Contains the AWS SAM template defining your application's AWS resources.
   └── tests/
       └── unit/
           ├── __init__.py
           └── test_handler.py

template.yaml

Update template file if you need more info on syntax, refer to the link.

AWS::Serverless::Function

template.yaml ```

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
sample-app
Sample SAM Template for sample-app

More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst

Globals:
Function:
Timeout: 3

Resources:
SampleAppdFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: sample_app/
DeadLetterQueue:
Type: SNS
TargetArn: arn:aws:sns:::sample-dlq
Handler: app.lambda_handler
Runtime: python3.8
Description: sample lambda
EventInvokeConfig:
MaximumEventAgeInSeconds: 60
MaximumRetryAttempts: 2
FunctionName: SampleApp
Role: arn:aws:iam:::role/sample_lambda_execution_role
Environment:
Variables:
S3_URL: ""

Outputs:
SampleAppdFunction:
Description: "Sample app Function ARN"
Value: !GetAtt SampleAppdFunction.Arn

- requirements.txt

boto3

- app.py
This sample app receives image_url as parameter and puts the image on s3bucket name provided on s3 bucket. (AWSLambdaExecute policy has s3 access permission)

import boto3
import os
import requests

def lambda_handler(event, context):
image_url = event['image_url']
S3_BUCKET_PARAM = os.environ['S3_BUCKET_PARAM']

# download

image = requests.get(image_url).content


ssm_client = boto3.client('ssm', region_name="ap-northeast-2")
response = ssm_client.get_parameter(
    Name=S3_BUCKET_PARAM,
    WithDecryption=True
)
bucket_name = response['Parameter']['Value']

s3_client = boto3.client('s3', region_name="ap-northeast-2")
response = s3_client.put_object(
    Bucket=bucket_name,
    Key="image.png",
    Body=image
)

return response


## Build

Once your logic code is ready, you can build it using following command 

```yaml


sam build -t template.yaml --region <region_name>

You can test your function with following command



$ sam local invoke
# if you need to provide environment variable or paramters, use following command
$ sam local invoke -e events/event.json --env-vars env.json



#env.json
{
  "SampleAppdFunction": {
        "S3_BUCKET_PARAM_ARN": "parameterstore_arn",
  }
}
#event.json
{
  "image_url": "https://picsum.photos/200/300"
}

Deploy

Once your function is tested locally, let's deploy the function to cloud. You have two options. Enter appropriate response to prompt



$ sam deploy --guided

Configuring SAM deploy
======================

    Looking for samconfig.toml :  Found
    Reading default arguments  :  Success

    Setting default arguments for 'sam deploy'
    =========================================
    Stack Name [sample-app]: y
    AWS Region [ap-northeast-2]: y
    #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
    Confirm changes before deploy [Y/n]: y
    #SAM needs permission to be able to create roles to connect to the resources in your template
    Allow SAM CLI IAM role creation [Y/n]: y
    Save arguments to samconfig.toml [Y/n]: y

output should look like this




    Looking for resources needed for deployment: Found!

        Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
        A different default S3 bucket can be set in samconfig.toml

    Deploying with following values
    ===============================
    Stack name                 : sample-app
    Region                     : ap-northeast-2
    Confirm changeset          : True
    Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
    Capabilities               : ["CAPABILITY_IAM"]
    Parameter overrides        : {}

Initiating deployment
=====================

    Saved arguments to config file
    Running 'sam deploy' for future deployments will use the parameters saved above.
    The above parameters can be changed by modifying samconfig.toml
    Learn more about samconfig.toml syntax at 
    https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html

    Deploying with following values
    ===============================
    Stack name                 : sample-app
    Region                     : ap-northeast-2
    Confirm changeset          : True
    Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
    Capabilities               : ["CAPABILITY_IAM"]
    Parameter overrides        : {}

Initiating deployment
=====================

Waiting for changeset to be created..

CloudFormation stack changeset
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Operation                                                    LogicalResourceId                                            ResourceType                                               
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Add                                                        SampleAppdFunctionEventInvokeConfig                          AWS::Lambda::EventInvokeConfig                             
+ Add                                                        SampleAppdFunction                                           AWS::Lambda::Function                                      
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Changeset created successfully. arn:aws:cloudformation:ap-northeast-2:982947632035:changeSet/samcli-deploy1602235720/1bb1b8a3-da32-4de9-9a14-49aaae3f066f


Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]: y

2020-10-09 18:29:04 - Waiting for stack create/update to complete

CloudFormation events from changeset
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus                                ResourceType                                  LogicalResourceId                             ResourceStatusReason                        
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS                            AWS::Lambda::Function                         SampleAppdFunction                            -                                           
CREATE_IN_PROGRESS                            AWS::Lambda::Function                         SampleAppdFunction                            Resource creation Initiated                 
CREATE_COMPLETE                               AWS::Lambda::Function                         SampleAppdFunction                            -                                           
CREATE_IN_PROGRESS                            AWS::Lambda::EventInvokeConfig                SampleAppdFunctionEventInvokeConfig           -                                           
CREATE_IN_PROGRESS                            AWS::Lambda::EventInvokeConfig                SampleAppdFunctionEventInvokeConfig           Resource creation Initiated                 
CREATE_COMPLETE                               AWS::Lambda::EventInvokeConfig                SampleAppdFunctionEventInvokeConfig           -                                           
CREATE_COMPLETE                               AWS::CloudFormation::Stack                    sample-app                                    -                                           
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

CloudFormation outputs from deployed stack
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Outputs                                                                                                                                                                              
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Key                 SampleAppdFunction                                                                                                                                               
Description         Hello World Lambda Function ARN                                                                                                                                  
Value               arn:aws:lambda:ap-northeast-2:982947632035:function:SampleApp                                                                                                    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Successfully created/updated stack - sample-app in ap-northeast-2

Check cloudformation and lambda console for successful deployment.

Invoke function in lambda console

Once Lambda is deployed, you can create test event in console.

You will get successful log data.

Invoke from python app

Now you need to invoke lambda function from your app code. Here are examples you can use.



# invoke_from_app.py
import boto3
import json

SETTINGS = {
    "AWS_DEFAULT_REGION": "ap-northeast-2",
    "ENV": "prod"
}
def main():
    lambda_client = boto3.client('lambda', region_name=SETTINGS['AWS_DEFAULT_REGION'])

    payload = {
        "image_url": "https://picsum.photos/200/300"
    }

    ret = lambda_client.invoke(
        FunctionName="SampleApp",
        InvocationType="DryRun" if SETTINGS['ENV'] == "test" else "Event",
        Payload=json.dumps(payload)
    )
    print(ret)

if __name__ == "__main__":
    main()

Monitoring

Celery or rq provides native or 3rd party too for monitoring such as sentry . There are some options for monitoring lambda functions but SAM application also provides minimal monitoring environment. Go to lambda service and application menu. Select Monitoring tab to dashboard and cloudwatch logs. You can also configure x-ray for tracing.

CD

Let's talk about CD pipeline, I will use conditional trigger on github workflow and run the

Github workflow

following workflow file will build and deploy to lambda. You need to set secret key with AWS credential.



on:
  push:
    branches:
      - master

name: Lambda deployment
jobs:
  deploy:
    name: Lambda deploy
    runs-on: ubuntu-latest

    steps:
    - name: Checkout
      uses: actions/checkout@v2

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: ap-northeast-2

    - name: Build, and deploy
      run: |
        /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
        test -d ~/.linuxbrew && eval $(~/.linuxbrew/bin/brew shellenv)
        test -d /home/linuxbrew/.linuxbrew && eval $(/home/linuxbrew/.linuxbrew/bin/brew shellenv)
        test -r ~/.bash_profile && echo "eval \$($(brew --prefix)/bin/brew shellenv)" >>~/.bash_profile
        brew --version
        brew tap aws/tap
        brew install aws-sam-cli
        sam --version

        sam build
        sam deploy --stack-name sample-app --region ap-northeast-2 -t .aws-sam/build/template.yaml --capabilities CAPABILITY_IAM --no-confirm-changeset --s3-bucket aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1

If you want to implement canary or linear deployment using codedeploy, refer to the following link for more detail.

Deploying serverless applications gradually

Conclusion

Once lambda is deployed, I removed 20% of my app code related to RQ. I also removed broker(Elasticsearch Redis) service anymore. Moreover since I was using ECS and EKS, I can remove additional task and pod for more api tasks and pods. I am pretty happy with the result that I don't need to manage another container on production. If you have simple workers running on RQ or Celery, I recommend that you try this setup. Thank you for reading.

Top comments (3)

jaradc • Feb 4 '22

Good stuff. Thanks for illustrating how to invoke a Lambda function from Python code. This was my starting question. I am trying to remove celery as a dependency which would also remove the need for Redis (in my case) which is a nice decrease in project complexity and long-term cost (if I can do it). What is FasS and RQ? I figured out that DLQ meant "Dead Letter Queue" but was unsure about some of these acronyms you are using.

Ramses Alexander Coraspe • Jul 24 '22

Rabbit Queue?