Intro
When you build a web service, there is a time that you need to implement simple async worker for background process. I am big fan of Django and whenever I need a async worker, I used celery or rq for convenience. celery or other async framework provides a lot of cool features but most of the time all I need is a simple background process that would not interrupt user experiences.
Overtime, as paradigm shifts to cloud, I have been thinking lambda as new async worker solution due to deployment complexity, scaling and cost constraint. Here are pros and cons for using lambda instead of celery on cloud environment(non-local environment).
Advantage
- FasS advantage
- scaling benefit - cost, ops, etc
- Resiliency
- faster development
- No need to maintain Broker (e.g elastic search redis)
- Smaller side effect on code change
- Fully-managed
Disadvantage
- FasS disadvantage
- limited state
- Hard to reuse app code
- Django ORM or settings module
- You can still use them with django package installed but inefficient
- can't reuse code from django app
- Django package is too heavy to use on lambda (still usable but too complex to setup
- Latency on startup
- Json formatted Invocation parameter
- If you need to pass binary or big size paras, need to consider S3 or other medium
Limitation
Most of early stage limitation are lifted as lambda is in mature state (e.g conccurent executions, layers, etc)
- timeout - 15min
- If your async worker take more than 5min, you need to break up codes in to multiple lambda functions or use other distributed system such as EMR, etc.
- Invocation payload
- asynchronous - 256KB
Consideration
- VPC, Subnet, Security Group
- If you are accessing Resources in private VPC, you need to consider placing lambda in same subnet.
- If you are lambda needs to talk to outside world, private subnets needs NAT gateway
- Maintenance
Setup
AWS recently came up with SAM(Serverless Application Model) which utilize cloudformation for easier deployment of lambda application. SAM provides functionality such as API gateway and statemachine, but we will only be using AWS::Serverless::Function resource for our purpose. You can consider using Zappa for other option.
Github
refer to this repo for details
kokospapa8/async-lambda-sample-app
Prerequisite
You need following AWS resource created in order to invoke asynchronous lambda.
DLQ
In order use asynchronous lambda, you need to provide DLQ in case lambda fails and retry. Click link for details. I will create SNS topic in this post.
SECRET
Let's assume that you have mysql db in public VPC or s3 and need to access them in order to process some data, you can add secrets to lambda env, but it is not a good idea to put secret in plaintext for env variable. You can use aws secret manger , ssm parameterstore or vault . I will use ssm parameterstore for this post. Refer to previous post for creating secrets.(parameter store section)
How to deploy django app to ECS Fargate part3
arn:aws:ssm:<region>:<account_id>:parameter/SAMPLE/BUCKET
IAM ROLE
You need lambda execution role with following permission and polcies attached.
Policies
- AWSLambdaExecute
Permission
- SNS – sns:Publish
- kms - kms:Decrypt
- ssm - ssm:GetParameters, ssm:DescribeParameters, ssm:GetParameter
You need following permission if you want to place your lambda function in a designated VPC
- ec2 - ec2:CreateNetworkInterface, ec2:DeleteNetworkInterface, ec2:DescribeSecurityGroups
SAM
What is the AWS Serverless Application Model (AWS SAM)?
Install (on macOS)
Installing the AWS SAM CLI on macOS
$ aws configure
AWS Access Key ID [None]: your_access_key_id
AWS Secret Access Key [None]: your_secret_access_key
Default region name [None]:
Default output format [None]:
$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
$ brew tap aws/tap
$ brew install aws-sam-cli
Development
$ sam init
This code will create following structure
sam-app/
├── README.md
├── events/
│ └── event.json
├── hello_world/
│ ├── __init__.py
│ ├── app.py #Contains your AWS Lambda handler logic.
│ └── requirements.txt #Contains any Python dependencies the application requires, used for sam build
├── template.yaml #Contains the AWS SAM template defining your application's AWS resources.
└── tests/
└── unit/
├── __init__.py
└── test_handler.py
template.yaml
Update template file if you need more info on syntax, refer to the link.
- template.yaml ```
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
sample-app
Sample SAM Template for sample-app
More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 3
Resources:
SampleAppdFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: sample_app/
DeadLetterQueue:
Type: SNS
TargetArn: arn:aws:sns:::sample-dlq
Handler: app.lambda_handler
Runtime: python3.8
Description: sample lambda
EventInvokeConfig:
MaximumEventAgeInSeconds: 60
MaximumRetryAttempts: 2
FunctionName: SampleApp
Role: arn:aws:iam:::role/sample_lambda_execution_role
Environment:
Variables:
S3_URL: ""
Outputs:
SampleAppdFunction:
Description: "Sample app Function ARN"
Value: !GetAtt SampleAppdFunction.Arn
- requirements.txt
boto3
- app.py
This sample app receives image_url as parameter and puts the image on s3bucket name provided on s3 bucket. (AWSLambdaExecute policy has s3 access permission)
import boto3
import os
import requests
def lambda_handler(event, context):
image_url = event['image_url']
S3_BUCKET_PARAM = os.environ['S3_BUCKET_PARAM']
# download
image = requests.get(image_url).content
ssm_client = boto3.client('ssm', region_name="ap-northeast-2")
response = ssm_client.get_parameter(
Name=S3_BUCKET_PARAM,
WithDecryption=True
)
bucket_name = response['Parameter']['Value']
s3_client = boto3.client('s3', region_name="ap-northeast-2")
response = s3_client.put_object(
Bucket=bucket_name,
Key="image.png",
Body=image
)
return response
## Build
Once your logic code is ready, you can build it using following command
```yaml
sam build -t template.yaml --region <region_name>
You can test your function with following command
$ sam local invoke
# if you need to provide environment variable or paramters, use following command
$ sam local invoke -e events/event.json --env-vars env.json
#env.json
{
"SampleAppdFunction": {
"S3_BUCKET_PARAM_ARN": "parameterstore_arn",
}
}
#event.json
{
"image_url": "https://picsum.photos/200/300"
}
Deploy
Once your function is tested locally, let's deploy the function to cloud. You have two options. Enter appropriate response to prompt
$ sam deploy --guided
Configuring SAM deploy
======================
Looking for samconfig.toml : Found
Reading default arguments : Success
Setting default arguments for 'sam deploy'
=========================================
Stack Name [sample-app]: y
AWS Region [ap-northeast-2]: y
#Shows you resources changes to be deployed and require a 'Y' to initiate deploy
Confirm changes before deploy [Y/n]: y
#SAM needs permission to be able to create roles to connect to the resources in your template
Allow SAM CLI IAM role creation [Y/n]: y
Save arguments to samconfig.toml [Y/n]: y
output should look like this
Looking for resources needed for deployment: Found!
Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
A different default S3 bucket can be set in samconfig.toml
Deploying with following values
===============================
Stack name : sample-app
Region : ap-northeast-2
Confirm changeset : True
Deployment s3 bucket : aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
Capabilities : ["CAPABILITY_IAM"]
Parameter overrides : {}
Initiating deployment
=====================
Saved arguments to config file
Running 'sam deploy' for future deployments will use the parameters saved above.
The above parameters can be changed by modifying samconfig.toml
Learn more about samconfig.toml syntax at
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html
Deploying with following values
===============================
Stack name : sample-app
Region : ap-northeast-2
Confirm changeset : True
Deployment s3 bucket : aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
Capabilities : ["CAPABILITY_IAM"]
Parameter overrides : {}
Initiating deployment
=====================
Waiting for changeset to be created..
CloudFormation stack changeset
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Operation LogicalResourceId ResourceType
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Add SampleAppdFunctionEventInvokeConfig AWS::Lambda::EventInvokeConfig
+ Add SampleAppdFunction AWS::Lambda::Function
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Changeset created successfully. arn:aws:cloudformation:ap-northeast-2:982947632035:changeSet/samcli-deploy1602235720/1bb1b8a3-da32-4de9-9a14-49aaae3f066f
Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]: y
2020-10-09 18:29:04 - Waiting for stack create/update to complete
CloudFormation events from changeset
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus ResourceType LogicalResourceId ResourceStatusReason
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS AWS::Lambda::Function SampleAppdFunction -
CREATE_IN_PROGRESS AWS::Lambda::Function SampleAppdFunction Resource creation Initiated
CREATE_COMPLETE AWS::Lambda::Function SampleAppdFunction -
CREATE_IN_PROGRESS AWS::Lambda::EventInvokeConfig SampleAppdFunctionEventInvokeConfig -
CREATE_IN_PROGRESS AWS::Lambda::EventInvokeConfig SampleAppdFunctionEventInvokeConfig Resource creation Initiated
CREATE_COMPLETE AWS::Lambda::EventInvokeConfig SampleAppdFunctionEventInvokeConfig -
CREATE_COMPLETE AWS::CloudFormation::Stack sample-app -
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CloudFormation outputs from deployed stack
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Outputs
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Key SampleAppdFunction
Description Hello World Lambda Function ARN
Value arn:aws:lambda:ap-northeast-2:982947632035:function:SampleApp
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Successfully created/updated stack - sample-app in ap-northeast-2
Check cloudformation and lambda console for successful deployment.
Invoke function in lambda console
Once Lambda is deployed, you can create test event in console.
You will get successful log data.
Invoke from python app
Now you need to invoke lambda function from your app code. Here are examples you can use.
# invoke_from_app.py
import boto3
import json
SETTINGS = {
"AWS_DEFAULT_REGION": "ap-northeast-2",
"ENV": "prod"
}
def main():
lambda_client = boto3.client('lambda', region_name=SETTINGS['AWS_DEFAULT_REGION'])
payload = {
"image_url": "https://picsum.photos/200/300"
}
ret = lambda_client.invoke(
FunctionName="SampleApp",
InvocationType="DryRun" if SETTINGS['ENV'] == "test" else "Event",
Payload=json.dumps(payload)
)
print(ret)
if __name__ == "__main__":
main()
Monitoring
Celery or rq provides native or 3rd party too for monitoring such as sentry . There are some options for monitoring lambda functions but SAM application also provides minimal monitoring environment. Go to lambda service and application menu. Select Monitoring tab to dashboard and cloudwatch logs. You can also configure x-ray for tracing.
CD
Let's talk about CD pipeline, I will use conditional trigger on github workflow and run the
Github workflow
following workflow file will build and deploy to lambda. You need to set secret key with AWS credential.
on:
push:
branches:
- master
name: Lambda deployment
jobs:
deploy:
name: Lambda deploy
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ap-northeast-2
- name: Build, and deploy
run: |
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
test -d ~/.linuxbrew && eval $(~/.linuxbrew/bin/brew shellenv)
test -d /home/linuxbrew/.linuxbrew && eval $(/home/linuxbrew/.linuxbrew/bin/brew shellenv)
test -r ~/.bash_profile && echo "eval \$($(brew --prefix)/bin/brew shellenv)" >>~/.bash_profile
brew --version
brew tap aws/tap
brew install aws-sam-cli
sam --version
sam build
sam deploy --stack-name sample-app --region ap-northeast-2 -t .aws-sam/build/template.yaml --capabilities CAPABILITY_IAM --no-confirm-changeset --s3-bucket aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
If you want to implement canary or linear deployment using codedeploy, refer to the following link for more detail.
Deploying serverless applications gradually
Conclusion
Once lambda is deployed, I removed 20% of my app code related to RQ. I also removed broker(Elasticsearch Redis) service anymore. Moreover since I was using ECS and EKS, I can remove additional task and pod for more api tasks and pods. I am pretty happy with the result that I don't need to manage another container on production. If you have simple workers running on RQ or Celery, I recommend that you try this setup. Thank you for reading.
Top comments (3)
Good stuff. Thanks for illustrating how to invoke a Lambda function from Python code. This was my starting question. I am trying to remove celery as a dependency which would also remove the need for Redis (in my case) which is a nice decrease in project complexity and long-term cost (if I can do it). What is FasS and RQ? I figured out that DLQ meant "Dead Letter Queue" but was unsure about some of these acronyms you are using.
Rabbit Queue?
what is RQ?