In this article we wanted to share an overview of a recent project to build a logistics platform from scratch using AWS cloud, Go and serverless. Yes, there is already a great abundance of blog posts focusing on all of these individual technologies. What we find often lacks is how to glue all of those together, which is exactly what we describe in this post.
Here is our full code supporting this blog post.
Pre-Requisites
If you want to try to play around with our examples, you'll need to set up those things in case you don't have already set it up:
- AWS account
- AWS CLI
- AWS SAM CLI
- Terraform
- Docker
- Make
- AWS VPC
- AWS S3 buckets
To be able to completely follow up an article, some basic understanding of those tools and frameworks would be nice, but it's not required.
The concept of this post and examples is to give you an idea how you can set up your own architecture and use those techniques.
Overview - architecture
So basically our architecture consists of:
- AWS Lambdas - Go runtime running our business logic
- AWS S3 buckets - for hosting frontend and Terraform backend configuration
- AWS RDS (Postgres version 12) - instance for storing business related data
- AWS DynamoDB - used for Terraform locking mechanism, preventing multiple users to apply simultaneously
- AWS SSM - used to connect to our RDS instance within private VPC to be able to monitor and to maintain data
- AWS Cognito - used for identity management and authentication
- Terraform and AWS SAM (Serverless Application Model) as glue to keep all states synced and up-to-date
From an outside networking point of view, we decided to go with cloudflare as a DNS service. The main benefit of using cloudflare is it’s DDoS protection. The experiences that we had previously with cloudflare were very good, so it was logical to stay with cloudflare. Most of all, adding and maintaining DNS records, we found to be user friendly even for one’s without a networking background.
With this feature on, one thing less to think about while designing a solution. Last awesomeness is that Cloudflare solutions/features (such as automatically setting up SSL certificates for our (sub)domains) are completely free.
The whole project is handled by the GitLab CI/CD pipeline.
You can get free minutes on Gitlab to run your own CI/CD pipelines, when it comes to small projects, this is a fair deal.
You can see in the next picture how we designed a solution for the project.
This illustration represents an overview of the technical solution.
Programming Language
Regarding the programming language choice, Go seemed like an easy choice: low memory footprint (allowing thin run times), fast startup times, static typing, and ease of usage and readability. A lot of work in this platform involved integration with different partners, so the simplicity in Go’s concurrency model was also a big appeal.
Regarding ORM side of things, in the mkops project this was handled by a java 11 backend based on spring boot and hibernate, and go was a pure stateless supporting microservice. This time we had a chance to dive into GORM to model our entities persisted on a Postgres 12.
Serverless model
Due to the nature of most of our projects (related to big data), our default infrastructure goto platform is kubernetes. Here, however, a serverless model for the API provided a quite nice fit.
After reviewing all the workflows that needed to happen, it was clear to us: the logistics platform would be heavy on the background processes that integrate between different platforms that could be started in a cron fashion. The database provided all state locking required, so the processes could run independently and without communicating to each other.
Another argument was the relative low amount of user traffic expected. The cost and administration effort of running a standalone and permanent fargate instance (ECS/EKS) did not add up.
In summary, the motives for our choice are much of the usual suspects: low administration effort, higher security due to ephemeral nature, and lower operating costs.
In order to provide a seamless local development environment AWS provides a cool framework named AWS SAM. It allows you to run AWS Lambdas locally, and, most importantly, with an option to easily create the relevant Service events like API Gateway. So our solution for running things locally and for testing involved Docker with SAM template and of course Localstack for simulating other AWS resources that we were using, such as AWS SES, AWS S3 and AWS SecretManager
.
AWS SAM
In the spirit of keeping low a maintenance effort while having a speedy implementation, we decided to start by considering serverless frameworks. The two main options boiled down to AWS SAM or the Serverless Framework.
The main difference between the two is that Serverless is used to deploy FaaS (Function as a Service) functions with support for different providers. SAM, on the other hand, is used specifically for AWS as a cloud provider, deploying not only cloud functions but also the (minimum) underlying infrastructure to expose some business logic with an HTTP Endpoint.
AWS SAM allows us to spin up (most) of the resources it manages locally, including Lambdas and the API Gateway. With this in mind, it made it easier for us to develop and test Lambda functions before deploying them to AWS.
We were fairly set on using AWS as a cloud provider, which helped our decision to go for AWS SAM.
AWS SAM follows a declarative approach for defining the stack and makes use of a command line utility to deploy: resources are declared in one key file called template.yaml
. More details on the anatomy of the AWS SAM template here.
Under the hood, AWS SAM is an oversimplified version of CloudFormation albeit very similar. Resources declared in the template.yaml
get automatically converted, by the sam cli to CloudFormation so it’s very easy to inspect what exactly is getting created on our behalf.
After installing sam cli
for your operating system, these are the commands that you will need to create AWS resources are:
- sam package --template-file template.yaml --s3-bucket your_s3_bucket --output-template-file package.yaml
- sam deploy --template-file package.yaml --stack-name your_stack_name --capabilities CAPABILITY_IAM
sam package is used for converting the template.yaml
into deployable artifacts as well as uploading said artifacts to an AWS S3 bucket. It’s important to note that this bucket needs to be created beforehand.
An important caveat to these deployments to note is that there is a hard limit of how much your total package size can be: 250MB. Meaning, if you have multiple lambdas defined, and their total size exceeds the limit, SAM will throw an error. If you are tempted to create an individual function for each endpoint, then it’s likely you will reach this limit quite fast. One way to solve this, is to use your lambas as docker images (a recent feature of Lambdas, where the hard limit is 10GB),upload it to AWS ECR and reference it in the SAM template or mount EFS volumes within Lambda. More details you can find here. Another way is to write a single lambda with its own router and point all traffic there (read more for an example).
The output generated from command sam package
is the input template file for next command, sam deploy
.
To delete your resources created by AWS SAM run following command:
aws cloudformation delete-stack --stack-name *your_stack_name*
Here is an example of the AWS SAM template.yaml file. The file is used to create AWS resources like AWS Lambda, and API Gateway with Custom domain through which lambda will be exposed on the Internet.
We’re setting up a Custom domain for API Gateway to be exposed to the Internet over a user-friendly domain name, instead of a generic one created by AWS.
---
AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Parameters:
TargetStage:
Description: "dev/prd"
Type: String
DomainName:
Type: String
AcmCertificateArn:
Type: String
VPCSecurityGroupIDs:
Description: "An comma-delimited list of strings - the security groups that your Lambda function should be in"
Type: CommaDelimitedList
VPCSubnetIDs:
Description: "An comma-delimited list of strings - the subnet IDs that your Lambda function should be assigned to"
Type: CommaDelimitedList
Globals:
Api:
Cors:
AllowMethods: "'GET, POST, PUT, OPTIONS, DELETE'"
AllowHeaders: "'*'"
AllowOrigin: "'*'"
Function:
Runtime: go1.x
Tracing: Active # https://docs.aws.amazon.com/lambda/latest/dg/lambda-x-ray.html
Timeout: 30
VpcConfig:
SecurityGroupIds:
Ref: VPCSecurityGroupIDs
SubnetIds:
Ref: VPCSubnetIDs
Resources:
ApiDetails:
Type: AWS::Serverless::Api
Properties:
StageName: !Ref TargetStage
Auth:
UsagePlan:
CreateUsagePlan: PER_API
Description: Usage plan for this API
Quota:
Limit: 3000
Period: MONTH
Throttle:
BurstLimit: 50
RateLimit: 20
ApiDomain:
Type: AWS::ApiGateway::DomainName
Properties:
RegionalCertificateArn: !Ref AcmCertificateArn
DomainName: !Ref DomainName
EndpointConfiguration:
Types:
- REGIONAL
SecurityPolicy: TLS_1_2
ApiDomainMappings:
Type: AWS::ApiGateway::BasePathMapping
Properties:
DomainName: !Ref DomainName
RestApiId: !Ref ApiDetails
Stage: !Ref ApiDetails.Stage
EchoFunction:
Type: AWS::Serverless::Function
Properties:
Handler: bin/echo-sample
MemorySize: 128
Events:
AllEvents:
Type: Api
Properties:
Path: /{proxy+}
Method: any
RestApiId: !Ref ApiDetails
Outputs:
ApiCustomDomainRegionalDomainName:
Description: 'Regional domain name for the API'
Value: !GetAtt ApiDomain.RegionalDomainName
As you can see in example, there are some sections defined like Parameters, Globals, Resources, Outputs
. More details about each section you can read here.
To set up some parameters or override them, you can use option --parameter-overrides
for aws sam deploy command. Like an example: sam deploy --stack-name test--template-file packaged.yaml --capabilities CAPABILITY_IAM --parameter-overrides TargetStage=dev
, where we’re setting to parameter TargetStage
a value dev
.
The section of interest is the Resource
: this is where we describe how our API Gateway, Custom Domain
used for API Gateway and our simple Lambda should look like.
It’s also worthwhile mentioning that, for each API, you can define an Invocation Quota which can be both (1) a way to protect your deployment and (2) avoid cost escalation.
In this example we’re setting API Gateway to Custom Domain. To do so, we’re defining two resources: ApiDomain
and ApiDomainMappings
. To completely set up a custom domain on API Gateway you have to be an owner of that domain and it doesn’t need to be registered on AWS Route53 - it can be any DNS Registrar. We’re using Cloudflare as our DNS Registrar, so in this example we will use the same DNS.
Next step is to create an ACM certificate. AWS Certificate Manager (ACM) is a service that provisions and manages SSL certificates on AWS. This is a prerequisite if we want to do anything custom domain related in API Gateway. Note: you have to create an ACM certificate in the same AWS region as your API Gateway will be, if you are using an API endpoint type as regional as we do in our example
.
In our case, we’re using Cloudflare, and our domain is registered on Cloudflare service. In that case, we're going to need to create an SSL certificate on Cloudflare and import that one into AWS ACM.
Here is a blog post from Cloudflare which can help you to create a certificate, it’s pretty straightforward. After successfully creating an origin certificate, you need to store the origin certificate and private key generated for you in PEM format. You will need this data for importing the certificate into AWS ACM. Note if you don’t save a private key and exit from the page with the generated one, you won’t be able to retrieve it again. In that case you will have to first revoke the certificate and then create a new one again.
To import an existing origin certificate published by Cloudflare, follow these instructions.
Save the newly generated ARN key of the imported certificate, you will assign it in the SAM template in resource ApiDomain
.
Note that your custom DomainName property in resource ApiDomain must match the one that was added in the ACM certificate, otherwise it won't be created. In this example we’re using an SSL certificate for api-test.your.domain.com, so we have to set the parameter DomainName exactly like that subdomain.
The last thing to set up for API Gateway and custom domain is to define resource ApiDomainMappings
. Basically it will allow us to point the custom domain at an existing API Gateway. It’s a simple mapping that lets us point API stages at specific paths. This is a useful feature if you have multiple APIs making up a suite that you want behind a shared domain.
We ran into an issue while setting up Stage property. At first we tried to simply add value as reference to Parameter Stage: !Ref TargetStage
, but we got a strange error saying >“Invalid stage identifier specified”. After some digging we found a github topic with the same problem.
The problem lies in that creating a Stage resource is outside of the SAM template and with just referencing it to name will fail, because the resource ApiDomainMappings
it’s being created before resource Stage. Thanks to fellows on github, this suggestion worked.
So we changed Stage property to: Stage: !Ref ApiDetails.Stage
.
The very last thing to finish setting up API Gateway is to update over DNS with new records pointing to the domain name of API Gateway generated by AWS.
Generic domain name looks like https://api-id.execute-api.region.amazonaws.com/stage, where api-id is generated by API Gateway, region (AWS Region) is specified by you when creating the API, and stage is specified by you when deploying the API.
All you have to do is to login to your Cloudflare account and add a new CNAME
record with a subdomain that matches the one you decided to add into the SAM template in DomainName
parameter and pointing to the generated domain name of your API Gateway.
With this set up, you just provide to your developers newly created user-friendly REST API endpoint (exp. https://app-test.your-domain.com/stage).
If you want to avoid manually adding new DNS records to your Cloudflare provider, this is an example of how we managed to automate adding new records to Cloudflare.
resource "null_resource" "get_api_gateway_endpoint" {
triggers = {
template = sha1(file("../../../backend/template.yaml"))
stack_name = local.stack_name
}
provisioner "local-exec" {
interpreter = ["bash", "-c"]
command = "aws cloudformation describe-stacks --stack-name ${self.triggers.stack_name} | jq '.Stacks[0].Outputs[0].OutputValue'| sed 's/\"//g' > ${path.module}/gateway_endpoint.txt"
}
depends_on = [null_resource.build_deploy_sam_resource]
}
In this part, we used null_resource to call aws command aws cloudformation describe-stacks
to get API Gateway domain name (we defined Output section in the AWS SAM template as what it should look like) and store it in file. With sed command we’re stripping away https:// from domain name, and only leaving domain name that we going to add to new DNS record.
data "local_file" "api_gateway_endpoint" {
filename = "${path.module}/gateway_endpoint.txt"
depends_on = [null_resource.get_api_gateway_endpoint]
}
resource "cloudflare_record" "api_gateway_endpoint" {
depends_on = [data.local_file.api_gateway_endpoint]
name = var.subdomain_name_backend
value = data.local_file.api_gateway_endpoint.content
type = "CNAME"
proxied = true
zone_id = lookup(data.cloudflare_zones.default.zones[0], "id")
}
Using Terraform resource cloudflare_record, we set the value as content of the file that we generated above (it’s value where our CNAME record will point to), and the name value represents the subdomain that we want to use for our REST API endpoint.
The last resource defined in the SAM template example is our lambda EchoFunction.
Key things to keep in mind is to set up a lambda handler (pointing to Go executable file located in our project structure) and events (defining what kind of type will trigger our lambda).
In this example we are using the API as a trigger for our lambda.
Type: Api
Properties:
Path: /{proxy+}
Method: any
RestApiId: !Ref ApiDetails
As you can see, we defined that any path and method can trigger our lambda over API Gateway that we had previously created. With this approach and with using this solution with the Go Echo framework, we reduced the number of lambdas (also with this approach we solved the problem with a hard limit of total size that all lambdas can have). Using the aws-lambda-go-api-proxy with echo framework we defined a REST controller that is managing real paths and methods that our lambda consumes.
Note, if your lambda needs more permissions to access some other AWS resources, you can set up Role property. One way you can create a role with Terraform and then create a property in the SAM template to pass it ARN of the created role and reference it to Role property.
Another way is to create a role inside of a SAM template and reference it to the Role property.
To expose your lambda over API Gateway the last thing to do is to enable, you already guess, CORS
. To do so, all you have to is to define CORS section in the SAM template. In our case we set up as globals, so our API Gateway will automatically have CORS enabled.
Even with enabled CORS on API Gateway, you still have to handle in your lambda CORS headers, otherwise invocation of your lambda will not work.
With the Echo framework we created CORS middlerware to make our lambda functional.
Here is an example how to do it:
e := echo.New()
e.Use(middleware.CORSWithConfig(middleware.CORSConfig{
AllowOrigins: []string{"*"},
AllowMethods: []string{echo.GET, echo.PUT, echo.POST, echo.DELETE, echo.OPTIONS},
AllowHeaders: []string{"Accept", "Content-Type", "Content-Length", "Accept-Encoding", "X-CSRF-Token", "Authorization"},
}))
AWS Lambda
Lambda is the Serverless computing platform service provided on AWS. Using a Lambda function you can run or execute your application code without actually provisioning any application servers.
Since December of 2020, AWS has increased memory that can be assigned to lambda function to max 10GB and increased the number of vCPU to allocate to max 6 vCPU.
Also a new cool feature is that you can now package and deploy Lambda functions as container images of up to 10 GB in size. If you have some big lambda function with huge dependencies, you can then easily use this approach to solve the situation where your lambda handler can have a max file size of 250MB.
The way we approach this hard limit, we used the aws-lambda-go-api-proxy solution to packt multiple lambas into one. Basically this solution allows us to use one lambda to handle multiple methods or paths for one API Gateway, so we don’t need to have multiple lambdas for each path or method. It makes it easier to maintain a code base.
Very cool thing is, if you have already developed your REST API with Echo framework
or some other like Gin, Iris, Negroni, Mux, Fiber or Chi
and you’re planning to move to serverless (AWS Lambda, GCP Cloud Function, ...) you can easily achieve it by using this solution.
If usage goes up and it becomes price prohibitive, it's a one line change to swap it to a container & run via AWS services like Fargate, ECS.
In the next example we’ll show you how to use the Echo framework with aws-lambda-go-api-proxy to achieve the solution of having one lambda function for multiple paths and methods.
We have defined two paths, /user and /organization, with two supported methods each.
package main
import (
"context"
"net/http"
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
echoadapter "github.com/awslabs/aws-lambda-go-api-proxy/echo"
"github.com/labstack/echo/v4"
"github.com/labstack/echo/v4/middleware"
)
type UserData struct {
UserId string `json:"userId"`
DisplayName string `json:"displayname"`
Status string `json:"status"`
}
type User struct {
User UserData `json:"user"`
}
type OrganizationData struct {
OrganizationId string `json:"organizationId"`
DisplayName string `json:"displayname"`
Status string `json:"status"`
}
type Organization struct {
Organization OrganizationData `json:"organization"`
}
var echoLambda *echoadapter.EchoLambda
func init() {
e := echo.New()
//define routes
registerUserRoutes(e)
registerOrganizationRoutes(e)
//define CORS
e.Use(middleware.CORSWithConfig(middleware.CORSConfig{
AllowOrigins: []string{"*"},
AllowMethods: []string{echo.GET, echo.PUT, echo.POST, echo.DELETE, echo.OPTIONS},
AllowHeaders: []string{"Accept", "Content-Type", "Content-Length", "Accept-Encoding", "X-CSRF-Token", "Authorization"},
}))
e.Use(middleware.Logger())
echoLambda = echoadapter.New(e)
}
func handler(ctx context.Context, req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
return echoLambda.ProxyWithContext(ctx, req)
}
func main() {
lambda.Start(handler)
}
func registerUserRoutes(e *echo.Echo) {
user := e.Group("/user")
user.GET("", getUser)
user.POST("", createUser)
}
func registerOrganizationRoutes(e *echo.Echo) {
organization := e.Group("/organization")
organization.GET("", getOrganization)
organization.PUT("", updateOrganization)
}
func getUser(c echo.Context) error {
return c.JSON(http.StatusOK, User{
User: UserData{UserId: "1", DisplayName: "Test", Status: "active"},
})
}
func createUser(c echo.Context) error {
return c.JSON(http.StatusOK, User{
User: UserData{UserId: "2", DisplayName: "Test2", Status: "created"},
})
}
func getOrganization(c echo.Context) error {
return c.JSON(http.StatusOK, Organization{
Organization: OrganizationData{
OrganizationId: "1",
DisplayName: "Test organization",
Status: "active",
},
})
}
func updateOrganization(c echo.Context) error {
return c.JSON(http.StatusOK, Organization{
Organization: OrganizationData{
OrganizationId: "1",
DisplayName: "Test organization",
Status: "updated",
},
})
}
This simple lambda function is returning a JSON response in case you have sent a request to the correct path using the correct method.
One cool way of using AWS Lambda is for running migration codes. AS mentioned earlier, we are using GORM as an ORM library for modeling and interacting with RDS-based Postgres.
GORM has the ability to define data models which represent database tables with appropriate constraints, which allows us to have a dedicated Lambda that runs migration/creations of resources (such as tables, indexes, constraints, pre-populated data, etc) in the database without the need to maintain incremental SQL scripts. If you are curious about GORM checkout their documentation here (bonus: major version 2 was recently released with some very nice improvements).
Local testing
To test our lambda functions locally we’re using Docker
and AWS SAM
.
As we’re using Go runtime environment for our lambda functions, we have defined Docker file with Go base image and also installed awscli
and asw-sam-cli
tools.
FROM golang:1.15
RUN apt-get update
RUN apt-get install python3 python3-pip -y
RUN pip3 install --upgrade pip
RUN pip3 install awscli
RUN pip3 install aws-sam-cli==1.12
WORKDIR /var/opt
EXPOSE 3003
Next step is to create a docker-compose
service that will use our docker base image defined in Dockerfile
, and serve our Lambda function with an API Gateway-like local deployment.
In docker-compose
service we’re mounting our backend folder which includes SAM template.yaml and the lambda executable. To run a lambda function with API Gateway defined in template.yaml, all you have to do is to define an entrypoint to use SAM command sam local start-api
.
version: "3.5"
services:
backend:
build:
context: ./backend
dockerfile: Dockerfile
image: backend:local
hostname: backend
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./backend:/var/opt
working_dir: /var/opt
environment:
# don’t share statistics usage with AWS
SAM_CLI_TELEMETRY: 0
LOG_LEVEL: DEBUG
ports:
- "3003:3003"
networks:
- sample
entrypoint:
- /bin/bash
- -c
- |
sam local start-api \
--host 0.0.0.0 -p 3003 \
--docker-volume-basedir "$PWD"/backend \
--docker-network sample
networks:
sample:
name: sample
To start local development api, just use command docker-compose up
, or our make command make up-be
.
The API is now ready to be tested.
If you try to run a simple curl command curl http://localhost:3003/user
you will get a response from the api.
Cool thing about using SAM local command in combo with docker-compose, is that you can easily define new docker-compose services like postgres, localstack (simulate other AWS services) or frontend and connect your lambda function to those services.
This approach allows us to have a full local development cycle without the need to deploy to AWS to test.
Terraform
Terraform is a general purpose infrastructure as code tool that can create infrastructure on different cloud providers like AWS, GCP, AZURE and Oracle Cloud Infrastructure.
Besides support for cloud providers, terraform has support for plenty of different kinds of providers, such as Cloudflare, Helm, etc.
If interested, you can find a full list of supported providers here.
Having the very useful modules system, it's relatively straightforward to deploy large infrastructures to any kind of cloud provider.
We start with Terraform, as it is the glue for all these services and frameworks; not just because we’re big fans of it. In practice we are stitching AWS and Cloudflare together; and, in a clunky way, AWS SAM. Yes, by far not our favorite solution, using null_resource
.
At the moment of writing this post there is no better option to run AWS SAM via Terraform, or at least we couldn’t find one. Hoping in future Terraform will support creating SAM resources directly within a native provider.
To glue things up, we go with the previous release of Terraform, at the moment of writing it was version 0.12. To find out some cool features that were released with this version or newer versions, you can read about in our blog post here.
If you take a closer look at our architecture, you’ll notice that Terraform is responsible directly or indirectly for managing all resources that run on AWS.
It’s directly responsible for creating AWS VPC, AWS RDS, AWS Cognito, AWS S3 buckets, AWS DynamoDB, AWS CloudFront distribution, AWS SSM and Cloudflare DNS records
.
Indirectly, it’s responsible for creating AWS Lambdas, AWS API Gateway. Meaning that the creation of those resources is managed by AWS SAM cli, defined by SAM template via a null_resouce
.
To manage different stages, we’re using the Terraform workspaces. We choose a strategy of naming Terraform workspaces by names of our branches, allowing us to have simple flow, mostly because of managing resources by Gitlab CI/CD.
Below follows an example of how you could deploy SAM resources using a null_resource
within your Terraform scripts. Essentially, you need to run both package
and deploy
commands and ensure all parameters (e.g. VPC, SGs, etc.) are injected into the deployment. To ensure the resource is called when the dependent files change, one could use the sha1
of template as a trigger (as shown).
resource "null_resource" "build_deploy_sam_resource" {
triggers = {
s3_bucket = var.s3_bucket_artifacts
template = sha1(file("../../../backend/template.yaml"))
stack_name = local.stack_name
sec_group_ids = join(",", aws_security_group.lambda_sg.*.id)
subnet_ids = join(",", var.private_subnet_ids)
domain_name = "${var.subdomain_name_backend}.${var.domain_name}"
target_stage = "dev"
}
provisioner "local-exec" {
interpreter = ["bash", "-c"]
command = <<-EOT
sam package --template-file ${local.template} --s3-bucket ${self.triggers.s3_bucket} --output-template-file ${local.packaged}
sam deploy --stack-name ${self.triggers.stack_name} --template-file ${local.packaged} --capabilities CAPABILITY_IAM --parameter-overrides TargetStage=${self.triggers.target_stage} DomainName=${self.triggers.domain_name} VPCSecurityGroupIDs=${self.triggers.sec_group_ids} VPCSubnetIDs=${self.triggers.subnet_ids} AcmCertificateArn=${aws_acm_certificate_validation.backend.certificate_arn}
EOT
}
depends_on = [aws_security_group.lambda_sg, aws_acm_certificate_validation.backend]
}
Cron jobs
Another use case we needed to implement required some business logic to run on a schedule, which you can also create easily with SAM - which will create a CloudWatch alarm to trigger the appropriate Lambda. AWS supports two ways of defining scheduled time to trigger some AWS resources. One is to use cron expression - in format cron(Minutes Hours Day-of-month Month Day-of-week Year)
. Second one is to use rate expression - in format rate(Value Unit)
, Where Value is a positive integer and Unit can be minute(s), hour(s), or day(s).
More details on how to define schedule expressions can be found on the AWS site.
Keep in mind that rate expression is being executed when a rule is created. For example, if you set up a scheduler to run every 1 hour using keyword rate, rate(1 hour)
, the next run time will be the time when your rule was created plus 1 hour. Let's say you created a rule at 10:20 AM, so the next run will occur at 11:20 AM.
An example of how to define scheduled resources within the SAM template.
ScheduledLambda:
Type: AWS::Serverless::Function
Properties:
Role: !Ref Role
Timeout: 30
Handler: bin/scheduled_lambda
Events:
ScheduleExample:
Type: Schedule
Properties:
Schedule: cron(0/30 * * * ? *)
Description: "Example of cron jobs that runs lambda every 30 min"
The most crucial part to define scheduler is assigning type as Scheduler and setting valid cron or rate values for Schedule properties.
Events:
ScheduleExample: #custom name of event
Type: Schedule #mandatory so AWS SAM can know what kind of event this gonna be
Properties:
Schedule: cron(0/30 * * * ? *)
Inside the Schedule property you can define your own expressions.
Here is the list of all properties that you can assign to the cloudwatch event resource in the SAM template.
AWS SSM Sessions
After setting up our environment on AWS, we were looking at how to access a database located in a private network. We found System Manager remote sessions
service is a very nice solution, because it allows us to have access to private networks without exposing any service publicly. This is great for troubleshooting purposes, accessing our database.
Also allowing us easily create a cron job for backuping databases or any kind of extra work that is required for maintenance.
To be able to create SSM sessions/tunnels, you need to create an EC2 instance running the SSM Agent
. Note that SSM Agent is preinstalled, by default, on Amazon's AMIs. In our case,we used Amazon Linux 2 for convenience Of course, in our case, we are using Terraform to spawn an instance.
In the following example, we’ll show you how to create this subset of our stack, including the installation of a Postgres client in the EC2 instance.
locals {
// internal user used for admin maintenance tasks
ssm_instance_user = "example"
ssm_instance_group = "example"
}
/**
* SSM instance security group
*/
resource "aws_security_group" "ssm_instance" {
vpc_id = var.vpc_id
name = "ssm-sg"
description = "Allow egress from SSM Agent to Internet."
tags = {
"Name" = "ssm-sg"
}
}
resource "aws_security_group_rule" "ssm_instance_allow_egress_https" {
type = "egress"
from_port = 443
to_port = 443
protocol = "TCP"
cidr_blocks = ["0.0.0.0/0"]
security_group_id = aws_security_group.ssm_instance.id
}
resource "aws_security_group_rule" "ssm_instance_allow_egress_http" {
type = "egress"
from_port = 80
to_port = 80
protocol = "TCP"
cidr_blocks = ["0.0.0.0/0"]
security_group_id = aws_security_group.ssm_instance.id
}
data "aws_iam_policy_document" "ssm_instance_default" {
statement {
effect = "Allow"
actions = ["sts:AssumeRole"]
principals {
type = "Service"
identifiers = ["ec2.amazonaws.com"]
}
}
}
resource "aws_iam_role" "ssm_instance" {
name = "ssm-iam-role"
assume_role_policy = data.aws_iam_policy_document.ssm_instance_default.json
}
resource "aws_iam_role_policy_attachment" "ssm_instance_policy" {
role = aws_iam_role.ssm_instance.name
policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}
resource "aws_iam_instance_profile" "ssm_instance" {
name = "ssm-iam-instance-profile"
role = aws_iam_role.ssm_instance.name
}
data "template_file" "install_ssm_instance" {
template = file("install-ssm-instance.yaml")
vars = {
user = local.ssm_instance_user
group = local.ssm_instance_group
}
}
/*
* Ref: https://registry.terraform.io/providers/hashicorp/template/latest/docs/data-sources/cloudinit_config
*/
data "template_cloudinit_config" "ssm_instance_config" {
part {
content_type = "text/cloud-config"
content = data.template_file.install_ssm_instance.rendered
}
// cloud-init has apparently a 8 year unresolved issue (face-palm) [1], and is unable
// to create users before write_files directive . Thus, we use this hack.
// [1] - https://bugs.launchpad.net/cloud-init/+bug/1486113
part {
content_type = "text/x-shellscript"
content = "/usr/bin/install-pg.sh"
}
}
resource "aws_instance" "ec2" {
ami = "ami-0d712b3e6e1f798ef"
instance_type = "t2.micro"
subnet_id = var.private_subnet_ids[0]
vpc_security_group_ids = [aws_security_group.ssm_instance.id]
iam_instance_profile = aws_iam_instance_profile.ssm_instance.name
root_block_device {
delete_on_termination = true
volume_type = "gp2"
volume_size = 20
}
user_data = data.template_file.install_ssm_instance.rendered
tags = {
"Name" = "ssm-ec2"
}
}
Example of template install-ssm-instance.yaml
#cloud-config
# See docs for more details: https://cloudinit.readthedocs.io/en/latest/topics/examples.html
# Upgrade database on first boot (run 'apt-get upgrade').
package_upgrade: true
users:
- default
- name: ${user}
gecos: ${user}
shell: /bin/bash
primary_group: ${group}
sudo: ALL=(ALL) NOPASSWD:ALL
groups: users, admin
lock_passwd: false
# download & install following packages
packages:
- curl
write_files:
- permissions: '0750'
owner: root:root
content: |
#!/bin/bash
set -euo pipefail
tee /etc/yum.repos.d/pgdg.repo<<EOF
[pgdg12]
name=PostgreSQL 12 for RHEL/CentOS 7 - x86_64
baseurl=https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-7-x86_64
enabled=1
gpgcheck=0
EOF
yum makecache
yum install -y postgresql12 postgresql12-server
chown ${user}:${group} -R /home/${user}/
path: /usr/bin/install-pg.sh
final_message: "The system is finally up, after $UPTIME seconds"
After setting up an EC2 instance with the installed postgres client, the next step is to connect to the EC2 instance, using the following command.
aws ssm start-session --target “instance name” - instance name is you EC2 instance name
In case you are having problems remembering the EC2 instance name, you can easily use make command to make it more straightforward.
Basically everything that you need is EC2 tag value, so using aws cli command to describe EC2 instance filtering by tag will get us the desired instance name.
After everything is ready, to get connected to your EC2 instance, all you have to do is run make command.
In our example we are calling that command ssm-instance, so running make ssm-instance
, you will get to the place you wanted to be :).
Here is an example how can you achieve it using make:
SSM_TAG=ssm-ec2
SSM_INSTANCE := $(shell aws ec2 describe-instances --filter "Name=tag:Name,Values=$(SSM_TAG)" --query "Reservations[].Instances[?State.Name == 'running'].InstanceId[]" --output text)
.PHONY: ssm-instance
ssm-instance:
@echo Connecting to SSM INSTANCE
@aws ssm start-session --target $(SSM_INSTANCE)
The last thing is to verify if the install-pg.sh
script has been created on your EC2 instance by the Terraform example above.
Just type sudo -s
in the terminal and run command ./install-pg.sh
. Now your postgres client should be downloaded and installed. Voila!
To verify that you have successfully installed postgres client, just run command psql
.
In this example you saw how you can use AWS SSM and the EC2 running instance to get deployed to a private VPC.
Cost
If you are a startup at an early stage, then you should definitely consider applying to the AWS startup program as early as possible. While an AWS VPC itself doesn’t cost anything, it does cost to have the bare minimum resources for it to be production ready (e.g. Private/Public subnets with NAT Gateway). We have set up a VPC with a single NAT and internet gateway as a shared resource for our multiple stages (development, integration, production). We agree it’s not ideal, but for projects at early stages, we were avoiding escalation on expenses - we can all agree on this :).
Besides the NAT Gateway, our biggest contributors to cost are of course the encrypted RDS instances db.t2.small
.
To sum up, this basic (but very functional) setup costs us around 100´euros per month, which is very fair considering we have multiple stages and the encryption used for all possible resources (RDS, S3, secrets).
Cloudflare
There is a bit of a challenge using Cloudflare strict mode with frontend deployed on AWS S3 bucket. Cloudflare's strict mode requires that the origin server needs to have an SSL certificate and in case of using AWS S3 as hosting there is a bit of a problem.
As you might know, AWS S3 bucket is publicly exposed only on HTTP protocol.
So to overcome this issue, we needed to create an SSL certificate on AWS Route53 associated with our publicly exposed AWS S3 bucket, alongside with AWS Cloudfront distribution
applying that SSL certificate. Basically AWS Cloudfront distribution allows us to expose AWS S3 buckets over HTTPS protocol.
In this example we are creating an ACM certificate for a domain that will be used for our frontend exposed via Cloudfront distribution.
provider "aws" {
alias = "virginia"
region = "us-east-1"
}
This provider in region us-east-1
needs to be defined so an ACM certificate can be created.
resource "aws_acm_certificate" "default" {
provider = aws.virginia
domain_name = "${var.subdomain_name}.${var.domain_name}"
validation_method = "DNS"
lifecycle {
create_before_destroy = true
}
}
data "cloudflare_zones" "default" {
filter {
name = var.domain_name
}
}
resource "cloudflare_record" "validation_domain" {
name = aws_acm_certificate.default.domain_validation_options[0]["resource_record_name"]
value = trimsuffix(aws_acm_certificate.default.domain_validation_options[0]["resource_record_value"], ".")
type = aws_acm_certificate.default.domain_validation_options[0]["resource_record_type"]
zone_id = lookup(data.cloudflare_zones.default.zones[0], "id")
depends_on = [aws_acm_certificate.default]
}
This resource will create CNAME records in cloudflare DNS to validate our ACM certificate.
resource "aws_acm_certificate_validation" "default" {
provider = aws.virginia
certificate_arn = aws_acm_certificate.default.arn
validation_record_fqdns = cloudflare_record.validation_domain.*.hostname
}
Note that certificate validation can take up to 20min, so have some patience.
The next part is an example of creating a Cloudfront distribution and exposing it’s generic domain name as a user-friendly one that we will assign in Cloudflare.
resource "aws_cloudfront_distribution" "frontend" {
enabled = true
aliases = ["${var.subdomain_name}.${var.domain_name}"]
is_ipv6_enabled = true
// cheapest: https://github.com/laurilehmijoki/s3_website/issues/150
price_class = "PriceClass_100"
default_cache_behavior {
allowed_methods = ["GET", "HEAD", "OPTIONS"]
cached_methods = ["GET", "HEAD"]
target_origin_id = var.frontend_s3_origin_id
viewer_protocol_policy = "redirect-to-https"
default_ttl = 0
max_ttl = 0
forwarded_values {
query_string = false
cookies {
forward = "none"
}
}
}
origin {
domain_name = var.frontennd_s3_origin_domain_name
origin_id = var.frontend_s3_origin_id
custom_origin_config {
http_port = 80
https_port = 443
origin_keepalive_timeout = 5
origin_protocol_policy = "http-only" // setting defined after terraform import. can try with https-only
origin_read_timeout = 30
origin_ssl_protocols = ["TLSv1", "TLSv1.1", "TLSv1.2"]
}
}
restrictions {
geo_restriction {
restriction_type = "none"
}
}
viewer_certificate {
acm_certificate_arn = aws_acm_certificate_validation.default.certificate_arn
cloudfront_default_certificate = false
minimum_protocol_version = "TLSv1.2_2019"
ssl_support_method = "sni-only"
}
}
resource "cloudflare_record" "frontend_service" {
name = "${var.subdomain_name}.${var.domain_name}"
value = aws_cloudfront_distribution.frontend.domain_name
type = "CNAME"
proxied = true
zone_id = lookup(data.cloudflare_zones.default.zones[0], "id")
}
After successfully creating those resources, your frontend will be now available over the internet on domain app-test.your-domain.com. Ta-da! We are now happy, having everything set up on HTTPS. Hurray!
Also worth noticing it that at some point AWS marked cloudflare as non-trusted authority (there are references to Mozilla’s authorities list, more can be found here), so resources that we found online to solve our issue with exposing AWS S3 bucket over SSL weren’t helpful.
Those solutions were suggesting to use Cloudflare SSL certificates and import them into AWS certificate manager and use it within AWS Cloudfront distribution.
Gitlab runners
We used Gitlab CI/CD to power up our Terraform scripts to automatically handle deploying on different environments like test, staging and production. To set up the Gitlab runners to run successful workflows, we simply ensure that AWS cli, SAM cli, Terraform and Make are available/installed during the execution. We are aware this step is highly improvable - we could have a prebuilt Docker image containing all of these or use a community maintained one, but we wanted to first focus on product use cases and later focus on improvements.
Here is an example of how to set up a Gitlab project with gitlab-ci.yaml. The most important part of this example is in before_script
section of gitlab-ci.yaml
file.
workflow:
variables:
AWS_DEFAULT_REGION: "eu-west-1"
WORKSPACE: "dev"
TF_ENVIRONMENT: "terraform"
stages:
- terraform-apply
terraform-apply:
image: google/cloud-sdk:slim
stage: terraform-apply
before_script:
- apt-get install -y unzip make jq
- curl https://releases.hashicorp.com/terraform/0.12.29/terraform_0.12.29_linux_amd64.zip --output /tmp/terraform.zip
- unzip /tmp/terraform.zip -d /tmp
- chmod +x /tmp/terraform
- mv /tmp/terraform /usr/local/bin/
- curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-2.0.30.zip" -o "awscliv2.zip"
- unzip -q awscliv2.zip
- ./aws/install
- aws sts get-caller-identity
- curl --location "https://github.com/aws/aws-sam-cli/releases/download/v1.18.1/aws-sam-cli-linux-x86_64.zip" -o "awssamcli.zip"
- unzip -q awssamcli.zip
- ./install
- sam --version
- curl --location https://github.com/terraform-linters/tflint/releases/download/v0.21.0/tflint_linux_amd64.zip -o /tmp/tflint.zip
- unzip /tmp/tflint.zip -d /tmp
- chmod +x /tmp/tflint
- mv /tmp/tflint /usr/local/bin/
script:
- echo yes | make tf-apply
Final thoughts
Overall we are pretty satisfied with the usability of AWS framework for serverless environments.
The major drawback with this solution that we want to make sure we highlight is the usage of null_resource to integrate with AWS SAM. Terraform null_resources is something everyone should try to stay away from, so we sincerely hope to see future integration of SAM in the AWS provider.
And that is it. Thank you for reading, we hope this has been useful. Feel free to reach out to us if you have questions or suggestions.
Once again, you can find all the code supporting this post here.
Top comments (0)