A few days ago, I encountered an issue with AWS RDS services, specifically with SQL server databases that could not be backed up using the AWS Backup service. Currently, there are limitations for requesting RDS to backup databases. However, it is still possible to create backup requests for our database using the RDS API.
Therefore, I want to remove the activity of engineers having to create backups manually through API or web console, and instead develop Serverless services to automate this process. This approach can also be extended to solve other issues in the future.
Let’s go from this guideline
Preparations
Before creating any Lambda function, we need to read a document that explains how to develop them. You can find it here. Then, you should have the following.
-
Golang
version 1.16 or higher -
Pre-commit
that use to ensure you are pushing a verified source code to the SCM -
Taskfile
the optional, that use to reduce routine step in your development.
Overview solution is…
We focus on events generated by RDS when a snapshot is created. These events are triggered by AWS EventBridge, which provisions a function in AWS Lambda to execute the necessary actions.
First thing
You can provide a folder structure for develop a Lambda function in your workspace.
-
Create new Github repository and clone it into your local machine.
git clone <your_repo_url>
-
Go to your folder and init a new Golang project. it will create a new
go.mod
file.
go mod init <your_repo_url> #Example go mod init github.com/StartloJ/lambda-rds-utils
-
(Optional) Create a pre-commit policy file to help you improve development quality.
#.pre-commit-config.yaml repos: - repo: https://github.com/dnephin/pre-commit-golang rev: master hooks: - id: go-fmt - id: go-vet - id: go-imports - id: go-mod-tidy
Create a first function
Now, we are ready to develop the Lambda function to handle the event. You can follow the steps below to create a real use case function that replicates the RDS snapshot across regions.
Remember the event structure from AWS EventBridge
You can see more event on official docs here.
{
"version": "0",
"id": "844e2571-85d4-695f-b930-0153b71dcb42",
"detail-type": "RDS DB Snapshot Event",
"source": "aws.rds",
"account": "123456789012",
"time": "2018-10-06T12:26:13Z",
"region": "us-east-1",
"resources": ["arn:aws:rds:us-east-1:123456789012:snapshot:rds:snapshot-replica-2018-10-06-12-24"],
"detail": {
"EventCategories": ["creation"],
"SourceType": "SNAPSHOT",
"SourceArn": "arn:aws:rds:us-east-1:123456789012:snapshot:rds:snapshot-replica-2018-10-06-12-24",
"Date": "2018-10-06T12:26:13.882Z",
"SourceIdentifier": "rds:snapshot-replica-2018-10-06-12-24",
"Message": "Automated snapshot created",
"EventID": "RDS-EVENT-0091"
}
}
Define a configuration for functions
I will define parameters to reuse this function below.
-
OPT_SRC_REGION
is a source region of RDS snapshots for copy. -
OPT_TARGET_REGION
is a target region that we need to store snapshots. -
OPT_DB_NAME
is a identity of RDS DB name that we need to copy snapshots. -
OPT_OPTION_GROUP_NAME
is a target option group to attachment to replicate snapshot in the target region. -
OPT_KMS_KEY_ID
is a KMS key that use to encrypt snapshot on target region.
Define main func
package main
import (
"github.com/aws/aws-lambda-go/lambda"
"github.com/sirupsen/logrus"
)
func main() {
// Make the hangler available for remote procedure call by Lambda
logrus.Info("we starting handle lambda...")
lambda.Start(HandleEvents)
}
Create a function handle by lambda
// Define func to receive a variable from `BridgeEvent`
// and it should return `error` if any mistake.
func HandleEvents(events BridgeEvent) error {
return nil
}
Define a new AWS session
// define AWS session for source and target region to copy snapshot
func HandleEvents(event BridgeEvent) error {
// start copy code
des_sess := session.Must(session.NewSessionWithOptions(session.Options{
Config: aws.Config{
Region: aws.String(viper.GetString("target_region")),
},
}))
src_sess := session.Must(session.NewSessionWithOptions(session.Options{
Config: aws.Config{
Region: aws.String(viper.GetString("src_region")),
},
}))
// end copy code
return nil
}
Create func to get all existing snapshot
// This func input session of RDS service. Then, it get all list of database
// snapshot that already in your resources in List of Snapshot name.
func ListAllSnapshot(svc *rds.RDS) ([]string, error) {
db := viper.GetString("DB_NAME")
// This line will return a list of snapshots included snapshot name.
out, err := svc.DescribeDBSnapshots(&rds.DescribeDBSnapshotsInput{
DBInstanceIdentifier: &db,
})
if err != nil {
panic(err)
}
var snapShotName []string
// this loop is extract a snapshot name only.
for _, b := range out.DBSnapshots {
logrus.Infof("We have %s", ConvertToString(*b.DBSnapshotArn))
snapShotName = append(snapShotName, ConvertToString(*b.DBSnapshotArn))
}
return snapShotName, nil
}
Get a list of RDS snapshot name from source and target region
func HandleEvents(event BridgeEvent) error {
...
// start copy code
targetSvc := rds.New(des_sess)
rep_dbSnapshots, err := ListAllSnapshot(targetSvc)
if err != nil {
return fmt.Errorf("we can't get any Snapshot name")
}
srcSvc := rds.New(src_sess)
src_dbSnapshots, err := ListAllSnapshot(srcSvc)
if err != nil {
return fmt.Errorf("we can't get any Snapshot name")
}
// End copy code
...
}
Create a func to remove duplicate snapshot as source and target
// this func use to remove duplicate name source if found in target
// you shouldn't switch position input to this func.
// `t` is mean a target that use double check to `s`
// it will remove a value in `s` if found it in `t`
func GetUniqueSnapShots(t, s []string) ([]string, error) {
//Create a map to keep track a unique strings
uniqueMap := make(map[string]bool)
//Iterate over `s` and add each string to map
for _, str := range s {
uniqueMap[str] = true
}
//Iterate over `t` and remove any string that are already in uniqueMap
for _, str2 := range t {
delete(uniqueMap, str2)
}
//Convert the unique string from Map to slice string
result := make([]string, 0, len(uniqueMap))
for str := range uniqueMap {
result = append(result, str)
}
if len(result) == 0 {
return nil, fmt.Errorf("not any Snapshot unique between source and target region")
}
return result, nil
}
Define a unique snapshot
func HandleEvents(event BridgeEvent) error {
...
// start copy code
dbSnapShots2Copy, err := GetUniqueSnapShots(rep_dbSnapshots, src_dbSnapshots)
if err != nil {
logrus.Warnf("it doesn't any task copy snapshot to %s", viper.GetString("target_region"))
return nil
}
// End copy code
...
}
Create a func copy RDS snapshot across region
// Request to AWS RDS API for create event copy a specific snapshot
// into across region and encrypt it by KMS key multi-region
func CopySnapshotToTarget(svc *rds.RDS, snap string) (string, error) {
targetSnapArn := strings.Split(snap, ":")
targetSnapName := targetSnapArn[len(targetSnapArn)-1]
// Copy the snapshot to the target region
copySnapshotInput := &rds.CopyDBSnapshotInput{
OptionGroupName: aws.String(viper.GetString("option_group_name")),
KmsKeyId: aws.String(viper.GetString("kms_key_id")),
CopyTags: aws.Bool(true),
SourceDBSnapshotIdentifier: aws.String(snap),
TargetDBSnapshotIdentifier: aws.String(targetSnapName),
SourceRegion: aws.String(viper.GetString("src_region")),
}
_, err := svc.CopyDBSnapshot(copySnapshotInput)
if err != nil {
logrus.Errorf("Copy request %s is failed", snap)
return "", err
}
logrus.Infof("Copy %s is created", snap)
return fmt.Sprintf("Copy %s is created", snap), nil
}
Define loop to execute all snapshot value
func HandleEvents(event BridgeEvent) error {
...
// start copy code
for s := range dbSnapShots2Copy {
logrus.Infof("trying to copy DBSnapshot to %s...", viper.GetString("target_region"))
_, err := CopySnapshotToTarget(targetSvc, dbSnapShots2Copy[s])
if err != nil {
logrus.Error(err.Error())
}
}
// End copy code
...
}
If all workflow is complete, it will normal end function from AWS Lambda controller.
Finally steps
-
If you finished to develop function, you will build and pack it with ZIP format.
export BINARY_NAME="lambda-rds-util" #you can change this name. go build -o $BINARY_NAME main.go zip lambda-$BINARY_NAME.zip $BINARY_NAME
-
You can upload your compress file(also ZIP file) to S3.
#you can push it with below command export S3_NAME="<your_s3_bucket_name>" aws s3api put-object --bucket $S3_NAME --key lambda-$BINARY_NAME.zip --body ./lambda-$BINARY_NAME.zip
In the web console of AWS Lambda, you can define a new resource and select source from S3.
In the web console, you can test function in AWS Lambda to ensure your code is work!.
Conclusion
You can automate certain activities on AWS or other cloud providers using the Serverless
service, such as AWS Lambda. It can enhance your development experience and enable you to explore new possibilities. You can even extend this idea to develop a new workflow in your work.
I hope this blog has inspired you to consider using Serverless
in your development journey.
Top comments (0)