DEV Community

Cover image for AWS Lambda replicate RDS snapshot
Watcharin(start)
Watcharin(start)

Posted on

AWS Lambda replicate RDS snapshot

A few days ago, I encountered an issue with AWS RDS services, specifically with SQL server databases that could not be backed up using the AWS Backup service. Currently, there are limitations for requesting RDS to backup databases. However, it is still possible to create backup requests for our database using the RDS API.

Therefore, I want to remove the activity of engineers having to create backups manually through API or web console, and instead develop Serverless services to automate this process. This approach can also be extended to solve other issues in the future.

Let’s go from this guideline

Preparations

Before creating any Lambda function, we need to read a document that explains how to develop them. You can find it here. Then, you should have the following.

  1. Golang version 1.16 or higher
  2. Pre-commit that use to ensure you are pushing a verified source code to the SCM
  3. Taskfile the optional, that use to reduce routine step in your development.

Overview solution is…

We focus on events generated by RDS when a snapshot is created. These events are triggered by AWS EventBridge, which provisions a function in AWS Lambda to execute the necessary actions.

Lambda with EventBridge

First thing

You can provide a folder structure for develop a Lambda function in your workspace.

  1. Create new Github repository and clone it into your local machine.

    git clone <your_repo_url> 
    
  2. Go to your folder and init a new Golang project. it will create a new go.mod file.

    go mod init <your_repo_url>
    #Example
    go mod init github.com/StartloJ/lambda-rds-utils
    
  3. (Optional) Create a pre-commit policy file to help you improve development quality.

    #.pre-commit-config.yaml
    repos:
    - repo: https://github.com/dnephin/pre-commit-golang
      rev: master
      hooks:
      - id: go-fmt
      - id: go-vet
      - id: go-imports
      - id: go-mod-tidy
    

Create a first function

Now, we are ready to develop the Lambda function to handle the event. You can follow the steps below to create a real use case function that replicates the RDS snapshot across regions.

Remember the event structure from AWS EventBridge

You can see more event on official docs here.

{
    "version": "0",
    "id": "844e2571-85d4-695f-b930-0153b71dcb42",
    "detail-type": "RDS DB Snapshot Event",
    "source": "aws.rds",
    "account": "123456789012",
    "time": "2018-10-06T12:26:13Z",
    "region": "us-east-1",
    "resources": ["arn:aws:rds:us-east-1:123456789012:snapshot:rds:snapshot-replica-2018-10-06-12-24"],
    "detail": {
      "EventCategories": ["creation"],
      "SourceType": "SNAPSHOT",
      "SourceArn": "arn:aws:rds:us-east-1:123456789012:snapshot:rds:snapshot-replica-2018-10-06-12-24",
      "Date": "2018-10-06T12:26:13.882Z",
      "SourceIdentifier": "rds:snapshot-replica-2018-10-06-12-24",
      "Message": "Automated snapshot created",
      "EventID": "RDS-EVENT-0091"
    }
}
Enter fullscreen mode Exit fullscreen mode

Define a configuration for functions

I will define parameters to reuse this function below.

  • OPT_SRC_REGION is a source region of RDS snapshots for copy.
  • OPT_TARGET_REGION is a target region that we need to store snapshots.
  • OPT_DB_NAME is a identity of RDS DB name that we need to copy snapshots.
  • OPT_OPTION_GROUP_NAME is a target option group to attachment to replicate snapshot in the target region.
  • OPT_KMS_KEY_ID is a KMS key that use to encrypt snapshot on target region.

Define main func

package main

import (
    "github.com/aws/aws-lambda-go/lambda"
    "github.com/sirupsen/logrus"
)

func main() {
    // Make the hangler available for remote procedure call by Lambda
    logrus.Info("we starting handle lambda...")
    lambda.Start(HandleEvents)
}
Enter fullscreen mode Exit fullscreen mode

Create a function handle by lambda

// Define func to receive a variable from `BridgeEvent`
// and it should return `error` if any mistake.
func HandleEvents(events BridgeEvent) error {
    return nil
}
Enter fullscreen mode Exit fullscreen mode

Define a new AWS session

// define AWS session for source and target region to copy snapshot
func HandleEvents(event BridgeEvent) error {
    // start copy code
    des_sess := session.Must(session.NewSessionWithOptions(session.Options{
        Config: aws.Config{
            Region: aws.String(viper.GetString("target_region")),
        },
    }))
    src_sess := session.Must(session.NewSessionWithOptions(session.Options{
        Config: aws.Config{
            Region: aws.String(viper.GetString("src_region")),
        },
    }))
    // end copy code
    return nil
}
Enter fullscreen mode Exit fullscreen mode

Create func to get all existing snapshot

// This func input session of RDS service. Then, it get all list of database
// snapshot that already in your resources in List of Snapshot name.
func ListAllSnapshot(svc *rds.RDS) ([]string, error) {

    db := viper.GetString("DB_NAME")

    // This line will return a list of snapshots included snapshot name.
    out, err := svc.DescribeDBSnapshots(&rds.DescribeDBSnapshotsInput{
        DBInstanceIdentifier: &db,
    })
    if err != nil {
        panic(err)
    }

    var snapShotName []string

    // this loop is extract a snapshot name only.
    for _, b := range out.DBSnapshots {
        logrus.Infof("We have %s", ConvertToString(*b.DBSnapshotArn))
        snapShotName = append(snapShotName, ConvertToString(*b.DBSnapshotArn))
    }

    return snapShotName, nil
}
Enter fullscreen mode Exit fullscreen mode

Get a list of RDS snapshot name from source and target region

func HandleEvents(event BridgeEvent) error {
    ...
    // start copy code
    targetSvc := rds.New(des_sess)
    rep_dbSnapshots, err := ListAllSnapshot(targetSvc)
    if err != nil {
        return fmt.Errorf("we can't get any Snapshot name")
    }

    srcSvc := rds.New(src_sess)
    src_dbSnapshots, err := ListAllSnapshot(srcSvc)
    if err != nil {
        return fmt.Errorf("we can't get any Snapshot name")
    }
    // End copy code
    ...
}
Enter fullscreen mode Exit fullscreen mode

Create a func to remove duplicate snapshot as source and target

// this func use to remove duplicate name source if found in target
// you shouldn't switch position input to this func.
// `t` is mean a target that use double check to `s`
// it will remove a value in `s` if found it in `t`
func GetUniqueSnapShots(t, s []string) ([]string, error) {
    //Create a map to keep track a unique strings
    uniqueMap := make(map[string]bool)

    //Iterate over `s` and add each string to map
    for _, str := range s {
        uniqueMap[str] = true
    }

    //Iterate over `t` and remove any string that are already in uniqueMap
    for _, str2 := range t {
        delete(uniqueMap, str2)
    }

    //Convert the unique string from Map to slice string
    result := make([]string, 0, len(uniqueMap))
    for str := range uniqueMap {
        result = append(result, str)
    }

    if len(result) == 0 {
        return nil, fmt.Errorf("not any Snapshot unique between source and target region")
    }

    return result, nil
}
Enter fullscreen mode Exit fullscreen mode

Define a unique snapshot

func HandleEvents(event BridgeEvent) error {
    ...
    // start copy code
    dbSnapShots2Copy, err := GetUniqueSnapShots(rep_dbSnapshots, src_dbSnapshots)
    if err != nil {
        logrus.Warnf("it doesn't any task copy snapshot to %s", viper.GetString("target_region"))
        return nil
    }
    // End copy code
    ...
}
Enter fullscreen mode Exit fullscreen mode

Create a func copy RDS snapshot across region

// Request to AWS RDS API for create event copy a specific snapshot
// into across region and encrypt it by KMS key multi-region
func CopySnapshotToTarget(svc *rds.RDS, snap string) (string, error) {
    targetSnapArn := strings.Split(snap, ":")
    targetSnapName := targetSnapArn[len(targetSnapArn)-1]

    // Copy the snapshot to the target region
    copySnapshotInput := &rds.CopyDBSnapshotInput{
        OptionGroupName:            aws.String(viper.GetString("option_group_name")),
        KmsKeyId:                   aws.String(viper.GetString("kms_key_id")),
        CopyTags:                   aws.Bool(true),
        SourceDBSnapshotIdentifier: aws.String(snap),
        TargetDBSnapshotIdentifier: aws.String(targetSnapName),
        SourceRegion:               aws.String(viper.GetString("src_region")),
    }

    _, err := svc.CopyDBSnapshot(copySnapshotInput)
    if err != nil {
        logrus.Errorf("Copy request %s is failed", snap)
        return "", err
    }

    logrus.Infof("Copy %s is created", snap)
    return fmt.Sprintf("Copy %s is created", snap), nil
}
Enter fullscreen mode Exit fullscreen mode

Define loop to execute all snapshot value

func HandleEvents(event BridgeEvent) error {
    ...
    // start copy code
    for s := range dbSnapShots2Copy {
        logrus.Infof("trying to copy DBSnapshot to %s...", viper.GetString("target_region"))
        _, err := CopySnapshotToTarget(targetSvc, dbSnapShots2Copy[s])
        if err != nil {
            logrus.Error(err.Error())
        }
    }
    // End copy code
    ...
}
Enter fullscreen mode Exit fullscreen mode

If all workflow is complete, it will normal end function from AWS Lambda controller.

Finally steps

  1. If you finished to develop function, you will build and pack it with ZIP format.

    export BINARY_NAME="lambda-rds-util" #you can change this name.
    go build -o $BINARY_NAME main.go
    zip lambda-$BINARY_NAME.zip $BINARY_NAME
    
  2. You can upload your compress file(also ZIP file) to S3.

    #you can push it with below command
    export S3_NAME="<your_s3_bucket_name>"
    aws s3api put-object --bucket $S3_NAME --key lambda-$BINARY_NAME.zip --body ./lambda-$BINARY_NAME.zip
    
  3. In the web console of AWS Lambda, you can define a new resource and select source from S3.

  4. In the web console, you can test function in AWS Lambda to ensure your code is work!.

Conclusion

You can automate certain activities on AWS or other cloud providers using the Serverless service, such as AWS Lambda. It can enhance your development experience and enable you to explore new possibilities. You can even extend this idea to develop a new workflow in your work.

I hope this blog has inspired you to consider using Serverless in your development journey.

Top comments (0)