DEV Community

Binh Nguyen for AWS Community Builders

Posted on

Use S3 Batch Replication to replicate objects to another account and encrypt with AWS KMS

Our scenario?

We have an existing S3 bucket in an AWS account (A) and we have to somehow move the existing data onto AWS account (B) because of internal requirements or compliance decisions. Before the data landing onto the destination S3 bucket, it must be encrypted with an AWS KMS key which belongs to the destination account.

For this post, I am going to use S3 Batch Replication and show steps by steps from AWS console.

When to use S3 Batch Replication?

As per AWS official documentation, here are the main points:

  1. Existing objects
  2. Objects that failed to replicate previously
  3. Objects that were already replicated
  4. Replicas that were created from a replication rule

The 1st point is related to our scenario.

Prerequisites

  1. Enable bucket versioning on both source and destination S3 buckets.
  2. The service, which is Amazon S3, must have permissions to perform the tasks.
  3. Destination account (B) must allow Source acccount (A) to use its AWS KMS encryption key.

Preparation

Enable Versioning

To be applied on both Source and Destination AWS accounts

  1. Go to your S3 bucket and locate Properties tab from AWS console.
  2. From Bucket Versioning, ensure the value is Enabled.

Source Account A

Create a new S3 bucket

Let's quickly create a new bucket for storing replication reports or you can use any existing ones.

aws s3api create-bucket \
    --bucket <S3_BUCKET_REPORT_A> \
    --region ap-southeast-1 \
    --create-bucket-configuration LocationConstraint=ap-southeast-1
Enter fullscreen mode Exit fullscreen mode

IAM role for S3 replication

Let's quickly create an IAM role for our later usage of S3 replication rule.

aws iam create-role \
    --role-name <IAM_ROLE_REPLICATION_A> \
    --assume-role-policy-document file://trust.json
aws iam put-role-policy \
    --role-name <IAM_ROLE_REPLICATION_A> \
    --policy-name replication-policy \
    --policy-document file://policy.json
Enter fullscreen mode Exit fullscreen mode

Here is the IAM policy (policy.json) to be used.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:InitiateReplication",
                "s3:GetReplicationConfiguration",
                "s3:PutInventoryConfiguration",
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetObjectTagging",
                "s3:GetObjectVersionForReplication",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersionTagging"
            ],
            "Resource": [
                "arn:aws:s3:::<S3_BUCKET_A>",
                "arn:aws:s3:::<S3_BUCKET_A>/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:ReplicateObject",
                "s3:ReplicateTags",
                "s3:ReplicateDelete",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:PutObjectVersionAcl",
                "s3:PutObjectTagging",
                "s3:PutObjectVersionTagging",
                "s3:GetObjectVersionForReplication",
                "s3:ObjectOwnerOverrideToBucketOwner"
            ],
            "Resource": [
                "arn:aws:s3:::<S3_BUCKET_B>",
                "arn:aws:s3:::<S3_BUCKET_B>/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::<S3_BUCKET_REPORT_A>/*"
        },
        {
            "Effect": "Allow",
            "Action": "kms:*",
            "Resource": "*"
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

Here is the trust relationship (trust.json) to be used.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "batchoperations.s3.amazonaws.com",
                    "s3.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

Destination Account B

Update S3 bucket policy

Ensure the S3 bucket policy to be updated.

{
    "Version": "2012-10-17",
    "Id": "PolicyForDestinationBucket",
    "Statement": [
        {
            "Sid": "S3PolicyStmt-DO-NOT-MODIFY-1234567890123",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<AWS_ID_A>:root",
                    "arn:aws:iam::<AWS_ID_A>:role/service-role/<IAM_ROLE_REPLICATION_A>"
                ]
            },
            "Action": [
                "s3:GetBucketVersioning",
                "s3:GetObjectAcl",
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:PutObjectAcl",
                "s3:PutObjectVersionAcl",
                "s3:PutBucketVersioning",
                "s3:ObjectOwnerOverrideToBucketOwner",
                "s3:Put*"
            ],
            "Resource": [
                "arn:aws:s3:::<S3_BUCKET_B>",
                "arn:aws:s3:::<S3_BUCKET_B>/*"
            ]
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

Update KMS key policy

Ensure the KMS key policy to be updated.

{
    "Version": "2012-10-17",
    "Id": "my-key-policy",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<AWS_ID_B>:root"
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "Allow an external account to use this KMS key",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<AWS_ID_A>:root",
                    "arn:aws:iam::<AWS_ID_A>:role/service-role/<IAM_ROLE_REPLICATION_A>"
                ]
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

Create a replication rule

From the source S3 bucket, create an S3 replication rule.
create-s3-replication-rule

Choose the rule scope to all objects and specify destination bucket with AWS account ID.
specify-destination-bucket

Choose the option to change the bucket ownership and specify the IAM role that we have created earlier.
specify-replication-role

Enter the ARN of the destination AWS KMS encryption key to be used.
specify-encryption-key

For destination classes and replication options, keep them as optional if you don't have any related requirements. Then, choose option to replicate existing objects. This will direct you to a new page which is the next step below.

Create Batch Operations job

For Job run options, we choose Automatically run the job when it's ready or it is up to you.

In order to gain visibility of replication process, we enable reports generation option and store them onto S3 bucket.
generate-replication-reports

For Permissions, you can reuse the earlier IAM Role.

Once you clicked the Save button, it will create a new Job ID and based on my job run option earlier, it automatically runs so we will wait for the result.
new-batch-operations-job

If it is failed, you can check the job status or report for further details. In my case, my replication IAM role does not have enough permissions to generate reports to the specified bucket.

job-status-failed

If it is progresses, you will observe the % of objects that have been replicated or objects have been failed to replicate.

References

  1. https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html
  2. https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-batch-replication-batch.html
  3. https://aws.amazon.com/blogs/aws/new-replicate-existing-objects-with-amazon-s3-batch-replication/

Top comments (1)

Collapse
 
erezhazan1 profile image
Erez Hazan

Pretty sure you could be more specific then "s3:Put*"