DEV Community

Cover image for Migrate Data into S3 using AWS Transfer Family
Robina for AWS Community Builders

Posted on • Edited on

Migrate Data into S3 using AWS Transfer Family

What is AWS Transfer Family?

AWS Transfer Family is a fully managed AWS service that you can use to transfer files into and out of Amazon Simple Storage Service (Amazon S3) storage or Amazon Elastic File System (Amazon EFS) file systems over the following protocols:

  • Secure Shell (SSH)
  • File Transfer Protocol (SFTP)
  • File Transfer Protocol (SFTP)
  • File Transfer Protocol Secure (FTPS)
  • File Transfer Protocol (FTP)

Read more: AWS Transfer Family

Benefits Of AWS Transfer Family?

  • It is a fully managed, serverless File Transfer Workflow service that makes it simple to set up, execute, automate, and monitor file processing utilizing AWS Transfer Family.
  • With your data safely stored in Amazon S3, you can utilize native AWS services for processing, analytics, reporting, auditing, and archiving.
  • There are no upfront costs, and you pay only for the use of the service.
  • You don't need to modify your applications or run any file transfer protocol infrastructure.

How it works?

AWS Transfer Family Architecture

AWS Transfer Family Architecture

AWS Transfer Family (SFTP, FTP, FTPS) – Step by Step Implementation Process

Setup the prerequisites for AWS transfer for SFTP

 - S3 bucket
 - EC2 instances (Linux and Windows)
Enter fullscreen mode Exit fullscreen mode

Create an AWS IAM role and policy

- Edit the policy to provide the S3 access

Enter fullscreen mode Exit fullscreen mode

Create the SFTP server

 - Create users
 - Create and assign public keys
Enter fullscreen mode Exit fullscreen mode

Test the file transfer from SFTP clients

- For Linux
- For Windows
Enter fullscreen mode Exit fullscreen mode

What is S3 Bucket?

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance.

Read more: Amazon Simple Storage Service

Create a S3 Bucket

- Open AWS Console Click on Services
- In the Storage Section, Choose S3
Enter fullscreen mode Exit fullscreen mode

Choose Storage

- Click Create Bucket and Create Your Bucket
- Give Bucket Name (Should be unique Name use 
small alphabets Without Space. hyphens and Periods}
- Choose your Region. (Where user wants to store their data)
 rest settings choose default.
Enter fullscreen mode Exit fullscreen mode

Create Bucket

Bucket Created

Create a VPC

- Open AWS Console Click on services
- In the Networking & Content Delivery Section, Click on VPC
Enter fullscreen mode Exit fullscreen mode

VpC

VPC configuration
VPC Created

Create Subnets

create subnets

VPc Settings

Create Two EC2 Instances (Linux & Windows)

- Open AWS Console Click on services 
- In the Compute section, click on EC2
- Click Launch instance

Enter fullscreen mode Exit fullscreen mode

launch instance
Image description
Image description

Security Group rules for Linux Instance

Security Group Rule Linux

Security Group rules for Windows Instance

Security Group Rule Windows

Create an IAM Role

- Open AWS Console Click on services 
- In the Security, Identity & Compliance section, click on IAM 
Enter fullscreen mode Exit fullscreen mode

Services

Create IAM roles and policies from the IAM console (Choose AWS service as Transfer)

The policy to access the S3 bucket

S3 Bucket Policy
Image description

Policy to access CloudWatch Logs

Image description
Image description

Create two roles and attached the policies with each role (Choose AWS service as Transfer)

Image description

Create the SFTP server, Users, Public Keys and Assign the keys

- Open AWS Console Click on services 
- In the Migration & Transfer section, click on AWS Transfer Family
Enter fullscreen mode Exit fullscreen mode

Image description

Click on create Server

Image description

Choose Protocols

Image description

Choose an Identity Provider

Image description

Choose an Endpoint

Image description

Choose a Domain

Image description

Image description

Image description

Image description

Add User to the Server

Image description

Image description
Image description

Generate the public key from the Linux machine?

Log in to your EC2 machine connect via Putty

Image description

Image description

Generate the key with the below command

ssh-keygen -P "" -f key 
Enter fullscreen mode Exit fullscreen mode

Image description

Image description

Image description

Image description

Image description

Transfer the files using WinSCP for Linux

Image description

Image description

Test the file transfer from the Linux SFTP client

Image description

Connect to the server

- sftp -i key rm-transfer-family-user@
s-c0ca408efe3d4c3e8.server.transfer.ap-southeast-1.amazonaws.com

- put AWS-Transfer-Family-Linux.txt rm-transfer-family-2022 
AWS-Transfer-Family-Linux.txt 
to/rm-transfer-family-2022/rm-transfer-family-user/AWS-Transfer-Family-Linux.txt 
Enter fullscreen mode Exit fullscreen mode

Successfully transfer File from Server to Bucket

Image description

Now do the same using the Windows Machine

Image description

Image description

Image description

Image description

Generate Public key for Windows user

Image description
Image description

Two Public keys are added

Image description

Now transfer the files using WinSCP for Windows

Image description

Connection established using WinSCP for Windows

Image description

Test the file transfer from SFTP client for Windows

Image description

Successfully Tested the file transfer from SFTP client and added to the Bucket

Image description

Top comments (2)

Collapse
 
hseritt profile image
Harlin Seritt

Bookmarking! Thanks for posting!

Collapse
 
robinamirbahar profile image
Robina

Thank you so much