Vikram Aruchamy for AWS Community Builders

Posted on Jun 19, 2021 • Originally published at stackvidhya.com

How to write a file or data to an S3 object using boto3

#python #aws #machinelearning

S3 is an object storage service proved by AWS. You may need to upload data or file to S3 when working with AWS Sagemaker notebook or a normal jupyter notebook in Python.

You can Write a file or data into S3 Using Boto3 using,

Object.put()
Upload_File()
Client.putObject()

If You're in Hurry...

You can use the below code snippet to write a file to S3.

import boto3

#Creating Session With Boto3.
session = boto3.Session(
aws_access_key_id='<your_access_key_id>',
aws_secret_access_key='<your_secret_access_key>'
)

#Creating S3 Resource From the Session.
s3 = session.resource('s3')

txt_data = b'This is the content of the file uploaded from python boto3 asdfasdf'

object = s3.Object('<bucket_name>', 'file_uploaded_by_boto3.txt')

result = object.put(Body=txt_data)

This is how you can upload file to S3 from Jupyter notebook and Python using Boto3.

If You Want to Understand Details, Read on…

In this tutorial, you'll learn how to write a file or a data to S3 using Boto3.

Prerequisites

Generate the security credentials by clicking Your Profile Name -> My security Credentials -> Access keys (access key ID and secret access key) option. This is necessary to create session to your S3 bucket.
Understand the difference between boto3 resource and boto3 client. Object.put() and the upload_file() methods are from boto3 resource where as put_object() is from boto3 client.

Using Object.put()

You can use the Object.put() method available in the S3 object.

It allows two ways to write into the S3.

Writing a text content to an S3 object
Writing contents from the local file to the S3 object

Write Text Data To S3 Object

In this section, you'll learn how to write a normal text data to the s3 object.

Follow the below steps to write a text data to an S3 Object.

Create a Boto3 session using the security credentials
With the session, create a resource object for the S3 service
Create an S3 object using the object method. It accepts two parameters. BucketName and the File_Key. File_Key is the name you want to give it for the S3 object. If you would like to create sub-folders inside the bucket, you can prefix the locations in this File_key variable. For example, /subfolder/file_name.txt
1. Create an text object which holds the text to be updated to the S3 object
2. Use the put() action available in the S3 object and the set the body as the text data. E.g. Body=txt_data.
3. put() actions returns a JSON response metadata. This metadata contains the HttpStatusCode which shows if the file upload is successful or not. If the status code is 200, then the file upload is successful. Else, it is not.

Note: Using this method will replace the existing S3 object in the same name. Hence ensure you're using a unique name to this object.

Snippet

import boto3

#Creating Session With Boto3.
session = boto3.Session(
aws_access_key_id='<your_access_key_id>',
aws_secret_access_key='<your_secret_access_key>'
)

#Creating S3 Resource From the Session.
s3 = session.resource('s3')

object = s3.Object('<bucket_name>', 'file_uploaded_by_boto3.txt')

txt_data = b'This is the content of the file uploaded from python boto3'

result = object.put(Body=txt_data)

res = result.get('ResponseMetadata')

if res.get('HTTPStatusCode') == 200:
    print('File Uploaded Successfully')
else:
    print('File Not Uploaded')

Output

    File Uploaded Successfully

This is how you can update the text data to an S3 object using Boto3.

Reading a File from Local and Updating it to S3

In this section, you'll learn how to read a file from local system and update it to an S3 object.

It is similar to the steps explained in the previous step except for one step.

You just need to open a file in the binary mode and send its content to the put() method using the below snippet.

Use only forward slash for the filepath. Backslash doesn't work.

Note: Using this method will replace the existing S3 object in the same name. Hence ensure you're using a unique name to this object.

result = object.put(Body=open('E:/temp/testfile.txt', 'rb'))

Snippet

import boto3

#Creating Session With Boto3.
session = boto3.Session(
aws_access_key_id='<your_access_key_id>',
aws_secret_access_key='<your_secret_access_key>'
)

#Creating S3 Resource From the Session.
s3 = session.resource('s3')

object = s3.Object('<bucket_name>', 'file3_uploaded_by_boto3.txt')

result = object.put(Body=open('E:/temp/testfile.txt', 'rb'))

res = result.get('ResponseMetadata')

if res.get('HTTPStatusCode') == 200:
    print('File Uploaded Successfully')
else:
    print('File Not Uploaded')

You can check if file is successfully uploaded or not using the HTTPStatusCode available in the responsemetadata.

Output

    File Uploaded Successfully

This is how you can write the data from the text file to an S3 object using Boto3.

Using Upload_File()

In this section, you'll learn how to use the upload_file() method to upload a file to an S3 bucket. It is a boto3 resource.

Follow the below steps to use the upload_file() action to upload file to S3 bucket.

Create a boto3 session
Create an object for S3 object
Access the bucket in the S3 resource using the s3.Bucket() method and invoke the upload_file() method to upload the files
upload_file() method accepts two parameters.

- `File_Path` - Path of the file from the local system that needs to be uploaded. Use only forward slash when you mention the path name
- `Object_name` - Name for the object that will be created by uploading this file

Unlike the other methods, the upload_file() method doesn't return an meta object to check the result. You can use the other methods to check if an object is available in the bucket.

Note: Using this method will replace the existing S3 object in the same name. Hence ensure you're using a unique name to this object.

Snippet

import boto3

#Creating Session With Boto3.
session = boto3.Session(
aws_access_key_id='<your_access_key_id>',
aws_secret_access_key='<your_secret_access_key>'
)

#Creating S3 Resource From the Session.
s3 = session.resource('s3')

result = s3.Bucket('<bucket_name>').upload_file('E:/temp/testfile.txt','file2_uploaded_by_boto3.txt')

print(result)

File is updated successfully. But you'll only see the status as None.

Output

    None

This is how you can use the upload_file() method to upload file to the S3 buckets.

Using Client.putObject()

In this section, you'll learn how to use the put_object method from the boto3 client.

Follow the below steps to use the client.put_object() method to upload a file as an S3 object.

Create a boto3 session using your AWS security credentials
Create a resource object for S3
Get the client from the S3 resource using s3.meta.client
Invoke the put_object() method from the client. It accepts two parameters.

 - `body`  - To pass the textual content for the S3 object. You can pass the text directly. Or you can use the file object by opening the file using `open('E:/temp/testfile.txt', 'rb')`
 - `Name` - Name for the new object that will be created.

put_object() also returns a ResponseMetaData which will let you know the status code to denote if the upload is successful or not.

Snippet

import boto3

#Creating Session With Boto3.
session = boto3.Session(
aws_access_key_id='<your_access_key_id>',
aws_secret_access_key='<your_secret_access_key>'
)

#Creating S3 Resource From the Session.
s3 = session.resource('s3')

result = s3.meta.client.put_object(Body='Text Contents', Bucket='<bucket_name>', Key='filename_by_client_put_object.txt')

res = result.get('ResponseMetadata')

if res.get('HTTPStatusCode') == 200:
    print('File Uploaded Succesfully')
else:
    print('File Not Uploaded')

A new S3 object will be created and the contents of the file will be uploaded.

Output

    File Uploaded Succesfully

This is how you can use the put_object() method available in boto3 S3 client to upload files to the S3 bucket.

Conclusion

To summarize, you've learnt what is boto3 client and boto3 resource in the prerequisites and also learnt the different methods available in the boto3 resource and boto3 client to upload file or data to the S3 buckets.