During my learning, I discovered five S3 functionalities that are only available through the AWS CLI:
Table of Contents
- Configuring S3 MFA delete
- S3 Pre-signed URLs
- Upload files to S3 Glacier Vault
- S3 Multi Part Upload
- S3 Access point through VPC
1. Configuring S3 MFA delete
When working with S3 Versioning in Amazon S3 buckets, you can optionally add another layer of security by configuring a bucket to enable MFA (multi-factor authentication) delete. When you do this, the bucket owner must include two forms of authentication in any request to delete a version or change the versioning state of the bucket.
MFA delete requires additional authentication for either of the following operations:
👉 Changing the versioning state of your bucket
👉 Permanently deleting an object version
MFA delete requires two forms of authentication together:
Your security credentials
The concatenation of a valid serial number, a space, and the six-digit code displayed on an approved authentication device
In order to enable MFA, we need to follow the steps below.
Configure AWS Client
Create an AWS access key and then execute aws configure to set up your credentials.
📓 This is not best practice for security purposes but we will use this for the current exercise only.
Command to enable MFA
📓 213849 is the authentication token
[ec2user@some-ip ~]$ aws s3api put-bucket-versioning --profile default --bucket testbucketmfa --versioning-configuration Status=Enabled,MFADelete=Enabled --mfa "arn:aws:iam::XXXXXXXXXXXX:mfa/account-mfa-device 213849"
Command to disable MFA
[ec2user@some-ip ~]$ aws s3api put-bucket-versioning --profile default --bucket testbucketmfa --versioning-configuration Status=Enabled,MFADelete=Disabled --mfa "arn:aws:iam::XXXXXXXXXXXX:mfa/account-mfa-device 987543"
2. S3 Pre-signed URLs
All objects in S3 are private by default. Only the object owner has permission to access these objects. However, the object owner can optionally share objects with others by creating a presigned URL, using their own security credentials, to grant time-limited permission to download the objects.
The commands below are using AWS CLI. We can also generate pre-signed URLs using AWS SDK.
aws s3 presign s3://testbucket202119/smiley.jpg --region ap-southeast-2
The above command will generate a URL, for example:
https://testbucket202119.s3.amazonaws.com/smiley.jpg?AWSAccessKeyId=AKIAYYDMCK6YRXWASPX2&Expires=1639901714&Signature=VGcqq8ilnCtkd8OTFJP4aMidqI4%3D
3. Upload files to S3 Glacier Vault
S3 Glacier Vault is a container for storing archives.
Command to create a vault
aws glacier create-vault --vault-name testvault --account-id [AccountId]
Command to upload an archive to a vault
aws glacier upload-archive --account-id [AccountId] --vault-name testvault --body HappyFace.jpg
To delete a non-empty vault
👉 If deleting a non-empty vault you must first delete all existing archives before deleting the vault. The commands below are using AWS CLI. We can also do this using Rest API or AWS SDK.
Use the initiate-job command to start an inventory retrieval job.
aws glacier initiate-job --vault-name testvault --account-id [AccountId] --job-parameters '{"Type": "inventory-retrieval"}'
{
...
"jobId": "Nm1J8J2qnjgiT68k09gCbSSm2wG_IsBFwlxYSQo2JsAbDgwCV0nPy-Sxcc5BeUnQF2y13HWF3zmWvj6wPK5sIlzZOh45"
}
Use the describe-job command to check the status of the previous retrieval job or enable notifications on the vault to get notified.
aws glacier describe-job --vault-name testvault --account-id [AccountId] --job-id "jobId from the above output"
{
"CompletionDate": "2021-12-20T03:53:59.230Z",
"VaultARN": "arn:aws:glacier:ap-southeast-2:[AccountId]:vaults/testvault",
"InventoryRetrievalParameters": {
"Format": "JSON"
},
"Completed": true,
"InventorySizeInBytes": 445,
"JobId": "Nm1J8J2qnjgiT68k09gCbSSm2wG_IsBFwlxYSQo2JsAbDgwCV0nPy-Sxcc5BeUnQF2y13HWF3zmWvj6wPK5sIlzZOh45",
"Action": "InventoryRetrieval",
"CreationDate": "2021-12-20T00:01:20.715Z",
"StatusMessage": "Succeeded",
"StatusCode": "Succeeded"
}
It took me approximately 3 hours to retrieve the file.
When it's complete, use the get-job-output command to download the retrieval job to the file output.json.
aws glacier get-job-output --vault-name testvault --account-id [AccountId] --job-id "Nm1J8J2qnjgiT68k09gCbSSm2wG_IsBFwlxYSQo2JsAbDgwCV0nPy-Sxcc5BeUnQF2y13HWF3zmWvj6wPK5sIlzZOh45" output.json
cat output.json
{
"VaultARN":"arn:aws:glacier:ap-southeast-2:[AccountId]:vaults/testvault",
"InventoryDate":"2021-12-19T21:20:06Z",
"ArchiveList":[
{
"ArchiveId":"HW0qIFuG4o6Ov4CGm8RpbzBFgftorVdKUSx5yBXssKg2wo5vqvXJwtyds29T86ALW3LmtOjtsLymoqh073gq2QBHr0Nitc3ot4HCu-LPOlkoHIhCtx6xU_JdvH8v9NFEMvsThpPJfA",
"ArchiveDescription":"",
"CreationDate":"2021-12-19T09:00:59Z",
"Size":131281,
"SHA256TreeHash":"f2216ef309ad918a2b3286652d5b5be8877f81a8d13181058f11d7d28f12c180"
}
]
}
👉 S3 Glacier prepares an inventory for each vault, about once every 24 hours. So we can only delete the vault after 24 hours as there should not have been any writes since the last inventory.
Use the delete-archive command to delete each archive from a vault until none remain.
aws glacier delete-archive --vault-name testvault --account-id [AccountId] --archive-id “archiveid from the above output"
You can find more information at Deleting an Archive in Amazon S3 Glacier Using the AWS Command Line Interface
4. S3 Multi Part Upload
Multipart upload allows you to upload a single object as a set of parts. Each part is a contiguous portion of the object's data. You can upload these parts in any order.
In general, when your object size reaches 100 MB, you should consider using multipart uploads instead of uploading the object in a single operation.
We can either use s3 or s3 api. You can find more information at How do I use the AWS CLI to perform a multipart upload of a file to Amazon S3?
5. S3 Access point through VPC
Amazon S3 Access Points, a feature of S3, simplify data access for any AWS service or customer application that stores data in S3. With S3 Access Points, customers can create unique access control policies for each access point to easily control access to shared datasets.
The S3 console doesn't support accessing bucket resources using a virtual private cloud (VPC) access point. To access bucket resources from a VPC access point, use the AWS CLI, AWS SDK, or Amazon S3 REST API.
Acknowledgements
Stephane Maarek's- Ultimate AWS Certified SysOps Administrator Associate 2021 on Udemy
AWS Tutorial - S3 Glacier Series - Part 2 of 8 - Create Vault using CLI & Console by
NamrataHShah
Please let me know your thoughts in the comments.
Top comments (0)