DEV Community

Jaeyson Anthony Y.
Jaeyson Anthony Y.

Posted on • Edited on

AWS S3 CLI docker for the impatient (also wasabi)

#s3 #wasabi #aws #docker

What's the problem with AWS cli guide (it has good docs anyway)? well, nothing really complex unless you really want to use it asap. This is more of a basic setup and quick run using docker.

0. Things needed: Docker and AWS S3 creds (access key id/secret access key)

There's a guide for linux distro, and a post-installation steps to run Docker as non-root

1. Pull Docker image 

docker image pull amazon/aws-cli
Enter fullscreen mode Exit fullscreen mode

2. Make credentials

Create ~/.aws dir if not exists, then inside that dir create a file credentials and config:

~/.aws/credentials

[default]
aws_access_key_id=fffff3333ff55555GGGG
aws_secret_access_key=kkkkkkkkkkkkkkkeeeeeeeeeeeeeyyyyyyyyyyyy
Enter fullscreen mode Exit fullscreen mode

~/.aws/config

[default]
region=us-west-2
output=json
Enter fullscreen mode Exit fullscreen mode

Or use an interactive setup

# interactive setup for creds
docker container run --rm -it \
  -v ~/.aws:/root/.aws \
  amazon/aws-cli \
  configure
Enter fullscreen mode Exit fullscreen mode

Check creds:

docker container run --rm -it \
  -v ~/.aws:/root/.aws \
  amazon/aws-cli \
  configure

# or view
cat ~/.aws/credentials
Enter fullscreen mode Exit fullscreen mode

3. cp copy, sync , mv move and ls list files

3.a. cp copy command

$ aws s3 cp <source> <target> [--options]

source could mean your files from your computer or from S3

Add --recursive option if dir:

# upload a local file hyena.png (current dir) to mammals dir
docker container run --rm \
  -v ~/.aws:/root/.aws \
  -v $(pwd):/aws \
  amazon/aws-cli s3 \
  cp hyena.png \
  s3://animal-bucket/mammals/
Enter fullscreen mode Exit fullscreen mode 
# download lynx.png from s3 to current dir
docker container run --rm \
  -v ~/.aws:/root/.aws \
  -v $(pwd):/aws amazon/aws-cli s3 \
  cp s3://animal-bucket/mammals/lynx.png ./
Enter fullscreen mode Exit fullscreen mode 
# upload all contents from reptiles dir (local) to s3
docker container run --rm \
  -v ~/.aws:/root/.aws \
  -v $(pwd):/aws amazon/aws-cli s3 \
  cp reptiles/ s3://animal-bucket/reptiles/ \
  --recursive
Enter fullscreen mode Exit fullscreen mode

3.b ls command 

# list all files in reptiles dir
docker container run --rm \
  -v ~/.aws:/root/.aws \
  amazon/aws-cli s3 \
  ls s3://animal-bucket/reptiles/
Enter fullscreen mode Exit fullscreen mode

e.g. get the size of a bucket

docker container run --rm \
  -v ~/.aws:/root/.aws \
  amazon/aws-cli s3 \
  ls s3://animal-bucket/ \
  --summarize \
  --human-readable \
  --recursive \
  --profile profile-name \
  --endpoint-url=https://s3.<region>.<provider>.com
Enter fullscreen mode Exit fullscreen mode

3.c sync command

aws s3 sync <source> <target> [--options]

Using cp will require the --recursive option to copy multiple files in a dir, while sync command will, by default, copy a whole directory. It will only copy new/modified files.

Add --delete option to remove <target> files that aren't in <source> dir.

# using new-reptiles folder as basis
docker container run --rm \
  -v ~/.aws:/root/.aws \
  -v $(pwd):/aws \
  amazon/aws-cli s3 \
  sync new-reptiles/ s3://animal-bucket/old-reptiles/
Enter fullscreen mode Exit fullscreen mode

3.c mv move command

aws s3 mv <source> <target> [--options]

# Only markdown files are moved from new-reptiles
# to animal-bucket/old-reptiles. The rest aren't included.

docker container run --rm \
  -v ~/.aws:/root/.aws \
  -v $(pwd):/aws \
  amazon/aws-cli s3 \
  mv s3://new-reptiles/ s3://animal-bucket/old-reptiles/ \
  --recursive --exclude "*" --include "*.md" \
  --profile wasabi \
  --endpoint-url=https://s3.<region>.<provider>.com
Enter fullscreen mode Exit fullscreen mode

The rest of the commands can be found in their docs.

4. Using another cloud storage (ie Wasabi)

Same thing with step 2:

~/.aws/credentials

[default]
aws_access_key_id=fffff3333ff55555GGGG
aws_secret_access_key=kkkkkkkkkkkkkkkeeeeeeeeeeeeeyyyyyyyyyyyy
[wasabi]
aws_access_key_id=wwwaaasssaaabbbiiiii
aws_secret_access_key=wwwwwwwaaaaaassssssaaaaaabbbbbbbiiiiiiii
Enter fullscreen mode Exit fullscreen mode

~/.aws/config

[default]
region=us-west-2
output=json
[wasabi]
region=us-east-1
output=table
Enter fullscreen mode Exit fullscreen mode

Or use an interactive setup. Notice I added --profile <PROFILE_NAME> option:

# interactive setup for creds
docker container run --rm -it \
  -v ~/.aws:/root/.aws \
  amazon/aws-cli \
  configure --profile wasabi
Enter fullscreen mode Exit fullscreen mode

Check creds:

docker container run --rm -it \
  -v ~/.aws:/root/.aws \
  amazon/aws-cli \
  configure --profile wasabi
Enter fullscreen mode Exit fullscreen mode

Doing some commands (with --profile and --endpoint-url):

docker container run --rm \
  -v ~/.aws:/root/.aws \
  amazon/aws-cli s3 \
  ls s3://wasabi-bucket/ \
  --profile wasabi \
  --endpoint-url=https://s3.us-west-2.wasabisys.com
Enter fullscreen mode Exit fullscreen mode

Shorten the command with alias:

alias aws=$(docker container run --rm -it \
  -v ~/.aws:/root/.aws \
  -v $(pwd):/aws \
  amazon/aws-cli)
Enter fullscreen mode Exit fullscreen mode

Errors 

An error occurred (NoSuchTagSetError) when calling the GetObjectTagging operation: There is no tag set associated with the bucket.
Enter fullscreen mode Exit fullscreen mode

If you came across this error while using sync, a fix could be either:

  • Grant the s3:GetObjectTagging permission to that user, or
  • adding this option --copy-props metadata-directive (it tells the cli not to get source object tags).

source can be found here.

Top comments (0)

Read next

jjoc007 profile image

Analysis of AWS Certification Question: AWS Solutions Architect Professional — VPC Flow Logs — Kinesis Data Firehose

juan jose orjuela -

vijay431 profile image

Streamlining AWS Secret Management: A Bash Script for Efficient onboarding of secrets into pipeline

Vijay Gangatharan -

karthiksakthiveltechie profile image

AWS CodePipeline V2 type pipelines introduces pipeline variable check rule

Karthik Sakthivel -

code42cate profile image

How to Dockerize and Deploy Fastify APIs

Jonas Scholz -