DEV Community

Cover image for Terraform Workshop: Automated Account and S3 Bucket Creation on AWS
Sebastian
Sebastian

Posted on • Edited on

Terraform Workshop: Automated Account and S3 Bucket Creation on AWS

#terraform #aws

This article originally appeared at my blog admantium.com.

Terraform is an infrastructure-as-code tool that helps you to manage different resources declaratively. Providers that offer an API for resource managing can be used out of the box, for example for the Hetzner cloud as shown in my last article. When using AWS, this is different: You need to install and configure and AWS cli tool which will then be used by Terraform to create and manage the resources.

This article is a complete introduction to AWS with Terraform. You will learn the installation steps, configuration, and see how to create an S3 bucket and upload objects to it.

The technical context for this article is Terraform v1.4.6, AWS CLI v2.8.2 and Terraform AWS provider v4.33, but it is applicable to newer versions as well.

AWS Prerequisites

To work with AWS, you need a fully registered account, the correct tool installed, and a valid technical user account that is used during the resource creation. Also, bear in mind that working with AWS will incur fees depending on which resources you create. Therefore, be prepared to spend some money when you follow this tutorial.

AWS Account

Go to the AWS startpage and start the onboarding process. During registration, you need to provide credit card information and a telephone number that will be used for the 2-factor authentication.

AWS CLI Tool

The AWS cli is a binary file that you can just download and copy to your executables' path. Just follow the official installation instructions. For an installation on Linux, execute these commands:

$> curl -O "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip"
$> unzip awscli-exe-linux-x86_64.zip
$> sudo ./aws/install

installer: Package name is AWS Command Line Interface
installer: Installing at base path /
installer: The install was successful.

$> aws --version

aws-cli/2.8.2 Python/3.9.11
Enter fullscreen mode Exit fullscreen mode

AWS IAM Admin User

An IAM user is a technical user account that is given detailed permissions to create, read or update different types of resource on AWS. We will create an admin user with full access rights, and with this create further fine-granular scoped users that can only access specific AWS resources

Follow these steps:

  • Give the user the access right "AmazonFullAccess"

  • On the final page, you will see the access key id and the secret access key, these two variables are required when working with terraform.

Be careful to keep the secrets secret on your machine. With this final step completed, all requirements are fulfilled. Lets start using Terraform.

Finally, To work with the AWS cli, you need to execute the command aws login which will then store the required access credentials in ~/.aws/credentials. Alternatively you can export these secrets in your shell as export AWS_ACCESS_KEY_ID= +++ ; export AWS_SECRET_ACCESS_KEY= +++.

Terraform Configuration

Now we are ready to continue the setup with Terraform. First, we create the provider config.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.33.0"
    }
  }
}

provider "aws" {
  region     = "eu-central-1"
}

provider "aws" {
  region     = "us-east-1"
  alias      = "us"
}
Enter fullscreen mode Exit fullscreen mode

Run terraform init once in this project.

$> terraform init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 4.33.0"...
- Installing hashicorp/aws v4.33.0...
- Installed hashicorp/aws v4.33.0 (signed by HashiCorp)

# ..

Terraform has been successfully initialized!
Enter fullscreen mode Exit fullscreen mode

Create an S3 Bucket User

We will use Terraform to create a dedicated user that just has access rights to create an S3 bucket. The relevant configuration is this:

resource "aws_iam_user" "terraform" {
  name = "terraform"
  path = "/"
  tags = {
    "purpose" = "terraform"
  }
  tags_all = {
    "purpose" = "terraform"
  }
}

resource "aws_iam_user_policy" "terraform_s3_access" {
  name = "terraform_s3_access"
  user = aws_iam_user.terraform.id

  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Action" : [
          "s3:*",
          "s3-object-lambda:*"
        ],
        "Resource" : "*"
      }
    ]
  })
}
Enter fullscreen mode Exit fullscreen mode

From the IAM user, we extract its id and the secret key as output variables with the following resource:

resource "aws_iam_access_key" "terraform" {
  user = aws_iam_user.terraform.id
}

output "terraform_iam_key" {
  value = aws_iam_access_key.terraform.id
}

output "terraform_iam_secret" {
  value = aws_iam_access_key.terraform.secret
  sensitive = true
}
Enter fullscreen mode Exit fullscreen mode

When the resources have been created, we can define the AWS access credentials as environment variables so that they will be used from now on.

Use terraform apply to create the user:

Terraform will perform the following actions:
  # aws_iam_user.terraform will be created
  # aws_iam_user_policy.terraform_s3_access will be created

  Enter a value: yes

aws_iam_user.terraform: Creating...
aws_iam_user.terraform: Creation complete after 1s [id=terraform]
aws_iam_user_policy.terraform_s3_access: Creating...
aws_iam_user_policy.terraform_s3_access: Creation complete after 1s [id=terraform:terraform_s3_access]
Enter fullscreen mode Exit fullscreen mode

Once the IAM user is created, export the required environment variables so that new resources will be created with this user:

export AWS_ACCESS_KEY_ID=${$(terraform output terraform_iam_key)}
export AWS_SECRET_ACCESS_KEY=${$(terraform output terraform_iam_secret)}
Enter fullscreen mode Exit fullscreen mode

Create an S3 Bucket

AWS S3 buckets are global unique entities, so you need to find a name that does not exist yet. To make this easy, we will use a Terraform built-in resource called random_pet which will just create a random pet name.

The complete resource definition is this:

resource "aws_s3_bucket" "test-bucket" {
  bucket = "test-bucket-${random_pet.bucket_name.id}"
}

resource "random_pet" "bucket_name" { }
Enter fullscreen mode Exit fullscreen mode

Create the resource:

terraform apply

aws_iam_user.terraform: Refreshing state... [id=terraform]
aws_iam_user_policy.terraform_s3_access: Refreshing state... [id=terraform:terraform_s3_access]
Terraform will perform the following actions:

  # aws_s3_bucket.test-bucket will be created
  + resource "aws_s3_bucket" "test-bucket" {
      + acceleration_status         = (known after apply)
      + acl                         = (known after apply)
      + arn                         = (known after apply)
      + bucket                      = (known after apply)

  #...

  # random_pet.bucket_name will be created
  + resource "random_pet" "bucket_name" {
      + id        = (known after apply)
      + length    = 2
      + separator = "-"
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
Enter fullscreen mode Exit fullscreen mode

The bucket resource is a complexly structured object. Once created, look at it with terraform show:

# aws_s3_bucket.test-bucket:
resource "aws_s3_bucket" "test-bucket" {
    arn                         = "arn:aws:s3:::test-bucket-sunny-ray"
    bucket                      = "test-bucket-sunny-ray"
    bucket_domain_name          = "test-bucket-sunny-ray.s3.amazonaws.com"
    bucket_regional_domain_name = "test-bucket-sunny-ray.s3.eu-central-1.amazonaws.com"
    force_destroy               = false
    hosted_zone_id              = "Z21DNDUVLTQW6Q"
    id                          = "test-bucket-sunny-ray"
    object_lock_enabled         = false
    region                      = "eu-central-1"
    request_payer               = "BucketOwner"
    tags_all                    = {}

    grant {
        id          = "4a678af5306cd2450b9c11bf8aa39685fc4ee991e5f55bbddb8749a609a34ee8"
        permissions = [
            "FULL_CONTROL",
        ]
        type        = "CanonicalUser"
    }

    versioning {
        enabled    = false
        mfa_delete = false
    }
}
Enter fullscreen mode Exit fullscreen mode

Making the Bucket Accessible

In its default state, the bucket is not accessible. If you make an HTTP request with curl to the bucket’s regional domain name, you will get the following XML document specifying the access error:

curl test-bucket-sunny-ray.s3.eu-central-1.amazonaws.com/index.html
<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>AccessDenied</Code>
  <Message>Access Denied</Message>
  <RequestId>REH5D0H02M8DVY16</RequestId>
  <HostId>b3XBqKWrE8XcXiQNmZx+6cV43XGyIqkJLgXjoYL2NQf0JzVmlLS7YGTl/DntuqvigXtFWOVINtU=</HostId>
</Error>
Enter fullscreen mode Exit fullscreen mode

We need to define an access policy first:

resource "aws_s3_bucket_policy" "allow_public_access" {
  bucket = aws_s3_bucket.test-bucket.id
  policy = data.aws_iam_policy_document.allow_public_access.json
}

data "aws_iam_policy_document" "allow_public_access" {
  statement {
    sid = "PublicReadGetObject"
    effect = "Allow"
    actions = ["s3:GetObject"]
    resources = [aws_s3_bucket.test-bucket.arn]
  }
}
Enter fullscreen mode Exit fullscreen mode

Put an Object Into the Bucket

An AWS S3 bucket can contain any number of objects. For simplicity I will just upload the following HTML file:

<!-- index.html -->
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>AWS S3 Bucket HTML File</title>
  </head>
  <body>
    <main>
        <h1>Hello World!</h1>
    </main>
  </body>
</html>
Enter fullscreen mode Exit fullscreen mode

The Terraform resource description is simple, you need to provide the bucket name, give the object an unique key, and reference the source file:

resource "aws_s3_object" "hello_world_html" {
  bucket = aws_s3_bucket.test-bucket.bucket
  key    = "index.html"
  source = "src/index.html"
  etag = filemd5("src/index.html")
}
Enter fullscreen mode Exit fullscreen mode

Then we upload the file ...

terraform apply

Terraform will perform the following actions:

  # aws_s3_object.hello_world_html will be created
  + resource "aws_s3_object" "hello_world_html" {
      + acl                    = "private"
      + bucket                 = "test-bucket-fancy-catfish"
      + bucket_key_enabled     = (known after apply)
      + content_type           = (known after apply)
      + etag                   = "d002020803f03b76bc1a0f2fdeaefd2e"
      + force_destroy          = false
      + id                     = (known after apply)
      + key                    = "index.html"
      + kms_key_id             = (known after apply)
      + server_side_encryption = (known after apply)
      + source                 = "src/index.html"
      + storage_class          = (known after apply)
      + tags_all               = (known after apply)
      + version_id             = (known after apply)
    }
Enter fullscreen mode Exit fullscreen mode

... and can access it with curl:

curl test-bucket-sunny-ray.s3.eu-central-1.amazonaws.com/index.html

<!-- index.html -->
<!DOCTYPE html>
<html lang="en">
#...
Enter fullscreen mode Exit fullscreen mode

Conclusion

Terraform is a universal infrastructure management language. When working with AWS, you need to have a fully registered account and install the AWS cli locally. Then, Terraform uses the cli and its stored access credentials to create AWS resources. This article showed you how to create an AWS bucket and upload objects to it. The particular steps are a) create and AWS account, b) install the AWS cli and authenticate with the AWS API, c) create an IAM user, d) create a Terraform project and define the providers, e) Create a specific IAM user that can only manage S3 bucket resources, f) create the bucket and an access policy, g) upload bucket objects. Overall, using Terraform to manage AWS resources provides all the benefits of versioned and declarative infrastructure as code.

Top comments (0)

Read next

danial profile image

Navigating Amazon Web Services: A Guide to Getting Help through Email

Danial Ranjha -

amandeol063 profile image

Week 3 in DevOps: Beginning with Advanced AWS Services and Security

Aman Deol -

ndzenyuy profile image

Streamlining Kubernetes Deployments: CI/CD with GitHub Actions and Helm for EKS

Jones Ndzenyuy -

094459 profile image

Amazon Q Developer Tips: No.10 Personalise Amazon Q's output

Ricardo Sueiras -