Setting Up AWS S3 Buckets + CloudFront CDN for your Assets

Using a cloud stor­age sys­tem like AWS S3 with a CDN dis­tri­b­u­tion can be a con­ve­nient and inex­pen­sive way to store your assets. Here’s how to set it up right.

Andrew Welch / nystudio107

Assets like images, PDFs, and oth­er files are often an impor­tant part of the ​“con­tent” that a Con­tent Man­age­ment Sys­tem handles.

Although this arti­cle was writ­ten with Craft CMS in mind, the vast major­i­ty of the arti­cle applies gener­i­cal­ly to any CMS or website.

Craft CMS has some fan­tas­tic native han­dling of said assets, which by default are stored in fold­ers on your server.

How­ev­er, it can be con­ve­nient to use a cloud-based stor­age sys­tem like Ama­zon Web Ser­vices (AWS) Sim­ple Stor­age Ser­vice (S3):

• You’ll nev­er run out of disk space
• Your assets are inher­ent­ly backed up & stored off-site
• Your assets can go to long-term back­up in AWS Glac­i­er easily
• You can lever­age a Con­tent Deliv­ery Net­work (CDN) in the form of Cloud­front
• Elim­i­nates the need to sync assets between local dev, stag­ing, and pro­duc­tion environments
• It’s cheap (or even free) & scalable

There are oth­er advan­tages as well, but we’ll just assume you’re on-board, and get right into how to set it up.

Set­ting up S3

S3 stores files in ​“buck­ets”, and while you can serve assets direct­ly from an S3 buck­et, I’d strong­ly rec­om­mend against it.

S3 buck­ets are intend­ed for asset stor­age, not serv­ing assets. It’ll work, but you’ll only be serv­ing them out of one geo­graph­ic region (wher­ev­er your S3 buck­et was cre­at­ed), and it real­ly was­n’t designed to for that.

Instead, use AWS Cloud­Front as your Con­tent Deliv­ery Net­work (CDN) that actu­al­ly serves up your assets.

Here’s what it looks like conceptually:

The idea is that when a per­son loads one of your web pages with an image on it, the image will point to a Cloud­Front URL.

If that image is in the cache, it’ll return it from a CDN Edge serv­er that is geo­graph­i­cal­ly near the per­son load­ing the page. This makes it quick, with low latency.

If the image isn’t in the cache, Cloud­Front pulls it from your S3 buck­et, returns it to the per­son, and prop­a­gates the image to the CDN Edge locations.

The rule of thumb here is that S3 buck­ets aren’t pub­licly acces­si­ble. Instead, Cloud­Front is giv­en per­mis­sion to pull assets from the S3 buckets.

That said, let’s get into set­ting it all up.

Step 1: Have an AWS account

If you don’t already have an AWS account, you’re going to need one. Go to your AWS Con­sole and either log in to your account, or cre­ate a new account.

If you want the billing to go direct­ly to your clients, you can either cre­ate an AWS account for them, or use their exist­ing account.

If you fac­tor in the rel­a­tive­ly low cost of S3 into your main­te­nance con­tracts or the like, you can just your account for all of your clients.

If you don’t have one already, you should also set up an admin account for AWS now.

Step 2: Cre­ate an S3 Bucket

Now that we have an AWS account, the first thing we’re going to do is set up our S3 buck­et. We’re going to use kebab-case for all of our AWS enti­ty names, and we’ll use the format:

project-name + - + descriptor

We’ll be set­ting up S3 + Cloud­Front for dev​Mode​.fm, so the project name is devmode, and the buck­et name is devmode-bucket.

In your AWS Con­sole, click on Ser­vices → S3 → + Cre­ate Buck­et. Fill in the name of the buck­et, and click through to the end (we won’t be chang­ing any set­tings from the default oth­er than the name):

Note that for Per­mis­sions, it’s set to Block all pub­lic access. As dis­cussed above, our S3 buck­et will be pri­vate, and the Cloud­Front dis­tri­b­u­tion will be public.

For most projects, I cre­ate one buck­et, and use sub-fold­ers in the buck­et for dif­fer­ent Asset Vol­umes. You can cer­tain­ly also cre­ate more than one buck­et if that’ll work bet­ter for you.

Step 3: Cre­ate a Cus­tom Policy

Next we’ll set up a cus­tom Iden­ti­ty and Access Man­age­ment (IAM) pol­i­cy to con­trol access. IAM poli­cies can be attached to any AWS object, and they con­trol who can access what, with some fair­ly fine-grained permissions.

We’re using a cus­tom pol­i­cy because we want to grant as lit­tle access as pos­si­ble, but still have it work correctly.

From your AWS Con­sole, click on Ser­vices → IAM → Poli­cies → Cre­ate Pol­i­cy.

Click on the JSON tab, and then paste this in:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "acm:ListCertificates",
                "cloudfront:GetDistribution",
                "cloudfront:GetStreamingDistribution",
                "cloudfront:GetDistributionConfig",
                "cloudfront:ListDistributions",
                "cloudfront:ListCloudFrontOriginAccessIdentities",
                "cloudfront:CreateInvalidation",
                "cloudfront:GetInvalidation",
                "cloudfront:ListInvalidations",
                "elasticloadbalancing:DescribeLoadBalancers",
                "iam:ListServerCertificates",
                "sns:ListSubscriptionsByTopic",
                "sns:ListTopics",
                "waf:GetWebACL",
                "waf:ListWebACLs"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListAllMyBuckets"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::REPLACE-WITH-BUCKET-NAME"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::REPLACE-WITH-BUCKET-NAME/*"
            ]
        }
    ]
}

Make sure you replace REPLACE-WITH-BUCKET-NAME with the name of the buck­et we cre­at­ed in Step 2 (in our case devmode-bucket), in both places.

Then click on Review Pol­i­cy , and give your pol­i­cy a name (in our case devmode-policy) and descrip­tion, then click on Cre­ate Pol­i­cy :

Step 4: Cre­ate a Group

Instead of attach­ing the IAM pol­i­cy we cre­at­ed to a buck­et or a user, we’re going to cre­ate a group, and attach it to the group.

We’re doing this because it makes it triv­ial to move users in and out of the group, and the group is what is con­trol­ling access permissions.

You also then won’t be out of luck if you some­how lose the user cre­den­tials, you can just cre­ate a new user and assign it to the group.

From your AWS Con­sole, click on Ser­vices → IAM → Groups → Cre­ate New Group , and give your group a name (in our case devmode-group).

At the Attach Pol­i­cy screen, search for the pol­i­cy we cre­at­ed in Step 3, and check it and click Next Step :

Then click on Cre­ate Group to cre­ate your new group.

Step 5: Cre­ate a User

Next we’re going to cre­ate a user, and assign it to the group we just cre­at­ed. The user we cre­ate will be a gener­ic project user, rather than an actu­al person.

From your AWS Con­sole, click on Ser­vices → IAM → Users → Add user , and give your user a name (in our case devmode-user). Also check only the Pro­gra­mat­ic access check­box under Access type:

This ensures that using these cre­den­tials, there’s AWS Man­age­ment Con­sole access. For that, you’ll use your reg­u­lar admin account & credentials.

Click on Next: Per­mis­sions , then add the user to the group we cre­at­ed in Step 4:

Then click on Next: Tags (we don’t set any tags here), then click on Next: Review :

Then click on Cre­ate User. You’ll be tak­en to a screen where you can see your Access key ID and Secret access key :

Click on Down­load .csv to down­load a CSV file that has your cre­den­tials in it. You will need these cre­den­tials to access your S3 buck­et, and this is the only time you’ll be able to retrieve them.

Step 6: Cre­ate a Cloud­Front Distribution

Now we need to cre­ate a Cloud­Front Dis­tri­b­u­tion that will be act­ing as our CDN, and actu­al­ly deliv­er­ing our assets to the users who request them.

From your AWS Con­sole, click on Ser­vices → Cloud­Front → Cre­ate Dis­tri­b­u­tion , and click on the Get Start­ed but­ton below the Web heading.

Alter the fol­low­ing settings:

• Ori­gin Domain Name  — choose the ori­gin for the S3 buck­et we cre­at­ed in Step 2
• Restrict Buck­et Access  — Yes
• Ori­gin Access Iden­ti­ty  — Cre­ate a New Identity
• Grant Read Per­mis­sions on Buck­et  — Yes, Update Buck­et Policy
• View­er Pro­to­col Pol­i­cy  — Redi­rect HTTP to HTTPS
• Allowed HTTP Meth­ods  — GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE
• Object Caching  — Customize
• Min­i­mum TTL  — 1296000
• Max­i­mum TTL  — 31536000
• Default TTL  — 1296000
• Com­press Objects Auto­mat­i­cal­ly  — Yes

Here’s a full screen­shot for the set­tings we’re using for dev​Mode​.fm:

Then click on Cre­ate Dis­tri­b­u­tion.

You’ll be tak­en to a strange ​“Cloud­Front Pri­vate Con­tent Get­ting Start­ed” page; this isn’t an error page, and there aren’t any more steps to take.

We just need to grab a cou­ple of set­tings from our new­ly cre­at­ed Cloud­Front distribution.

From your AWS Con­sole, click on Ser­vices → Cloud­Front → then click on the Cloud­Front dis­tri­b­u­tion we just created:

We’re going to need the Dis­tri­b­u­tion ID and Domain Name set­tings, so copy them down somewhere.

We’re all done with the AWS S3 + Cloud­Front setup!

Step 7: Con­fig­ure your Asset Vol­umes in Craft CMS

Next we need to con­fig­ure Craft CMS to use our new S3 buck­et set­up. If you’re using some­thing oth­er than Craft CMS, you’ll need to fill in the anal­o­gous settings.

The key point to remem­ber is that the pub­lic URL will be our Cloud­Front dis­tri­b­u­tion URL (in our case https://dnzwsrj1eic0g.cloudfront.net); make sure you add the https:// pro­to­col to it.

First, we’ll need to install the first-par­ty Ama­zon S3 plu­g­in.

Next we’ll need to set up our Asset Vol­umes. I use Envi­ron­ment Vari­ables in my .env file to store all of my secrets, so they aren’t in the data­base, and the don’t end up in Git via Project Con­fig:

# S3 settings
S3_KEY_ID=XXXXXXXXXX
S3_SECRET=XXXXXXXXXX
S3_BUCKET=devmode-bucket
S3_REGION=us-east-2

# CloudFront settings
CLOUDFRONT_URL=https://dnzwsrj1eic0g.cloudfront.net
CLOUDFRONT_DISTRIBUTION_ID=E17SKV1U1OTZKW
CLOUDFRONT_PATH_PREFIX=

In the Craft CMS CP, go to Set­tings → Assets → New vol­ume , and fill in the vol­ume set­tings as shown:

Note that Buck­et has been set to Man­u­al so we can use our envi­ron­ment vari­ables, and we’ve man­u­al­ly added episodes to Sub­fold­er.

Also note that we have turned Make Uploads Pub­lic OFF. If you enable it, it will set a man­u­al ACL on your S3 assets to allow pub­lic access, which isn’t what we want. We want pri­vate S3 assets with pub­lic access through Cloud­Front only

If you’re using Project Con­fig, your project.yaml will end up look­ing like the following:

  e69c8edb-d562-4367-9a05-91a6fd2c7d99:
    name: 'Devmode Episodes'
    handle: devmodeEpisodes
    type: craft\awss3\Volume
    hasUrls: true
    url: $CLOUDFRONT_URL
    settings:
      subfolder: episodes
      keyId: $S3_KEY_ID
      secret: $S3_SECRET
      bucketSelectionMode: manual
      bucket: $S3_BUCKET
      region: $S3_REGION
      expires: '3 months'
      makeUploadsPublic: ''
      storageClass: ''
      cfDistributionId: $CLOUDFRONT_DISTRIBUTION_ID
      cfPrefix: $CLOUDFRONT_PATH_PREFIX
      autoFocalPoint: ''
    sortOrder: 4

Look, mah, no secrets!

Then we can upload an image to our new Asset Volume:

And we can ver­i­fy that the image is indeed com­ing from our Cloud­Front dis­tri­b­u­tion URL:

Fill your Buckets

That’s all you need to start enjoy­ing the ben­e­fits of cloud stor­age in S3, and Cloud­Front as a glob­al Con­tent Deliv­ery Network.

This whole man­u­al set­up can be auto­mat­ed via an AWS Cloud­For­ma­tion stack… but that’s left as an exer­cise for the read­er (or maybe anoth­er article).

Thanks to all around great guy Jonathan Melville of CodeMDD​.io for this help with this article.

Hap­py uploading!

Further Reading

If you want to be notified about new articles, follow nystudio107 on Twitter.

Copyright ©2020 nystudio107. Designed by nystudio107