Shrine and AWS

I wanted a direct upload (client uploads a file, and then passes a link to the server) solution for a new app that I'm writing. The backend was going to be shared amongst a lot of different services. I looked around - Shrine seemed the most promising, so I went for it.

A lot of this info just applies to S3 direct uploads and API signing in general, so if that confuses you, read it anyway.

I didn't want to use a front end solution with a premade library, because I was trying to figure this out for a mobile app. I think I did - if you can do it in REST, you can do it anywhere.

I'm writing this as I have just figured out how to make Postman/Insomnia work, and I'm not gonna lie - I'm pretty annoyed. It really should not have taken days to get an easy example of how to use Shrine without subscribing to GoRails (I did) or using the Uppy library (Not an option a lot of the time).

Quick aside - While GoRails really wasn't worth it for me and I cancelled almost immediately when they used JS as well, Chris (one of the tutorial guys) was super cool and easy to understand in his videos. Not knocking the service as a whole, they just didn't have what I needed. Well, except for the sweet, sweet test image I used (Google "Money Sloth").

Okay. On to the main part.

The Set Up - The Break Down

Setting up Shrine is actually pretty straight forward for the server side. I'm going to skip the entire thing because the tutorials are really straight forward, I just want to explain the AWS part and give you a quick tip. Make sure to set up your bucket with CORS like shrine tells you, except insofar as we are concerned, leave the accepted domains to * because we're rest-client testing.

Quick tip - if you're hesitant to use shrine because you want to use devise or some kind of auth solution to protect your presign endpoints (and you should!), rest easy. The docs are straightforward.

We use POST over PUT because of a buncha reasons that are best articulated here (read the link, read the answer).

Quick overview of how it all works

You want the client to upload images to S3 without your server having to handle them. However, if you just let anyone upload anything, you're going to have a terrible day some day when you find out that someone has been using your AWS account to store their exotic collection of Giraffe Dancing videos. All 20 terabytes of it.

Signed posts let you avoid that problem by doing the following, in order -

1) Be authenticated and ask your server for permission to upload a file. You tell them what type of file (you CAN tell them the filename, but no. No need). For example, "I will upload a jpeg."

2) Shrine runs its magic and generates a BUNCH of specific stuff to let you do that. Like, and I am serious, 8 things that you will need to upload the file in addition to the file itself.

3) You take that mountain of information and do the upload.

ERROR LIST

If you see one of these errors, here is how you fix it:

Some error related to expired request - Expiration time has passed, do it again.
Some error related to 403 expired request policy something - your server time is messed up. In my ABSURD case, I had updated Postgresql through homebrew which screwed up the database clock. I restarted. Solved.

Some error related to signing - One of your headers is messed up. Of note - some policies, when generated, have an = sign at the end of them. Make sure that is written / deleted correctly when copy pasting.

Some error that says "Policy something something "eq", "Some header", "some value"" - you're missing that value. Put that value IN THE FORM DATA with the key/value that it says in the error. i.e. "You are missing eq, content-type, image/jpeg go into your form data and add key:content-type, value:image/jpeg

Before we even start

The post request that you send to S3 must be form-data. You don't need any headers, at all, because the form data will have everything you need. It must, obviously, be a post request. This next part is super, super, super important: the key for the object to be stored in the bucket MUST COME FIRST, and the attached file MUST COME LAST. Your post will not work if that doesn't happen. Also, once you write the whole request, if you aren't getting the file to post go ahead and check the actual curl code behind the request. Look for the part that says "filename=" and make sure the filename is actually attached. Mine didn't until I upgraded to the latest version of postman.

Now we can start

Okay. Wow. Let's actually start this.

First send a simple get request to your presign endpoint on your server. I set mine to http://localhost:3000/api/s3/params.

The get request should have one query parameter - "type: "

For example, if your rails code looks like this in your presign endpoint(and it should, more or less):

Shrine.storages = {
  cache: Shrine::Storage::S3.new(prefix: 'cache', **s3_options),
  store: Shrine::Storage::S3.new(**s3_options)
}

Shrine.plugin :activerecord
Shrine.plugin :restore_cached_data # refresh metadata when attaching the cached file
Shrine.plugin :determine_mime_type
Shrine.plugin :presign_endpoint, presign_options: lambda { |request|
  filename = request.params['filename']
  type     = request.params['type']
  {
    content_disposition: ContentDisposition.inline(filename), # set download filename
    content_type: type, # set content type (required if using DigitalOcean Spaces)
    content_length_range: 0..(1 * 1024 * 1024) # limit upload size to 1 MB
  }
}

At the lines

filename = request.params['filename']
type     = request.params['type']

That means that our GET request should use type as the key. If you set it to something like type = request.params['content'] then the key name should be content. Regardless, a valid param might look like type=image/jpeg(url escaped by your REST client, of course)

We actually don't need the filename param. You can include it, but if the filename param doesn't match exactly it won't send. So if you have a picture of say, a gorgeous money sloth and you want to name it something like MONAY.jpg, if you send the filename as MONAY.jpg and then attach that from your rest client, you won't get it to work. This is because your rest client will attach it as /User/me/Documents/MONAY.jpg, and those names do not match when you upload.

Anyway.

Okay, so after you send your straightforward get request (make sure to include auth headers if you're using an auth solution), you'll get back something that looks like this:

{
  "fields": {
    "key": "cache/ac48d4a7764610579df3f26c316f3947",
    "Content-Disposition": "inline",
    "Content-Type": "image/jpeg",
    "policy": "eyJleHBpcmF0aW9uIjoiMjAxOS0wMi0yMlQwOTo1MjozMloiLCJjb25kaXRpb25zIjpbeyJidWNrZXQiOiJ2aXNpb24td2ViLWltYWdlLWRldiJ9LHsia2V5IjoiY2FjaGUvYWM0OGQ0YTc3NjQ2MTA1NzlkZjNmMjZjMzE2ZjM5NDcifSx7IkNvbnRlbnQtRGlzcG9zaXRpb24iOiJpbmxpbmUifSx7IkNvbnRlbnQtVHlwZSI6ImltYWdlL2pwZWcifSxbImNvbnRlbnQtbGVuZ3RoLXJhbmdlIiwwLDEwNDg1NzZdLHsieC1hbXotY3JlZGVudGlhbCI6IkFLSUFKT0VSVTNMTkFaSExOUFRBLzIwMTkwMjIyL2FwLW5vcnRoZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LHsieC1hbXotYWxnb3JpdGhtIjoiQVdTNC1ITUFDLVNIQTI1NiJ9LHsieC1hbXotZGF0ZSI6IjIwMTkwMjIyVDA4NTIzMloifV19",
    "x-amz-credential": "********/20190222/ap-northeast-1/s3/aws4_request",
    "x-amz-algorithm": "AWS4-HMAC-SHA256",
    "x-amz-date": "20190222T085232Z",
    "x-amz-signature": "769489a916a228d142e9bff18688bf45f44c7e77eac81f4797b8c48e3b8ab756"
  },
  "headers": {},
  "method": "post",
  "url": "https://BUCKETNAME.s3.ap-northeast-1.amazonaws.com"
}

Don't panic. One by one -

key - what shrine named the object in s3 for you.
content-disposition - Irrelevant for us, but docs anyway. Just copy it.

Content-Type - yep, this is what we said our type was.
Policy - just copy it. This tells Amazon what is permitted to be uploaded, and how.

The next four that start with x-amx-* - these four, together, are part of the Amazon V4 signature protocol. They tell Amazon that the request is legitimate. That's really it, just make sure to copy them correctly every time and you won't have issues.

Headers - this is messed up. Ignore these entirely. I spent way, way too long messing with headers before I realized they were red herrings.

Method - Post it.
Url - Post it to this Url.

Okay, now I'm just going to go ahead and show you what your client needs to look like.

If you are using postman, copy your URL into the url bar and set it to post. Click the body tab, and set it to form-data. Now click the bulk-edit orange text in the upper-right hand corner, and past this in -

key:YourKeyHere
x-amz-credential:YourCredentialsHere
policy:YourPolicyHere
x-amz-algorithm:AWS4-HMAC-SHA256
x-amz-date:YourDateHere
x-amz-signature:YourSigHere
Content-Disposition:inline
Content-Type:YourContentTypeHere

Just replace the values with your values, add the file at the bottomand you're golden. Or, if you are using another rest client, use the above keys and fill in your own values a bit slower because you won't have bulk copy-paste, so you need to go line by line. then add the file at the bottom

Okay. Now, hit send. If everything worked, you should get back a 204, and the image is in your bucket. If everything did NOT work, weep and look at the error list above, or post a comment and I'll do my best to troubleshoot on the roughly one day/week that I read comments.

Using that response to save the file

This one is actually pretty simple. Make sure that you have the :restore_cached_data plugin available (see my Shrine code, above). In your Ruby code, once you've authenticated and such, call the following:

ModelInQuestion.update_attributes(attachment_name: {id: "ID OF S3 OBJECT", storage: "cache"}.to_json)

For example, if your object is something like books with your image uploader that you set up in the folder set to image:, assuming your AWS response looks like this:

x-amz-id-2 → stuff
x-amz-request-id → stuff
Date →Sat, 23 Feb 2019 10:04:20 GMT
ETag →"daa14e86f803fca11a1baff256fa259b"
Location →https://bucket.s3.ap-northeast-1.amazonaws.com/cache%2Fdabe031bee03ae8365e2ec48b523bb33
Content-Length →0
Server →AmazonS3

You want to call this code:

YourBookModelHere.update_attributes(image: {id: "dabe031bee03ae8365e2ec48b523bb33", storage: "cache"}.to_json)

We get that image ID from the end of the location tag. The documentation actually has a regex that parses that for you - last paragraph

Quick note - I found that if you want to pass the string in and use the regex, you're going to have to take the url string, restore it, and then remove the beginning of the line character from the regex.

Or, in code:

URI.decode(params[:link])[/cache\/(.+)/, 1]

And now when you run that update attributes, you should get this cool thing back:

<ImageUploader::UploadedFile:0x00007fbb995c8eb8
@data={
"id"=>"e68f1910d4fb723d6538383dbbac5f78",
"storage"=>"store",
"metadata"=>{"filename"=>nil, "size"=>92931, "mime_type"=>"image/jpeg"}}>

note that if your mime_type is nil, you need to add Shrine.plugin :determine_mime_type to your plugin list.

Now, if you call yourbook.image_url, you get a super-cool signed url that lasts for 15 minutes!

Lastly, if you want to set your expiration time for something a bit more narrow than 15 minutes (say, 5 minutes), use the default_url_options plugin -

  # 90 seconds to use the upload URL, 5 minutes to use the GET url
  Shrine.plugin :default_url_options,
    cache: { expires_in: 90 },
    store: { expires_in: 300 }

Huge shout out to Janko the gem maintainer for all of his help in the Shrine Google group. I recommend going over there if you have questions - he's very friendly!