When I was working on the serverless variant of imgn, a simple image sharing demo app, I ran into an issue around handling multipart form-data POST...
For further actions, you may consider blocking this person and/or reporting abuse
Just to validate that this isn't a "hack" and you did the right thing -- this is a pretty common workflow with S3 and pretty weird until you grok it fully. We use it for much larger files. You can also initiate a multipart s3 upload from the lambda and then upload individual segments via presigned put part requests. You'll still have to call the API to get those presigned put part urls.
Thanks a lot for confirming this, David! Really appreciate your feedback here. Still wondering why the Gateway causes the issue but I'm glad to learn this is indeed the proper way doing it.
API gateway has payload limits. docs.aws.amazon.com/apigateway/lat... 10MB for input payloads. Also lambda will have similar payload limits, so usually you want to sidestep this by only passing metadata about big files, similar to how you'd want to model databases for files.
An alternative to this URL-passing strategy is to utilize AWS cognito. docs.aws.amazon.com/IAM/latest/Use... Cognito users can be granted IAM permissions allowing them to write to restricted areas of S3 buckets. I'm not a huge fan of cognito user pools, and getting custom auth working through cognito for IAM is a pretty huge task.
One more thing to note: I typically tell people "don't request the presigned URL unless you are going to use it literally ASAP". This is important for when people present download links or upload links. The link should not pre-populate with presigned urls, but instead should be JS that fetches the URL and immediately fires it. I use a 300s time limit on all presigned URLs and have no issues. It's important to know that the time limit is only on the START of the request, not the end -- so people with slow connections only need to start the call in 300s, not finish it (good to know for uploads). Of course, when you use put-part, you should get each part URL right before putting the part -- not all at the beginning.
Hope this helps!
did you mean 300ms? i'm having the same problem with downloading large S3 objects via Api gateway and we are currently using a pre-signed URLs returned from the backend to the UI. However we were concerned with potential security breach if somebody malicious managed to get hold of the pre-signed URL. We currently set the time limit expiry to 5 minutes.
The alternative approach of using Cognito seemed like a large task to undertake given this should help mitigate the security risk
No, I meant 300s (5m). 300ms is far too low and you'd certainly have issues with people not having time to execute it. Keep in mind you're minting this URL and then sending it over network to a client who is expected to then call S3 within the time you have allotted, which includes the wire time to send it to them and their wire time to request it from s3. If I'm remembering correctly, S3 doesn't even deal in milliseconds, the lowest value you can set is 1 second which I think is still too low.
I think if your front end is using a pre-signed URL and it's never shown to a user you should ask yourself, what is the attack vector that you're worried about? A hacker that can get ahold of a 5min URL can certainly get ahold of a 1min URL or even a 30s URL. The closer you get to real world latencies on cell networks, the more likely your users are going to see failures when following the presigned URL. We decided 5minutes was low enough to mitigate risk, but I'd be cautious recommending anything less than 1 minute or like 30 seconds if you have cell users.
From experience though, it does seem like the timeout is until the user calls S3, not finishes the s3 call, so s3 won't hang up on users who have slow connections, which was something we worried would happen when we selected 5min. It is possible that their connection naturally times out or disconnects, which may have some recovery implications with lower presigned URL durations.
One of the things that you could consider is configuring your api gateway endpoint. Is just a question of encapsulating the content in a buffer in the s3 side (I have used encoding base64) and treating the http request as a binary media type by putting the tag. "multipart/form-data" in the settings -> media type in the api console (or at least that it is what I did recently to solve the same problem). I do not know if this solution is the best practice either but surely I believe it is the easiest.
Thank you for the follow-up here and your suggestion. Indeed, I've seen this method documented, for example here and looks sensible to me as well as an alternative.
Although it seems like a solution is different from what I did. My request went directly to lambda (via mock not integration at all) and the binary media types setting is just an input text on the api settings section.
Hi !
I think the problem was that in AWS API Gateway, you need to activate the handling of binary types (Binary Media types) manually and then redeploy your API (do not remember exactly where is setting is). If is is not done, your binary data get corrupted.
Bonus story: One other thing I met when sharing my bucket with other IAM users was that I did not put ACL and ended up not being able to access the data in the bucket and not being able to remove them (even though I am the bucket owner). Adding the ACL bucket-owner-full-control granted me all the rights I needed on these files. But I don't think you will meet this problem with Lambda or Pre signed links.
Thanks for the suggestion! If you look at the SO question I linked to in the post, that was indeed one of the things I tried (without success). Or maybe you mean something else?
More like this: docs.aws.amazon.com/apigateway/lat...
To be more precise, the passthrough thing as convert can imply messing up with you data.