Skip to main content

Subscribe to GoRails to get access to this episode and all other pro episodes, and new awesome content every month.

Subscribe Now
Only $19/month

Unlimited access. Cancel anytime.

34 Direct File Uploads to S3: Part 1

Episode 144 · October 5, 2016

Setup your Amazon S3 account to receive direct file uploads from users in your Rails app

File Uploading


Transcripts

Subscribe or login to view the transcript for this episode.

Discussion


Gravatar
Shawn Wilson (370 XP) on

how could you handle wildcard domains with CORS?? I have a multi-tenant site with dynamic subdomains? or can i just leave it defaulted to *?

Gravatar
Chris Oliver (170,390 XP) on

I would still recommend setting your domain explicitly. You can use in it to specify a wildcard subdomain so you can handle all of those.

From their docs:

In the AllowedOrigin element you specify the origins that you want to allow cross-domain requests from, for example, http://www.example.com. The origin string can contain at most one * wildcard character, such as http://.example.com. You can optionally specify * as the origin to enable all the origins to send cross-origin requests. You can also specify https to enable only secure origins.

Gravatar
Daniel Weaver (320 XP) on

Great walkthrough of setting up CORS and IAM permissions. I set these up a few years ago for my app by digging through the AWS docs and, honestly, I wasn't sure about anything that I was doing but it all eventually worked. Now I actually understand more of the process! Thanks!

Gravatar
Chris Oliver (170,390 XP) on

AWS docs are some of the worst. Plus every little thing has it's own acronym which makes it even tougher to sift through!

Gravatar
Nick on

Great episode, just subscribed. Everything much clearer in under 15 mins :)

+1 AWS docs are officially the worst. And seems there is no consistency to the style of the documents either.


Gravatar
W. Ama on

Hi, I have a question regarding what you think would be best to use for a new app. Carrierwave picture uploads or a direct upload like the one you are presenting here.
It seems like carrierwave would send data to the hosting server which would reach the limit allowed faster while a direct upload can help in avoiding that from happening.
Also, would it be simple to change to a direct upload approach later on along the road if I end up using carrierwave for now without having to make all the user re upload new pictures for their profiles?
Thank you.

Gravatar
Janko Marohnić (190 XP) on

Shrine has a distinction of temporary and permanent storage, and you can use any kind of storage for both. Direct S3 uploads just means that you're using S3 for caching files. So if you're already permanently storing your files on S3 with CarrierWave, adding direct uploads is just an addition, you wouldn't need to reupload any of your existing files (they are already permanently stored).

I listed here the main benefits of using direct S3 uploads. Even if you would like to ultimately store your files on the filesystem, with Shrine you can still upload files directly to S3, and then they will just be downloaded to filesystem on the server-side. This reduces the load on your server, because it doesn't need to accept file uploads. That's probably a rarer use case, but I just wanted to illustrate what do direct S3 uploads mean.

Gravatar
Chris Oliver (170,390 XP) on

Like Janko said, you can do uploads to your server or to S3 with Shrine just like you would with Carrierwave. Shrine can do everything you could do in Carrierwave (and better in my opinion) so I would recommend using it for everything.

My recommendation would be to never setup your files to upload to your server's filesystem unless it's a hobby project. You'd have to migrate the files once your harddrive is full and that's no fun.

With S3, you have two options:

1. You can upload your files to your Rails server and Shrine will send them over to S3.
2. You can directly upload files to S3 like I showed in this video.

The first option is simpler because you just have to configure Shrine to do the upload to S3.

Direct uploads use your user's browser to upload straight to S3 and it skips your server. That means the direct uploads are faster, but they're a fair amount more work (like you see in the video).

The simplest and best option to get started is to simply use S3 (#1) for both the cache and store for Shrine. You don't have to build any of the Javascript stuff right away and all your uploads will be saved to S3. When you're ready, you can add in the Javascript portion for direct uploads in order to make them upload faster. Since you'd already be using S3, it's a simple change this way.

Gravatar
W. Ama on

Allright thank you guys. That helps. Also, for resizing the pictures once they are displayed on a user's profile for example, they should be resized at the processing phase I would believe? or just through a css styling implemented to where they are displayed? Thanks again.

Gravatar
Chris Oliver (170,390 XP) on

Yeah, you'll probably want to crop the pictures to a standard size in the processing phase.

The problem with using only CSS is that your user might upload a 10MB image and you don't want all your users having to download that, so it's best to always crop to a few standard sizes to prevent that from happening so your site can continue being fast.

You can set like a standard avatar size (say 100px x 100px) and then resize it with CSS so the image can be displayed at various sizes but reused from the browser cache.

Gravatar
W. Ama on

Interesting. That makes sense. Allright, I will figure out how to do the resizing at the processing phase now. If I have any question, I will let you know. Thanks a lot.


Gravatar
W. Ama on

Hi,
As I am just launching an app, I thought of using these 3 sections of the course to upload users images to their profiles on heroku. I will thus be storing users images locally I suppose, I am still checking on that.
The reason why I thought of this is to avoid using amazon for now since I don't know how it charges if many users end up uploading their pictures, let me know what you think of that and what you would recommend.

Now, my question is. Would it be ok to store the users images (files) locally, and then if things pick up, move everything to amazon S3, or would this be a headache on the long time? Meaning that I should just use S3 from the start.

I have asked a similar question before, but I am making sure I understand things well before committing to a decision for now.
The reason I am asking this again is because of the free tier on amazon as I am not sure when and how it will start charging and so on if I end up using it.

Thank you. I appreciate the help.

Gravatar
Chris Oliver (170,390 XP) on

So Heroku won't actually allow you to store files permanently on their servers. You are required to use an external storage service like S3.

If you deployed to Digital Ocean, you can store files since you control the server. Migration isn't the worst, but it can take some time. If you plan on people uploading a lot of files, it's probably worth just using S3 from the beginning.

In either case, I guess I would say you might as well just use S3 from storage from the beginning if you plan to using S3 later.

Gravatar
W. Ama on

Allright, thank you!


Gravatar
Diego Felipe Lafuente on

Hello Chris, why did you add a "PutObjectAcl" permission into the user policy? What is the difference between "PutObject" and "PutObjectAcl"? Thanks.

Gravatar
Chris Oliver (170,390 XP) on

PutObjectAcl lets you change the ACL of an object in S3. This may not be required, but it's useful if you want to change permissions on files ever like making them public or private. http://docs.aws.amazon.com/...


Gravatar
Dave Woodall on

Just went through part 1 and AWS changed some paint but the process was 99.9% the same. Solid screencast, Chris. Thanks!


Login or create an account to join the conversation.