Skip to main content

Subscribe to GoRails to get access to this episode and all other pro episodes, and new awesome content every month.

Subscribe Now
Only $19/month

Unlimited access. Cancel anytime.

30 Direct File Uploads to S3: Part 3

Episode 146 · October 5, 2016

Using jQuery File Upload to upload presigned files directly to Amazon S3 and then store them with Shrine

File Uploading


Transcripts

Subscribe or login to view the transcript for this episode.

Discussion


Gravatar

Awesome as usual Chris.

How similar do you think this process would be to using Carrierwave Direct?

Gravatar

Should be very similar I would imagine although I haven't tried it.

Gravatar

The main difference is that with CarrierWaveDirect you generate the HTML form with fields for S3 request parameters, which you do while rendering the page, while with Shrine the JavaScript requests the S3 request parameters from your app dynamically (in JSON format) when a file is attached.

Among other things, this allows you to do multiple file uploads with Shrine, because an S3 presign can be requested for each selected file. With CarrierWaveDirect multiple uploads aren't really possible, because it can only generate HTML forms, it doesn't enable you to return these S3 request parameters in JSON format so that the JavaScript can just make the AJAX S3 request(s) itself.

Gravatar

Great answer! :D

Gravatar

Thanks for the clarification and insight!


Gravatar

Chris, can you cover how to upload multiple videos to S3 with progress bars (using Shrine direct upload) + AWS transcoding to web & mobile format after the record has been created in database using ActiveJob ie Sidekiq?


Gravatar

Chris any plans to do a 4th part of this series to add background processing with ActiveJob for large uploads (or multiple uploads) specifically on Heroku?


Gravatar

Thanks Chris. This has been really good. I'd like to learn more about what I can do with Shrine.


Gravatar

Thanks for connecting all the dots on this one! Really appreciate it.

Gravatar

Absolutely, glad you enjoyed it! :)


Gravatar

Hi Chris, I'm trying to get this working and following along with your files. I didn't do your final step of converting from JSON to js (so I can't copy/paste your source code file). However, when I try to use what we made with your video, I get js errors showing in the console. After I put the code in JSHint, it tells me to add semi-colons on the end of some of the lines, and an extra ")" on the end of the second last line (so it has 2 "))"), I then get an error that says: uploads.self-a6843c1….js?body=1:2 Uncaught TypeError: "[type=file]".fileupload is not a function(…). I'm not very good at understand js or the console output. Can you see if there might be a problem with that line? Thanks a lot :)

Gravatar

If you're getting the fileupload is not a function, then maybe jquery fileupload is not included in your application.js file?

Gravatar

Hi - it is in the application.js and I only started getting the error after I added the extra ; and ) in the upload.js file.

Gravatar

Might want to undo that? I'm not sure.

Gravatar

OK, well the extra ) and ; were inserted because JS hint says they were missing. I was getting js console errors showing errors when I used the code from the video. If I remove the new ) and ;, the file type error goes away but the JS errors return. I can't see how to fix the JS errors (except for following the JS hint advice).

Gravatar

Do you think there are no missing ; or ) in your source code file? When I try to use that (in JS Hint) I get 8 missing ; and one missing ). Please can you outline the rules for how to fix the js errors so that the file type error goes away too?

Gravatar

The missing semicolons are okay, the browser handle that just fine. The missing ) doesn't sound right, because that would actually cause the JS to break if that were the case. That could be breaking your code by adding that in, or you've got an extra opening ( somewhere.

I updated the file with the semicolons for ya: https://github.com/gorails-...

Gravatar

Thanks a lot Chris. It's so strange. I copied your file exactly, but my chrome console inspect shows an error that says: Uncaught SyntaxError: missing ) after argument list in this file. I've just finished the javascript course on code school, but still not clever enough to see where the ')' belongs.

Gravatar

Haha no worries! Want to upload the file for me to look at what you've got?

Gravatar

Chris - I'm a donkey. I just restarted the console and its fine now. I don't understand that, but moving on. Thanks so much again. :)

Gravatar

LOL! It might have been cached or something, awesome that it's working now! :D


Gravatar

Great episode! Just curious what are the advantages are of using Shrine vs. direct S3 upload described by Heroku: https://devcenter.heroku.co... ? Thanks!

Gravatar

It's the exact same process if you look into what they do. I recommend Shrine because then you can take advantage of all of the features of it after the file is uploaded such as image processing, thumbnails, background processing, you name it.

Gravatar

Gravatar

There is a subtle, but nasty bug in the video. Because of the way the form is created in the 'done' function, you create a form with a input type=file, which rails will attempt to process by uploading it and then discarding the data before it hits the controller.

If you are uploading a small picture, you might not even notice the extra work and delay, but I was uploading a 400Mb video, which Rails dutifully attempted to process -- adding over a minute of delay before the controller would respond.

The way I solved this problem (I would love to know if there was a better way) was to create a second form on the page that was hidden (this way, I get all the rails support for security and security tokens, etc.) and I used that second form to build up my data in the done function. This form DOES NOT have an input type=file -- it just has the shrine hidden field.

Gravatar

Hey Hal, good find. I think I noticed it but it was subtle enough with small files I didn't catch it.

The solution should be to delete the existing file field from the FormData object before you append the jquery file upload one.

      var form      = $(this).closest("form");
var form_data = new FormData(form[0]);
form_data.delete($(this).attr("name")); // Remove the existing form field
form_data.append($(this).attr("name"), JSON.stringify(image)); // Add our json version

Gravatar

This helped me out a lot, I'm wanting to build a Rails photo gallery app for the company I work for to display all the past jobs and such, currently using Wordpress 😞 Where it takes sweet time to load all the images and such. Though anyway is there a way where you can upload multiple images at once with this uploaded? For like having a gallery with lots of images. Instead of uploading one image at a time?

Gravatar

Yes, you can just add "multiple" HTML attribute to the file field, which enables it to accept multiple files:


<input type="file" name="file" multiple="true">

The files will still be uploaded in individual requests (there is no performance gain in sending multiple files in a single request anyway), but the uploads will happen in parallel. The shrine-rails-example repository demonstrates this flow.


Gravatar

Question: This could gives time outs on Heroku if the request to "/images/upload/cache/presign" takes longer than 30 secs?

Gravatar

The `/images/upload/cache/presign` endpoint is instantaneous, it doesn't make any HTTP requests or anything.


Gravatar

What if I want not to send the form to Rails automatically after upload? Because I want to fill some other fields and until the users hit the Submit button I want all the fields an the file fields get stored in the DB.
How can I accomplish this?

Gravatar

After the file is uploaded, you can write the JSON data to the hidden attachment field (which has the same "name" attribute as the file field; it's mentioned in the "Quick Start" section of the Shrine README). Then when you submit the form, Shrine will attach the file from the JSON data. So the idea is, you can either send a new file (multipart request via the file field) or send an already uploaded file (JSON data via the hidden field) as the attachment attribute.

Gravatar

Thanks I have solved this in the way the docs says as you mention.
Now I have a new problem, how to make the JS work for file fields added dynamically, after the page has fully loaded.

Gravatar

You can just call the `fileupload()` function on the new file field after it is added to the DOM.


Gravatar

How to make the upload.js works for file_fields added dynamically?

After the page has fully loaded.


Gravatar

@excid3:disqus Solid Series Sir! (S3 joke) But seriously, this was very timely and I look forward to digging into Shrine. One question - I'm noticing that images are being uploaded to both aws directories (/store and /cache). I can't tell if cache eventually goes away or this is a bug?

Gravatar

Hi @fakefarm:disqus. I had the same question. I found a discussion here: https://github.com/janko-m/... that suggests two options.

1. Use a Shrine plugin called 'moving' to delete the cached S3 file immediately after it is moved to the store bucket.
http://shrinerb.com/rdoc/cl...
I think you would include this line 'Shrine.plugin :moving, storages: [:store]' in 'config/initializers/shrine.rb'

2. Amazon can be set up to delete files in the cache bucket periodically, documentation here: http://docs.aws.amazon.com/...

@excid3:disqus, My question is... Why does Shrine upload to a 'cache' bucket first and then copy it to a 'store' bucket? I haven't been able to find an answer online yet. If you could explain why Shrine or S3 works this way or post a link it would be greatly appreciated!

Gravatar

The cache directory is exactly what you would imagine. When you're uploading a file with a form and the form validation fails, you want to keep a cached copy of that file temporarily until the user completes the form. You don't want to permanently store the file since the form wasn't valid, so you just need it temporarily stored somewhere. Hence the cache directory. When the validations pass, the file is simply copied from cache to store rather than re-uploading the file.

Gravatar

If we talk in the scope of direct S3 uploads, if Shrine used only one S3 storage, attacker could uncontrollably upload as many files as they want, because the user can "give up" on attaching an uploaded file by not clicking the "Save" button, so it creates orphan files that is not easy to detect and automatically delete. By separating temporary and permanent S3 storage the files that are attached and ones that aren't attached are on separate locations, which allows you to easily set a rule for clearing the unattached files, and it ensures that the permanent storage will never have orphan (unattached) files.

Gravatar

Thanks, Brian!


Gravatar

Chris! Just wanted to say that this series is beautifully done. Thank you so much for your hard work and attention to detail. I love how you explain how everything works the hard way or even the wrong way first which helps us understand how the final implementation decisions were made. It took me two full days to get through it all, but everything is working for me. Excellent. Awesome! Thanks again.

Gravatar

You're welcome and thanks for the encouragement and the support!! 🍻


Gravatar

@excid3:disqus @jankomarohnic:disqus I have everything working beautifully here, but when I run my tests I get: uninitialized constant ImageUploader::UploadEndpoint (NameError)

I found this discussion recommending putting the uploader class in the app/uploaders directory, but am still getting the same error.

https://groups.google.com/f...

Any ideas? Thank you so much for your help.

Gravatar

Oh, and I I restarted spring just in case, but that had no effect.

Gravatar

Can you post the question to the Google group? Make sure to include all your Shrine-related code. Did you remember to load the direct_upload plugin with Shrine.plugin :direct_upload?

Gravatar

@excid3:disqus

Removing this line from config/initializers/shrine.rb, fixes the issue, allowing my tests to run.

if !Rails.env.test?

I assume the tutorial included that if statement to prevent file uploads during tests, but it prevents all tests from running.

Do you see any downside to removing the if statement?


Gravatar

Hi Chris, nice series. I almost done but I'm having an issue. Everything was working fine but after place all javascript code I'm not able to upload the image to AS3. I'm having this error: "Failed to load resource: the server responded with a status of 400 (Bad Request)". I think could be something related with the url and the presing. I checked all the code and google about it, but I can't find a solution.

Here is my repo, I send it because this code is based on your first Shrine video (where you created a post with description and image): https://github.com/carlos99...

https://uploads.disquscdn.c...

Thanks in advance for any help @Chris Oliver


Gravatar

Hi Chris, I followed your series to do the direct upload, which worked for me locally, everything works fine, things are nice. But when i deployed my code to server(eb), direct upload stopped working. No error in the console, it just does not reach uploads.js is what I think, but the normal upload works, I can see the file in S3. As there is no error I cannot figure out what the problem is. Any suggestions? Thank you.


Gravatar
Hi Chris. I've been trying to adapt these series to implement a direct file upload with Vuejs. I'm using vue-resource as my http client and a file upload form from Element UI for Vuejs. So far I've succeeded in getting the presign json responde from the Shrine endpoint, and I sucessfully touch Amazon S3, but then get a 204 response and a strange file (XML?) gets created in my buckets instead of the file I'm trying to upload.

Any ideas on how to go about this without Jquery and/or jQuery file upload?

Login or create an account to join the conversation.