All threads / Direct File Uploads to S3: Part 3 Discussion
Ask A Question


You’re not receiving notifications from this thread.

Direct File Uploads to S3: Part 3 Discussion

Awesome as usual Chris.

How similar do you think this process would be to using Carrierwave Direct?

Should be very similar I would imagine although I haven't tried it.

The main difference is that with CarrierWaveDirect you generate the HTML form with fields for S3 request parameters, which you do while rendering the page, while with Shrine the JavaScript requests the S3 request parameters from your app dynamically (in JSON format) when a file is attached.

Among other things, this allows you to do multiple file uploads with Shrine, because an S3 presign can be requested for each selected file. With CarrierWaveDirect multiple uploads aren't really possible, because it can only generate HTML forms, it doesn't enable you to return these S3 request parameters in JSON format so that the JavaScript can just make the AJAX S3 request(s) itself.

Thanks for the clarification and insight!

Chris, can you cover how to upload multiple videos to S3 with progress bars (using Shrine direct upload) + AWS transcoding to web & mobile format after the record has been created in database using ActiveJob ie Sidekiq?

Chris any plans to do a 4th part of this series to add background processing with ActiveJob for large uploads (or multiple uploads) specifically on Heroku?

Thanks Chris. This has been really good. I'd like to learn more about what I can do with Shrine.

Thanks for connecting all the dots on this one! Really appreciate it.

Absolutely, glad you enjoyed it! :)

Hi Chris, I'm trying to get this working and following along with your files. I didn't do your final step of converting from JSON to js (so I can't copy/paste your source code file). However, when I try to use what we made with your video, I get js errors showing in the console. After I put the code in JSHint, it tells me to add semi-colons on the end of some of the lines, and an extra ")" on the end of the second last line (so it has 2 "))"), I then get an error that says: uploads.self-a6843c1….js?body=1:2 Uncaught TypeError: "[type=file]".fileupload is not a function(…). I'm not very good at understand js or the console output. Can you see if there might be a problem with that line? Thanks a lot :)

If you're getting the fileupload is not a function, then maybe jquery fileupload is not included in your application.js file?

Hi - it is in the application.js and I only started getting the error after I added the extra ; and ) in the upload.js file.

Might want to undo that? I'm not sure.

OK, well the extra ) and ; were inserted because JS hint says they were missing. I was getting js console errors showing errors when I used the code from the video. If I remove the new ) and ;, the file type error goes away but the JS errors return. I can't see how to fix the JS errors (except for following the JS hint advice).

Do you think there are no missing ; or ) in your source code file? When I try to use that (in JS Hint) I get 8 missing ; and one missing ). Please can you outline the rules for how to fix the js errors so that the file type error goes away too?

The missing semicolons are okay, the browser handle that just fine. The missing ) doesn't sound right, because that would actually cause the JS to break if that were the case. That could be breaking your code by adding that in, or you've got an extra opening ( somewhere.

I updated the file with the semicolons for ya:

Thanks a lot Chris. It's so strange. I copied your file exactly, but my chrome console inspect shows an error that says: Uncaught SyntaxError: missing ) after argument list in this file. I've just finished the javascript course on code school, but still not clever enough to see where the ')' belongs.

Haha no worries! Want to upload the file for me to look at what you've got?

Chris - I'm a donkey. I just restarted the console and its fine now. I don't understand that, but moving on. Thanks so much again. :)

LOL! It might have been cached or something, awesome that it's working now! :D

Great episode! Just curious what are the advantages are of using Shrine vs. direct S3 upload described by Heroku: ? Thanks!

It's the exact same process if you look into what they do. I recommend Shrine because then you can take advantage of all of the features of it after the file is uploaded such as image processing, thumbnails, background processing, you name it.

Thanks Chris!

There is a subtle, but nasty bug in the video. Because of the way the form is created in the 'done' function, you create a form with a input type=file, which rails will attempt to process by uploading it and then discarding the data before it hits the controller.

If you are uploading a small picture, you might not even notice the extra work and delay, but I was uploading a 400Mb video, which Rails dutifully attempted to process -- adding over a minute of delay before the controller would respond.

The way I solved this problem (I would love to know if there was a better way) was to create a second form on the page that was hidden (this way, I get all the rails support for security and security tokens, etc.) and I used that second form to build up my data in the done function. This form DOES NOT have an input type=file -- it just has the shrine hidden field.

Hey Hal, good find. I think I noticed it but it was subtle enough with small files I didn't catch it.

The solution should be to delete the existing file field from the FormData object before you append the jquery file upload one.

      var form      = $(this).closest("form");
var form_data = new FormData(form[0]);
form_data.delete($(this).attr("name")); // Remove the existing form field
form_data.append($(this).attr("name"), JSON.stringify(image)); // Add our json version
Austin Klenk ·

This helped me out a lot, I'm wanting to build a Rails photo gallery app for the company I work for to display all the past jobs and such, currently using Wordpress 😞 Where it takes sweet time to load all the images and such. Though anyway is there a way where you can upload multiple images at once with this uploaded? For like having a gallery with lots of images. Instead of uploading one image at a time?

Yes, you can just add "multiple" HTML attribute to the file field, which enables it to accept multiple files:

<input type="file" name="file" multiple="true">

The files will still be uploaded in individual requests (there is no performance gain in sending multiple files in a single request anyway), but the uploads will happen in parallel. The shrine-rails-example repository demonstrates this flow.

Question: This could gives time outs on Heroku if the request to "/images/upload/cache/presign" takes longer than 30 secs?

The `/images/upload/cache/presign` endpoint is instantaneous, it doesn't make any HTTP requests or anything.

Alejandro Ventura ·

What if I want not to send the form to Rails automatically after upload? Because I want to fill some other fields and until the users hit the Submit button I want all the fields an the file fields get stored in the DB.
How can I accomplish this?

After the file is uploaded, you can write the JSON data to the hidden attachment field (which has the same "name" attribute as the file field; it's mentioned in the "Quick Start" section of the Shrine README). Then when you submit the form, Shrine will attach the file from the JSON data. So the idea is, you can either send a new file (multipart request via the file field) or send an already uploaded file (JSON data via the hidden field) as the attachment attribute.

Alejandro Ventura ·

Thanks I have solved this in the way the docs says as you mention.
Now I have a new problem, how to make the JS work for file fields added dynamically, after the page has fully loaded.

You can just call the `fileupload()` function on the new file field after it is added to the DOM.

Alejandro Ventura ·

How to make the upload.js works for file_fields added dynamically?

After the page has fully loaded.

Dave Woodall ·

@excid3:disqus Solid Series Sir! (S3 joke) But seriously, this was very timely and I look forward to digging into Shrine. One question - I'm noticing that images are being uploaded to both aws directories (/store and /cache). I can't tell if cache eventually goes away or this is a bug?

Hi @fakefarm:disqus. I had the same question. I found a discussion here: that suggests two options.

1. Use a Shrine plugin called 'moving' to delete the cached S3 file immediately after it is moved to the store bucket.
I think you would include this line 'Shrine.plugin :moving, storages: [:store]' in 'config/initializers/shrine.rb'

2. Amazon can be set up to delete files in the cache bucket periodically, documentation here:

@excid3:disqus, My question is... Why does Shrine upload to a 'cache' bucket first and then copy it to a 'store' bucket? I haven't been able to find an answer online yet. If you could explain why Shrine or S3 works this way or post a link it would be greatly appreciated!

The cache directory is exactly what you would imagine. When you're uploading a file with a form and the form validation fails, you want to keep a cached copy of that file temporarily until the user completes the form. You don't want to permanently store the file since the form wasn't valid, so you just need it temporarily stored somewhere. Hence the cache directory. When the validations pass, the file is simply copied from cache to store rather than re-uploading the file.

If we talk in the scope of direct S3 uploads, if Shrine used only one S3 storage, attacker could uncontrollably upload as many files as they want, because the user can "give up" on attaching an uploaded file by not clicking the "Save" button, so it creates orphan files that is not easy to detect and automatically delete. By separating temporary and permanent S3 storage the files that are attached and ones that aren't attached are on separate locations, which allows you to easily set a rule for clearing the unattached files, and it ensures that the permanent storage will never have orphan (unattached) files.

Dave Woodall ·

Thanks, Brian!

Chris! Just wanted to say that this series is beautifully done. Thank you so much for your hard work and attention to detail. I love how you explain how everything works the hard way or even the wrong way first which helps us understand how the final implementation decisions were made. It took me two full days to get through it all, but everything is working for me. Excellent. Awesome! Thanks again.

You're welcome and thanks for the encouragement and the support!! 🍻

@excid3:disqus @jankomarohnic:disqus I have everything working beautifully here, but when I run my tests I get: uninitialized constant ImageUploader::UploadEndpoint (NameError)

I found this discussion recommending putting the uploader class in the app/uploaders directory, but am still getting the same error.

Any ideas? Thank you so much for your help.

Oh, and I I restarted spring just in case, but that had no effect.

Can you post the question to the Google group? Make sure to include all your Shrine-related code. Did you remember to load the direct_upload plugin with Shrine.plugin :direct_upload?


Removing this line from config/initializers/shrine.rb, fixes the issue, allowing my tests to run.

if !Rails.env.test?

I assume the tutorial included that if statement to prevent file uploads during tests, but it prevents all tests from running.

Do you see any downside to removing the if statement?

Hi Chris, nice series. I almost done but I'm having an issue. Everything was working fine but after place all javascript code I'm not able to upload the image to AS3. I'm having this error: "Failed to load resource: the server responded with a status of 400 (Bad Request)". I think could be something related with the url and the presing. I checked all the code and google about it, but I can't find a solution.

Here is my repo, I send it because this code is based on your first Shrine video (where you created a post with description and image):


Thanks in advance for any help @Chris Oliver

Hi Chris, I followed your series to do the direct upload, which worked for me locally, everything works fine, things are nice. But when i deployed my code to server(eb), direct upload stopped working. No error in the console, it just does not reach uploads.js is what I think, but the normal upload works, I can see the file in S3. As there is no error I cannot figure out what the problem is. Any suggestions? Thank you.

Hi Chris. I've been trying to adapt these series to implement a direct file upload with Vuejs. I'm using vue-resource as my http client and a file upload form from Element UI for Vuejs. So far I've succeeded in getting the presign json responde from the Shrine endpoint, and I sucessfully touch Amazon S3, but then get a 204 response and a strange file (XML?) gets created in my buckets instead of the file I'm trying to upload.

Any ideas on how to go about this without Jquery and/or jQuery file upload?

Hey Chris, do you have a tutorial for implementing Active Storage, Trix and S3 for images for Rails 6? Trying to follow these videos but am running into some issues I cannot find solutions online. Currently the two issues I cannot get past are errors running local server for:

Shrine.plugin :direct_upload


mount ImageUploader::UploadEndpoint => "/images/upload"

Looks like the issue was I was using a newer version of Shrine. This version works:

gem 'shrine', '~> 2.3.1'

I haven't done direct uploads with Shrine v3 yet, but that's on my todolist.

Join the discussion

Want to stay up-to-date with Ruby on Rails?

Join 70,078+ developers who get early access to new tutorials, screencasts, articles, and more.

    We care about the protection of your data. Read our Privacy Policy.

    logo Created with Sketch.

    Screencast tutorials to help you learn Ruby on Rails, Javascript, Hotwire, Turbo, Stimulus.js, PostgreSQL, MySQL, Ubuntu, and more. Icons by Icons8

    © 2023 GoRails, LLC. All rights reserved.