Ask A Question

Notifications

You’re not receiving notifications from this thread.

Direct File Uploads to S3: Part 3 Discussion

Awesome as usual Chris.

How similar do you think this process would be to using Carrierwave Direct?

Reply

Should be very similar I would imagine although I haven't tried it.

Reply

The main difference is that with CarrierWaveDirect you generate the HTML form with fields for S3 request parameters, which you do while rendering the page, while with Shrine the JavaScript requests the S3 request parameters from your app dynamically (in JSON format) when a file is attached.

Among other things, this allows you to do multiple file uploads with Shrine, because an S3 presign can be requested for each selected file. With CarrierWaveDirect multiple uploads aren't really possible, because it can only generate HTML forms, it doesn't enable you to return these S3 request parameters in JSON format so that the JavaScript can just make the AJAX S3 request(s) itself.

Reply

Chris, can you cover how to upload multiple videos to S3 with progress bars (using Shrine direct upload) + AWS transcoding to web & mobile format after the record has been created in database using ActiveJob ie Sidekiq?

Reply

Chris any plans to do a 4th part of this series to add background processing with ActiveJob for large uploads (or multiple uploads) specifically on Heroku?

Reply

Thanks Chris. This has been really good. I'd like to learn more about what I can do with Shrine.

Reply

Thanks for connecting all the dots on this one! Really appreciate it.

Reply

Absolutely, glad you enjoyed it! :)

Reply

Hi Chris, I'm trying to get this working and following along with your files. I didn't do your final step of converting from JSON to js (so I can't copy/paste your source code file). However, when I try to use what we made with your video, I get js errors showing in the console. After I put the code in JSHint, it tells me to add semi-colons on the end of some of the lines, and an extra ")" on the end of the second last line (so it has 2 "))"), I then get an error that says: uploads.self-a6843c1….js?body=1:2 Uncaught TypeError: "[type=file]".fileupload is not a function(…). I'm not very good at understand js or the console output. Can you see if there might be a problem with that line? Thanks a lot :)

Reply

If you're getting the fileupload is not a function, then maybe jquery fileupload is not included in your application.js file?

Reply

Great episode! Just curious what are the advantages are of using Shrine vs. direct S3 upload described by Heroku: https://devcenter.heroku.co... ? Thanks!

Reply

It's the exact same process if you look into what they do. I recommend Shrine because then you can take advantage of all of the features of it after the file is uploaded such as image processing, thumbnails, background processing, you name it.

Reply

There is a subtle, but nasty bug in the video. Because of the way the form is created in the 'done' function, you create a form with a input type=file, which rails will attempt to process by uploading it and then discarding the data before it hits the controller.

If you are uploading a small picture, you might not even notice the extra work and delay, but I was uploading a 400Mb video, which Rails dutifully attempted to process -- adding over a minute of delay before the controller would respond.

The way I solved this problem (I would love to know if there was a better way) was to create a second form on the page that was hidden (this way, I get all the rails support for security and security tokens, etc.) and I used that second form to build up my data in the done function. This form DOES NOT have an input type=file -- it just has the shrine hidden field.

Reply

Hey Hal, good find. I think I noticed it but it was subtle enough with small files I didn't catch it.

The solution should be to delete the existing file field from the FormData object before you append the jquery file upload one.

      var form      = $(this).closest("form");
var form_data = new FormData(form[0]);
form_data.delete($(this).attr("name")); // Remove the existing form field
form_data.append($(this).attr("name"), JSON.stringify(image)); // Add our json version

Reply

This helped me out a lot, I'm wanting to build a Rails photo gallery app for the company I work for to display all the past jobs and such, currently using Wordpress 😞 Where it takes sweet time to load all the images and such. Though anyway is there a way where you can upload multiple images at once with this uploaded? For like having a gallery with lots of images. Instead of uploading one image at a time?

Reply

Yes, you can just add "multiple" HTML attribute to the file field, which enables it to accept multiple files:


<input type="file" name="file" multiple="true">

The files will still be uploaded in individual requests (there is no performance gain in sending multiple files in a single request anyway), but the uploads will happen in parallel. The shrine-rails-example repository demonstrates this flow.

Reply

Question: This could gives time outs on Heroku if the request to "/images/upload/cache/presign" takes longer than 30 secs?

Reply

The `/images/upload/cache/presign` endpoint is instantaneous, it doesn't make any HTTP requests or anything.

Reply
Alejandro Ventura Alejandro Ventura

What if I want not to send the form to Rails automatically after upload? Because I want to fill some other fields and until the users hit the Submit button I want all the fields an the file fields get stored in the DB.
How can I accomplish this?

Reply

After the file is uploaded, you can write the JSON data to the hidden attachment field (which has the same "name" attribute as the file field; it's mentioned in the "Quick Start" section of the Shrine README). Then when you submit the form, Shrine will attach the file from the JSON data. So the idea is, you can either send a new file (multipart request via the file field) or send an already uploaded file (JSON data via the hidden field) as the attachment attribute.

Reply
Alejandro Ventura Alejandro Ventura

How to make the upload.js works for file_fields added dynamically?

After the page has fully loaded.

Reply
Dave Woodall Dave Woodall

@excid3:disqus Solid Series Sir! (S3 joke) But seriously, this was very timely and I look forward to digging into Shrine. One question - I'm noticing that images are being uploaded to both aws directories (/store and /cache). I can't tell if cache eventually goes away or this is a bug?

Reply

Hi @fakefarm:disqus. I had the same question. I found a discussion here: https://github.com/janko-m/... that suggests two options.

1. Use a Shrine plugin called 'moving' to delete the cached S3 file immediately after it is moved to the store bucket.
http://shrinerb.com/rdoc/cl...
I think you would include this line 'Shrine.plugin :moving, storages: [:store]' in 'config/initializers/shrine.rb'

2. Amazon can be set up to delete files in the cache bucket periodically, documentation here: http://docs.aws.amazon.com/...

@excid3:disqus, My question is... Why does Shrine upload to a 'cache' bucket first and then copy it to a 'store' bucket? I haven't been able to find an answer online yet. If you could explain why Shrine or S3 works this way or post a link it would be greatly appreciated!

Reply

Chris! Just wanted to say that this series is beautifully done. Thank you so much for your hard work and attention to detail. I love how you explain how everything works the hard way or even the wrong way first which helps us understand how the final implementation decisions were made. It took me two full days to get through it all, but everything is working for me. Excellent. Awesome! Thanks again.

Reply

You're welcome and thanks for the encouragement and the support!! 🍻

Reply

@excid3:disqus @jankomarohnic:disqus I have everything working beautifully here, but when I run my tests I get: uninitialized constant ImageUploader::UploadEndpoint (NameError)

I found this discussion recommending putting the uploader class in the app/uploaders directory, but am still getting the same error.

https://groups.google.com/f...

Any ideas? Thank you so much for your help.

Reply

Oh, and I I restarted spring just in case, but that had no effect.

Reply

Hi Chris, nice series. I almost done but I'm having an issue. Everything was working fine but after place all javascript code I'm not able to upload the image to AS3. I'm having this error: "Failed to load resource: the server responded with a status of 400 (Bad Request)". I think could be something related with the url and the presing. I checked all the code and google about it, but I can't find a solution.

Here is my repo, I send it because this code is based on your first Shrine video (where you created a post with description and image): https://github.com/carlos99...

https://uploads.disquscdn.c...

Thanks in advance for any help @Chris Oliver

Reply

Hi Chris, I followed your series to do the direct upload, which worked for me locally, everything works fine, things are nice. But when i deployed my code to server(eb), direct upload stopped working. No error in the console, it just does not reach uploads.js is what I think, but the normal upload works, I can see the file in S3. As there is no error I cannot figure out what the problem is. Any suggestions? Thank you.

Reply
Hi Chris. I've been trying to adapt these series to implement a direct file upload with Vuejs. I'm using vue-resource as my http client and a file upload form from Element UI for Vuejs. So far I've succeeded in getting the presign json responde from the Shrine endpoint, and I sucessfully touch Amazon S3, but then get a 204 response and a strange file (XML?) gets created in my buckets instead of the file I'm trying to upload.

Any ideas on how to go about this without Jquery and/or jQuery file upload?
Reply

Hey Chris, do you have a tutorial for implementing Active Storage, Trix and S3 for images for Rails 6? Trying to follow these videos but am running into some issues I cannot find solutions online. Currently the two issues I cannot get past are errors running local server for:

Shrine.plugin :direct_upload

and

mount ImageUploader::UploadEndpoint => "/images/upload"
Reply

Looks like the issue was I was using a newer version of Shrine. This version works:

gem 'shrine', '~> 2.3.1'
Reply

I haven't done direct uploads with Shrine v3 yet, but that's on my todolist.

Reply
Join the discussion
Create an account Log in

Want to stay up-to-date with Ruby on Rails?

Join 83,453+ developers who get early access to new tutorials, screencasts, articles, and more.

    We care about the protection of your data. Read our Privacy Policy.