Direct File Uploads to S3: Part 3 Discussion
Awesome as usual Chris.
How similar do you think this process would be to using Carrierwave Direct?
Chris, can you cover how to upload multiple videos to S3 with progress bars (using Shrine direct upload) + AWS transcoding to web & mobile format after the record has been created in database using ActiveJob ie Sidekiq?
Chris any plans to do a 4th part of this series to add background processing with ActiveJob for large uploads (or multiple uploads) specifically on Heroku?
Thanks Chris. This has been really good. I'd like to learn more about what I can do with Shrine.
Thanks for connecting all the dots on this one! Really appreciate it.
Hi Chris, I'm trying to get this working and following along with your files. I didn't do your final step of converting from JSON to js (so I can't copy/paste your source code file). However, when I try to use what we made with your video, I get js errors showing in the console. After I put the code in JSHint, it tells me to add semi-colons on the end of some of the lines, and an extra ")" on the end of the second last line (so it has 2 "))"), I then get an error that says: uploads.self-a6843c1….js?body=1:2 Uncaught TypeError: "[type=file]".fileupload is not a function(…). I'm not very good at understand js or the console output. Can you see if there might be a problem with that line? Thanks a lot :)
If you're getting the fileupload is not a function, then maybe jquery fileupload is not included in your application.js file?
Hi - it is in the application.js and I only started getting the error after I added the extra ; and ) in the upload.js file.
OK, well the extra ) and ; were inserted because JS hint says they were missing. I was getting js console errors showing errors when I used the code from the video. If I remove the new ) and ;, the file type error goes away but the JS errors return. I can't see how to fix the JS errors (except for following the JS hint advice).
Do you think there are no missing ; or ) in your source code file? When I try to use that (in JS Hint) I get 8 missing ; and one missing ). Please can you outline the rules for how to fix the js errors so that the file type error goes away too?
The missing semicolons are okay, the browser handle that just fine. The missing ) doesn't sound right, because that would actually cause the JS to break if that were the case. That could be breaking your code by adding that in, or you've got an extra opening ( somewhere.
I updated the file with the semicolons for ya: https://github.com/gorails-...
Haha no worries! Want to upload the file for me to look at what you've got?
Chris - I'm a donkey. I just restarted the console and its fine now. I don't understand that, but moving on. Thanks so much again. :)
LOL! It might have been cached or something, awesome that it's working now! :D
Great episode! Just curious what are the advantages are of using Shrine vs. direct S3 upload described by Heroku: https://devcenter.heroku.co... ? Thanks!
There is a subtle, but nasty bug in the video. Because of the way the form is created in the 'done' function, you create a form with a input type=file, which rails will attempt to process by uploading it and then discarding the data before it hits the controller.
If you are uploading a small picture, you might not even notice the extra work and delay, but I was uploading a 400Mb video, which Rails dutifully attempted to process -- adding over a minute of delay before the controller would respond.
The way I solved this problem (I would love to know if there was a better way) was to create a second form on the page that was hidden (this way, I get all the rails support for security and security tokens, etc.) and I used that second form to build up my data in the done function. This form DOES NOT have an input type=file -- it just has the shrine hidden field.
Hey Hal, good find. I think I noticed it but it was subtle enough with small files I didn't catch it.
The solution should be to delete the existing file field from the FormData object before you append the jquery file upload one.
var form = $(this).closest("form");
var form_data = new FormData(form);
form_data.delete($(this).attr("name")); // Remove the existing form field
form_data.append($(this).attr("name"), JSON.stringify(image)); // Add our json version
This helped me out a lot, I'm wanting to build a Rails photo gallery app for the company I work for to display all the past jobs and such, currently using Wordpress 😞 Where it takes sweet time to load all the images and such. Though anyway is there a way where you can upload multiple images at once with this uploaded? For like having a gallery with lots of images. Instead of uploading one image at a time?
Yes, you can just add "multiple" HTML attribute to the file field, which enables it to accept multiple files:
<input type="file" name="file" multiple="true">
The files will still be uploaded in individual requests (there is no performance gain in sending multiple files in a single request anyway), but the uploads will happen in parallel. The shrine-rails-example repository demonstrates this flow.
Question: This could gives time outs on Heroku if the request to "/images/upload/cache/presign" takes longer than 30 secs?
What if I want not to send the form to Rails automatically after upload? Because I want to fill some other fields and until the users hit the Submit button I want all the fields an the file fields get stored in the DB.
How can I accomplish this?
After the file is uploaded, you can write the JSON data to the hidden attachment field (which has the same "name" attribute as the file field; it's mentioned in the "Quick Start" section of the Shrine README). Then when you submit the form, Shrine will attach the file from the JSON data. So the idea is, you can either send a new file (multipart request via the file field) or send an already uploaded file (JSON data via the hidden field) as the attachment attribute.
Thanks I have solved this in the way the docs says as you mention.
Now I have a new problem, how to make the JS work for file fields added dynamically, after the page has fully loaded.
You can just call the `fileupload()` function on the new file field after it is added to the DOM.
How to make the upload.js works for file_fields added dynamically?
After the page has fully loaded.
@excid3:disqus Solid Series Sir! (S3 joke) But seriously, this was very timely and I look forward to digging into Shrine. One question - I'm noticing that images are being uploaded to both aws directories (/store and /cache). I can't tell if cache eventually goes away or this is a bug?
Hi @fakefarm:disqus. I had the same question. I found a discussion here: https://github.com/janko-m/... that suggests two options.
1. Use a Shrine plugin called 'moving' to delete the cached S3 file immediately after it is moved to the store bucket.
I think you would include this line 'Shrine.plugin :moving, storages: [:store]' in 'config/initializers/shrine.rb'
2. Amazon can be set up to delete files in the cache bucket periodically, documentation here: http://docs.aws.amazon.com/...
@excid3:disqus, My question is... Why does Shrine upload to a 'cache' bucket first and then copy it to a 'store' bucket? I haven't been able to find an answer online yet. If you could explain why Shrine or S3 works this way or post a link it would be greatly appreciated!
The cache directory is exactly what you would imagine. When you're uploading a file with a form and the form validation fails, you want to keep a cached copy of that file temporarily until the user completes the form. You don't want to permanently store the file since the form wasn't valid, so you just need it temporarily stored somewhere. Hence the cache directory. When the validations pass, the file is simply copied from cache to store rather than re-uploading the file.
If we talk in the scope of direct S3 uploads, if Shrine used only one S3 storage, attacker could uncontrollably upload as many files as they want, because the user can "give up" on attaching an uploaded file by not clicking the "Save" button, so it creates orphan files that is not easy to detect and automatically delete. By separating temporary and permanent S3 storage the files that are attached and ones that aren't attached are on separate locations, which allows you to easily set a rule for clearing the unattached files, and it ensures that the permanent storage will never have orphan (unattached) files.
Chris! Just wanted to say that this series is beautifully done. Thank you so much for your hard work and attention to detail. I love how you explain how everything works the hard way or even the wrong way first which helps us understand how the final implementation decisions were made. It took me two full days to get through it all, but everything is working for me. Excellent. Awesome! Thanks again.
@excid3:disqus @jankomarohnic:disqus I have everything working beautifully here, but when I run my tests I get: uninitialized constant ImageUploader::UploadEndpoint (NameError)
I found this discussion recommending putting the uploader class in the app/uploaders directory, but am still getting the same error.
Any ideas? Thank you so much for your help.
Can you post the question to the Google group? Make sure to include all your Shrine-related code. Did you remember to load the direct_upload plugin with
Removing this line from config/initializers/shrine.rb, fixes the issue, allowing my tests to run.
I assume the tutorial included that if statement to prevent file uploads during tests, but it prevents all tests from running.
Do you see any downside to removing the if statement?
Here is my repo, I send it because this code is based on your first Shrine video (where you created a post with description and image): https://github.com/carlos99...
Thanks in advance for any help @Chris Oliver
Hi Chris, I followed your series to do the direct upload, which worked for me locally, everything works fine, things are nice. But when i deployed my code to server(eb), direct upload stopped working. No error in the console, it just does not reach uploads.js is what I think, but the normal upload works, I can see the file in S3. As there is no error I cannot figure out what the problem is. Any suggestions? Thank you.
Any ideas on how to go about this without Jquery and/or jQuery file upload?
Hey Chris, do you have a tutorial for implementing Active Storage, Trix and S3 for images for Rails 6? Trying to follow these videos but am running into some issues I cannot find solutions online. Currently the two issues I cannot get past are errors running local server for:
mount ImageUploader::UploadEndpoint => "/images/upload"