Skip to main content

File Uploads with Refile Discussion

General • Asked by Chris Oliver

Thank you for teaching the new Refile gem. The information shared in the Pro episode for upload progress is worth the subscription to Go Rails. In thirteen minutes, you communicated what has taken hours of work to implement with other file upload gems.

Question: With the refile gem, what is your recommendation for uploading multiple files in a single form, each with progress indicator? There is some discussion at https://github.com/elabs/re.... Perhaps another episode is in order?

Thanks Stan! :)

I believe with multiple files (at least for separate things) you just say

class User
attachment :photo
attachment :resume
end

You may need to modify the Javascript somewhat so that it can tell the difference between the two. That likely is worth doing another episode on, some form of refactoring Javascript to handle this better.

The thing they don't handle right now is uploading multiple files (like User has_many :photos) but that should be coming soon. At least after reading that thread, it seems it should be easy for someone to implement it.


It is Refile good for uploading video files? I need upload video files (not host them on youtube, vimeo, etc.) and maybe do some post production (format convert/generate, add watermark...).

You can upload anything with it. You'll want to check out https://zencoder.com/en/ for transcoding. If you set Refile to upload to S3, zencoder can take it from there. https://github.com/zencoder...


If you are using the built in support for S3, it appears that you set that up in the initializer for refile:

# config/initializers/refile.rb

But what if you only want to use S3 in production, and the filesystem in dev and test? Is there a way to move this config to the environment level? For example, with Paperclip, the config to use AWS is here when you only want it in production:

# config/production.rb

Chris, you can just setup an environment conditional in your initializer:


if Rails.env.production?
# Use S3 Backend
else
# Use FileSystem Backend
end

Chris,
Rails 4.1 comes with pretty secrets feature so you can easily use it like this:


# config/initializers/refile.rb

aws = {
access_key_id: Rails.application.secrets.aws_api_key,
secret_access_key: Rails.application.secrets.aws_secret,
bucket: Rails.application.secrets.aws_bucket,
}

and then set different config options for different environments:


# config/secrets.yml
development:
aws_api_key: ...
aws_secret: ...
aws_bucket: ...

production:
aws_api_key: ...
aws_secret: ...
aws_bucket: ...

Looking through the documentation it does not seem like there is any way to limit or throttle the reformatting/re-streaming of assets. I understand you say use CDN, but I do not see any method of stopping malicious users of hard spinning the asset urls with different pixel sizes for example to DoS the server... Any ideas or am I right in seeing this as a baseline flaw of this gem?

Definitely a worthy discussion to be had with the author of the gem.


very cool gem indeed. Just implemented it to my new Rals app. Thanks for the very instructive video Chris!
I also need to implement document uploads (.pdf, .doc, .txt, etc ...) Can this also be done with Refile?

greetings,

Anthony

Yep, shouldn't be a problem. You'll just skip the image processing parts.

ok, thanks!

just tried it out, and it works. I'm just wondering if there is a way to print the name of the document.

found a solution to print the filename of an uploaded file on this page:

https://github.com/elabs/re...

just generate a migration to add a filename field to the table, for example:

add_column :news_articles, :document_filename, :string

now you can read out filenames like this:

<%= link_to @news_article.document_filename, attachment_url(@news_article, :document) %>


Can you use this to upload a CSV file and have each row render as a new object? What do you think is the best way to do this?

CSV upload is definitely a topic I need to cover soon. I usually don't save the CSV files so I traditionally just use a file_field and let the file get deleted after the request is done in most cases.

Refile could work for storage of the CSVs though and then the model could access the file and parse it as necessary if you want to store the CSVs permanently.

Thanks for the response. I'm going through this rails cast now http://railscasts.com/episo... to see if it works but I think I'm going to get a nokogiri error. I've had it before but going to see if I can debug. Would love to see this covered on GoRails though! Would be great to understand the parsing aspect.

I'll let you know how it goes for now.

I will definitely record a screencast on this soon! Several people have been asking about it.

Also, check out this blog post that Patrick McKenzie just posted about his advanced CSV upload that he built: http://www.kalzumeus.com/20...

I just read through the post. Seems pretty daunting.. Have been trouble shooting the rails cast. it appears he's able to upload csv files by about 3 min 12seconds in. I've got everything rendering correctly but am getting this error when I submit the file for upload. invalid byte sequence in UTF-8

I've been googling around but haven't found the right solution yet. Any ideas? I'm using ruby version 2.1.2
invalid byte sequence in UTF-8..

Yeah it's very picky. You can google it and lots of people have similar problems. http://stackoverflow.com/qu...

Ok yeah I found that one, went with the easy option of copy and pasting to google spreadsheets and then downloading it as a CSV and it worked. Now running into a pundit::NotDefinedError in SamplesController#import unable to find policy NilClassPolicy for. I'm a little confused because but I think I'll figure this one out.

Thanks again for the help.

That'll work. :) I've just exported as a plain CSV similarly because it's hard to tell the encoding of the file sometimes.

As for the Pundit issue, sounds like the variable you're using Pundit against is nil (which is why it looked up NilClassPolicy). Double check that variable is getting set correctly and that should fix it.

Thanks that pointed me in the right direction.. needed to define the variable @samples = Sample.all.. that got it working.

This worked great! Thanks!


Can you use this gem to upload multiple files at the same time?

Not at the same exact time just yet, but that's on his todo list to add as a feature. https://github.com/elabs/re...

would be nice to have that feature. are there are currently available gems that support multiple file uploads (ones you would recommend)?

I think Carrierwave does and I use it for almost everything right now.


Has anyone made a rake task to populate the database with images? I went through the documentation and didn't see how this could be done. Thanks!


I get this error when trying to use S3 backend:
/home/gabriel/.rvm/gems/ruby-2.1.5/gems/refile-0.5.3/lib/refile/backend/s3.rb:37:in `initialize': uninitialized constant Refile::Backend::S3::AWS (NameError)

Any ideas?

This is because Refile uses the 1.x version of the aws gem. You'll need to update your gemfile like this:

gem "aws-sdk", '< 2'

Now you'll be using the same version as Refile. The 2.x version of the aws-sdk gem uses Aws for name spacing instead of AWS so that you can use both versions of the gem in the same application.

is it still using aws-sdk < 2 now?
after 4 months.


I want to try and use Refile in my rails API app. Can I just attach a base64 encoded image in the post body?

I'm curious about this as well, let me know if you find a solution!


Great video! Still deciding between carrierwave and refile tho since it's still new and carrierwave's wiki is filled with howtos. Btw, I think you missed out on the uploading via url part. :)

I did forget to include that. It's an awesome little feature along with deleting uploads. It definitely still is pretty new and you can do everything with Carrierwave no problem.

Thanks Chris! Look forward to using it!


Can I check de md5, to prevent repeated uploads?

That's a great question. You should ask in an Issue on the github repo for Refile.


guys, how to add more than one image?

The best way to do it right now is to to upload each file to an associated has_many model.


Hi... Nice screencast! I'm getting a "no find image error" after deploy, using Capistrano... Seems like the app can't find the image after assets precompile... should i set up a directory for shared_children, or something like that? Thanks.


How would you do authorization on uploaded files with Pundit? I have uploaded files which should be restricted to the user that uploaded them, but anyone with the url that attachment_url generates can access the file. Since Refile uses the sinatra app, there's no controller for me to call Pundit's authorize method in.

That's a great question and I don't particularly know the answer to that. I'm sure you can open an issue on Github and get a response.

Thanks for the reply. I've opened an issue on the repo. Direct link https://github.com/refile/r...


I've been using Dragonfly gem (https://github.com/markevan... for a long time and I find it very similar to Refile. Has somebody compare them to have a better idea if it is worth adopting Refile?

Looks similar but I've never used Dragonfly. Unless you need some other feature, it probably isn't worth switching yet.


The problem I had with Carrierwave was that when I used my phone to upload an image, I couldn't use my Gallery to upload an image (the gallery did not pop up as one of the ways to upload the image). Instead, I had to find my images in different folders, so I had to go to My files folder-> DCIM -> select the image, which is not a very good user experience (I was using an Android by the way).

I was wondering if I use Refile on my phone, will I be able to select images from my gallery directly? Or, would it be the same UX as Carrierwave where I have to navigate into my folder (i.e. DCIM).

Unfortunately that wouldn't make a difference. That feature is controlled entirely by the browser on the device. It will work differently depending on the OS version and browser.

One thing you can try is to play with the accept and capture attributes on the file field itself and see if it will trigger different things on your Android browser. I can't find great documentation on it though, but check this out to get started. http://stackoverflow.com/qu...


Would you recommend Refile for users - posting videos? Or, would Zencoder work well with Refile??

Any uploader that submits the file to S3 will work with Zencoder. They expect that your original video file is already available on S3 so they don't have to worry about which uploader you use. You're free to go with Refile, Carrierwave, Paperclip, etc.

Thank you for the explanation. If you want to create a site like Udemy, where you allow users(lecturers) to upload multiple number of videos to sell, do you know any tutorial or resources that you recommend? Thank you in advance!

You would create a new "Lesson" record for each each video and have one video upload per Lesson. Do the same as this basically and you're all set. It would be just like creating multiple Film records for the same Actor.


hi there! love these tutorials! I would love to get your opinion about Transloadit !!! ???? It seems super awesome but I was unable to populate mysql with URLs after the robots did their jobs and packaged up JSON Assemblies on the Transloadit API.....just wondering if you think this service is worth using, etc. THANKS!

This looks super cool. I haven't used it before, so I'm not quite sure how they return the urls afterwards. I have previously seen a lot of people using https://www.filepicker.com/ in the past.

I think they're definitely great and give your users a lot of flexibility as well saving you from a lot of time managing the files and processing them. Would definitely recommend it if file uploading is something you don't really need to manage or customize too deeply.

- thanks again for the great tutorials! much needed! I will let my fellow rails friends know about this when I attend an upcoming Rails meetup!

Ah it looks like it's sending webhooks back. Those are always a bit tough to test in development because your dev server isn't publicly accessible on the internet.

Thanks for sharing GoRails! :)

gotcha! thanks!! go GoRails!


Thank you for your videos!! I want to upload a image from a external application. How I can do that? I didn't find documentation about that.

Lots of different options here. Are you talking about importing files from somewhere else like through a Ruby script or something? Where are the images in the external application? Does it have an API?

The image files (e.g. jpg) will send by android. The idea is that the rails app receive the base64 encoded image file as a HTTP POST request and save it with Refile.

Sending files via a POST won't be any different than a normal upload form on the webpage. The browser submits forms with a POST so if your Android app or mobile browser does the same thing, nothing would change. You just need to make sure the file is sent over with the same field name as it has in the browser so it can attach to the right model and attribute.

please can you make a deployment to heroku with aws video with refile gem

I don't use this anymore, but I would recommend just following the readme for the S3 example they have: https://github.com/refile/r...

Their Github would also be the best place to ask for questions on S3 as well.


Hi, thanks for this solution.
I new in Rails world.

Can i use this gem for upload large file (4gb+) on my locale-storage, without S3?
And in upload process "uploading file" writing in RAM memory or on hdd in temp?
(Because i have problem with simple upload form, first file write in RAM mem, and after in tmp file.)

Thanks.

That's a great question. I think Refile might not be ideal, but you can always ask the author on the Github issue tracker.

Another useful resource is: http://bclennox.com/extreme...

Thanks, Chris.


Wondering if refile supports renaming files, when directly uploading to S3


Everything seems to work except after i follow the S3 steps, it still uploads to localhost.. any thoughts?

I am having the same problem as your initial one. Did you figure it out and is that why you posted a link to your repo? Could you elaborate on what you did to correct it?

Check out this section in the Readme which should get you sorted out. https://github.com/refile/r...

I moved back to paperclip..

I thought that this was happening to me too, but when I actually thought about it I saw it was working. Even if you upload to a 3rd party host the rack app for refile will download that image, process it, and still give it a local URL. The true test to see if it is uploading to S3 is to look in your bucket to see if the images are being stored there. Turns out mine was working all along.


Is it possible to do image preview before we actually save the image?

Yes you can. This isn't specific to Refile, but just some regular Javascript hitting the browser File api. http://jsfiddle.net/LvsYc/


Looking for a way users can upload videos - there isnt much out there that clearly explains


Chris, I started using refile with more models (product model for avatars and message model for file sending) and realized all the things go to the same folder on S3. I also wanna upload files for the product model but it would be messy if everything was in the same folder (product avatar, product files and message files). I guess I will change the product avatar uploader to use carrierwave, but still the product files and message files will go to the same S3 folder which is not the best I guess. As I see in the docs/issues on github there is no way to put them into different folders. Did I miss something and I can put them into different buckets somehow? If I can't how I can make sure everything will be fine down the road?

I'm not sure why this isn't the default. It's a really important thing for managing a large set of files, so you'd think this was already in there.

I see a prefix option for the library that allows you to do this, but I don't see much information on how you'd set that dynamically.

This is the prefix option I'm talking about. You would need this to be set dynamically each time a different file type was uploaded which I don't know is very easy to customize.

Refile.store = Refile::S3.new(prefix: "store", **aws)

It's probably worth opening a Github Issue on the Refile to ask about this and see if there's currently a way to do that or if a patch needs to be made to support it.

I've checked it out already, but it's a bit high level, so I don't understand it. https://github.com/refile/r..., https://github.com/refile/r... I just thought you have also run into this problem since as far as I know you are using refile gem. Chris after writing this comment I found an answer on stackoverflow which might be useful. If so I will circle back with the solution. I guess it's important for everybody here.


What about paperclip ?


There appears to be a problem with refile gem when using nested forms? I have a profiles form nested inside a users form, but when I upload the image, nothing happens. No errors are given, but profile images are not uploaded either. Anyone noticed this?


Thanks for this episode Chris. I implemented this and it works on my development server just fine. But when I deploy to production (Digital Ocean Ubuntu 16 with Capistrano and Nginx), i see the images upload just fine to s3 bucket, but unable to retrieve them. My nginx error log says:
"App 494 stdout: Refile::App: Error -> ImageMagick/GraphicsMagick is not installed"

but I have both imagemagick and graphicmagick installed on my server. What could be the problem here?

It's possible that Refile just isn't able to find the executable for imagemagick. It's looking for the "identify" command so you'll want to make sure that's in one of the folders in the PATH for the same user that Nginx is running as. I'd poke around at that and see if you can figure out why it might not be able to find the identify executable. If you installed imagemagick with apt-get then it should be available, but if you compiled it or something else, then it's probably just not in the PATH.

Thanks Chris, but Yes I installed with apt-get and when I do "which identify" it shows /usr/bin/identify. I also have the /usr/bin in my PATH, but still getting the same error. I used your other tutorial on "Deploying to Digital Ocean" to do the deployment...think there's anything in there that could cause this?

Hmm, sounds like you've got everything correct! I have used the exact same setup before and it works fine. I'll have to think on it and see what comes to mind.

OK Chris I got this working...maybe someone else is facing the issue but this is what I realized...RoR was not picking up environment variables, so the PATH which indicates the location of Identify executable was not being read. What I did:

1- In my ~/.profile I made sure it was loading both .bashrc and .bash_profile (initially it was only loading .bashrc)

2- I then created a ~/.bash_profile, where I put in my environment variables:
eg
export PATH=$PATH
3- Restarted the server: sudo service nginx restart

Bammm...with this my images showed up and even my SMTP settings which were also not working are being picked up now. Thanks for your suggestions.

Glad you got that working! Certainly seems weird that it wasn't picking up any environment variables! I don't think I've ever run into that, but it makes sense why you were having that problem then.


Can any one please help. i need to crop the image before uploading the image file to refile
couldnt find any solution with refile to crop the image.
Thanks


please can you make a deployment to heroku with aws video with refile gem


If anyone runs into dependency issues with Rails 5 I found a good solution here: https://github.com/refile/r...

Thank you so many times over. This was a life saver!


Login or Create An Account to join the conversation.

Subscribe to the newsletter

Join 24,647+ developers who get early access to new screencasts, articles, guides, updates, and more.

    By clicking this button, you agree to the GoRails Terms of Service and Privacy Policy.

    More of a social being? We're also on Twitter and YouTube.