All threads / Streaming remote files to S3 πŸ€”

Ask A Question

Notifications

You’re not receiving notifications from this thread.

Streaming remote files to S3 πŸ€”

Daniel Weaver asked in Rails

For each user request I need to save two remote files to AWS S3. I have the remote URLs of each - one image JPG ~100KB and one audio MP3 ~1.5MB.

I know I can do this by downloading each file to temp then uploading to S3 using the aws-sdk gem. However I'm concerned about performance. The files aren't exactly huge but if I'm doing this often enough will it cause an IO bottleneck on my server?

Has anyone done this with streaming? There must be a way to open the remote file as a stream and send that stream direct to S3 without it being saved on my server disk? I assume this would all happen in memory. In fact, is that any less of a bottleneck than saving to temp?

Confused! Help! πŸ˜‚

Join the discussion

Want to stay up-to-date with Ruby on Rails?

Join 38,311+ developers who get early access to new tutorials, screencasts, articles, and more.

    We care about the protection of your data. Read our Privacy Policy.

    logo Created with Sketch.

    Ruby on Rails tutorials, guides, and screencasts for web developers learning Ruby, Rails, Javascript, Turbolinks, Stimulus.js, Vue.js, and more. Icons by Icons8

    © 2020 GoRails, LLC. All rights reserved.