Streaming remote files to S3 🤔
For each user request I need to save two remote files to AWS S3. I have the remote URLs of each - one image JPG ~100KB and one audio MP3 ~1.5MB.
I know I can do this by downloading each file to
temp then uploading to S3 using the
aws-sdk gem. However I'm concerned about performance. The files aren't exactly huge but if I'm doing this often enough will it cause an IO bottleneck on my server?
Has anyone done this with streaming? There must be a way to open the remote file as a stream and send that stream direct to S3 without it being saved on my server disk? I assume this would all happen in memory. In fact, is that any less of a bottleneck than saving to
Confused! Help! 😂