All threads / Backing up uploaded files (hundreds of gigs) to S3 using Backup gem without duplication on HD

Ask A Question

Notifications

You’re not receiving notifications from this thread.

Backing up uploaded files (hundreds of gigs) to S3 using Backup gem without duplication on HD

TL asked in Servers
Let's say I have 400GB of uploaded images inside a 800GB VPN machine. So we're confortably using 50% of storage.

Even tough we have our hosting's backup enabled, we want to have an extra backup sitting on S3, just because we can't get enough of backups right?

So, using `backup` gem, we are already sucessfully backing up Mysql and storing it on S3, so now we want to create a second model to backup these files.

The question is: if we create a zipped file with all theses images prior to uploading to S3, we would run out of disk space right?

Is there a way to just upload all 400gb of images to S3 and incrementally update it, on a daily basis, without having to duplicate all files (zipped or not) on hard drive?

Is RSync better suited for this, in which case I should opt out of S3 and buy a node (like in Digital Ocean) ?

What are our options?
Hey TL,

You're looking for the backup gem's Syncers concept: http://backup.github.io/backup/v4/syncers/

You can't use rsync with S3, but you can use this gem's S3 syncer to mirror all the files on disk with S3.

The S3 Syncer just takes some directories that you want to sync and will handle all the uploads. http://backup.github.io/backup/v4/syncer-s3/
Join the discussion

Want to stay up-to-date with Ruby on Rails?

Join 39,376+ developers who get early access to new tutorials, screencasts, articles, and more.

    We care about the protection of your data. Read our Privacy Policy.

    logo Created with Sketch.

    Ruby on Rails tutorials, guides, and screencasts for web developers learning Ruby, Rails, Javascript, Turbolinks, Stimulus.js, Vue.js, and more. Icons by Icons8

    © 2020 GoRails, LLC. All rights reserved.