Skip to main content

Backing up uploaded files (hundreds of gigs) to S3 using Backup gem without duplication on HD

Servers • Asked by TL
Let's say I have 400GB of uploaded images inside a 800GB VPN machine. So we're confortably using 50% of storage.

Even tough we have our hosting's backup enabled, we want to have an extra backup sitting on S3, just because we can't get enough of backups right?

So, using `backup` gem, we are already sucessfully backing up Mysql and storing it on S3, so now we want to create a second model to backup these files.

The question is: if we create a zipped file with all theses images prior to uploading to S3, we would run out of disk space right?

Is there a way to just upload all 400gb of images to S3 and incrementally update it, on a daily basis, without having to duplicate all files (zipped or not) on hard drive?

Is RSync better suited for this, in which case I should opt out of S3 and buy a node (like in Digital Ocean) ?

What are our options?

Hey TL,

You're looking for the backup gem's Syncers concept: http://backup.github.io/backup/v4/syncers/

You can't use rsync with S3, but you can use this gem's S3 syncer to mirror all the files on disk with S3.

The S3 Syncer just takes some directories that you want to sync and will handle all the uploads. http://backup.github.io/backup/v4/syncer-s3/

Login or Create An Account to join the conversation.

Subscribe to the newsletter

Join 24,647+ developers who get early access to new screencasts, articles, guides, updates, and more.

    By clicking this button, you agree to the GoRails Terms of Service and Privacy Policy.

    More of a social being? We're also on Twitter and YouTube.