Reducing Transfer costs with AWS S3
Have had my website up for about a month and a half now and have run into an issue where the data transfer cost for AWS S3 is crazy high (around $200/month). The website is a recipe sharing website so there are a lot of pictures being uploaded and loaded when people search and view the website but this still seems excessively high. Im using default configuration for active storage with some resizing for the different views that the images are display in.
Is there a setting or something that I mightve overlooked to reduce the costs or is this pretty standard to expect for costs? For reference im getting about 3k visitors a day.
Ive looked into trying to setup amazon cloudfront but havent been able to figure out how to set that up with AWS S3 and active storage so the active storage link uses the cloudfront url instead of the S3 url. Would that significantly decrease my costs?
I have and its the data transfer out for the S3 thats breaking past the free tier and costing all the money. Specifically its all under the bandwidth cost. Apparently Im transfering 2TB worth pictures out a month with about 3k visitors a day. Its $0.09/GB to transfer out.
Im using digital ocean for my servers, so I was considering transfering to their spaces, but not entirely sure how I would go about migrating over and transfering all the current images and active storage references to the new storage. Or if that would fix the issue totally. I know digital ocean is cheaper for storage though so definitely looking into that.
That's quite a lot. You probably are spending unncessary money doing this anyways if you're creating different sizes of images since you'll be downloading them from S3 after upload, creating your sized versions and uploading those back. That probably is extra cost. Spaces should be cheaper all around since it's not going to be going across the internet for that. Plus, Spaces is just cheaper all around.
$5/mo for Spaces includes 1TB of outbound traffic, so that may help by itself.
I don't know Cloudfront's pricing, but it may save some or add to the cost. It's going to be caching your files around the world with Cloudfront, so it's faster. If they're still charging you for bandwidth out, then may just be more expensive to use that.
You probably want to do some checking to make sure you're serving up the smallest images as much as you can and almost never serving the original upload. May want to add image compression as a step in your image processing as well. Possibly using Cloudflare's free CDN would help save some of the costs too if images can be cached in their CDN.
ActiveStorage isn't super great for this since all the requests for images go through Rails rather than being served directly from the S3 url. Rails 6.1 is going to have some official ways to link to files publicly, but I know there are a few hacks you can do for now to do that too.
Transfering to Spaces should be as simple as transferring all the files to Spaces and then updating the ActiveStorage endpoint to use Spaces. They're S3 compatible so it would be pretty easy, but you'd want to make sure you didn't miss any images that might get uploaded during the transfer. That might be the harder part if you don't want to pause new uploads temporarily.
Ok, so I'll definitely start looking into switching over to DO Spaces then. I saw digital ocean has a guide for transfering specifically from S3, so will need to check that out and see if I can get swapped over. Not too worried about new images, as right now Im the main person uploading recipes so I just wont upload anything. For the endpoint update, dont need to do any changes with the database info correct? Just update my config files to use the new digitalocean spaces endpoint and it pulls from that and there isnt anything saved in the database expect the filename correct?
I currently am using the ActiveStorage variant features to resize the images to the size I need and using those for display. I have also come across this guide for using DO and cloudfare CDN with activestorage. Looks like they override the public url function to use the CDN domain. So Ill give that a try also
Well was able to easily swap over to DigitalOcean Spaces and will see how that goes on cost. And will start looking into using cloudflares free CDN service. Thanks for the advice and info, youre awesome with helping
That was fast! Also might want to look into Spaces' CDN. https://blog.digitalocean.com/spaces-now-includes-cdn/
Curious to hear how things go and if it helps keep costs down over the next few weeks.