Skip to main content

Subscribe to GoRails to get access to this episode and all other pro episodes, and new awesome content every month.

Subscribe Now
Only $19/month

Unlimited access. Cancel anytime.

22 Direct File Uploads to S3: Part 2

Episode 145 · October 5, 2016

Setup Shrine to upload files to Amazon S3

File Uploading


Transcripts

Subscribe or login to view the transcript for this episode.

Discussion


Gravatar

Hi! help me please. error loading Image

Aws::S3::Errors::PermanentRedirect
in PhotosController#create

"The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint." ???? instead aws-sdk turn shrine-fog all works

Gravatar

This is a bug related to the configuration of your aws-sdk credentials. I found an issue on the aws-sdk repo that might help. Otherwise it would be good to open an issue on the aws-sdk repo.


Gravatar

i am getting the following error:

Aws::Errors::NoSuchEndpointError at /accounts
Encountered a `SocketError` while attempting to connect to:

https://patrolvault.s3.Oreg...

This is typically the result of an invalid `:region` option or a
poorly formatted `:endpoint` option.

Gravatar

resolved i had to add us-west-2

Gravatar

I get this error too. Does this only work if you use us-west region? I'm currently using Sydney region

Gravatar

For Syndey, you should have that set to "ap-southeast-2" I believe.


Gravatar

Hi Chris, I have 3 problems at this point.

1. At timestamp 11.55 you are showing how to enable the 'make-public' setting. I am trying to follow along, but my bucket is empty whereas yours has items listed for cache and store. How did you get those items into your bucket?

2. I have the same problem as Shawn Wilson below. I have a socket error when I attempt to save my form with the file upload. I'm trying to use the sydney region in my bucket. Is there something specific to the US west coast that needs to be worked around. It seems Shawn needed to use that region to get this working.

3. In the first video where you overview shrine, we added a hidden form field called "f.object.cached_avatar_data". I had to remove that when I was trying to render the form. Maybe that's because something we've just done with the plugins requires a different setting (I'll stay tuned to see if that comes up later.

Thanks a lot for these videos. :)

Gravatar

1. Those folders are automatically created in your bucket when you do a file upload. I was doing testing before the episode to make sure everything works so you might notice some differences. No problem if they aren't there, as they will be made for you.

2. I left a note and included a link in a new comment to the regions you have to specify. Each of them has a name, but then the region piece is the one you need to use here to configure. The one for Syndney looks to be "ap-southeast-2" so make sure you've got that set and that should work.

3. I didn't include the cached attachment data plugin in this episode. It's a separate module you can include: http://shrinerb.com/rdoc/cl...

Hope that helps! :)

Gravatar

Cool, thanks Chris! Now I have an error: The AWS Access Key Id you provided does not exist in our records. I'm using figaro on heroku so not using secrets.yml, but the key is correct. I'm going for a walk before I come back and give it another go.

Gravatar

Getting closer and taking a walk always helps! :)

I'd just double check in your Rails console that the ENV vars are set like you want and then make sure your S3 config is set to use them after that. Possibly one of the two is slightly misconfigured.


Gravatar

Here are the list of AWS regions if you are running into errors with the invalid "region" option: http://docs.aws.amazon.com/...

Gravatar

You can also find your region by looking at the URL when in your S3 bucket. Look for > ?region=us-west-2


Gravatar

You're a legend, Chris. Thanks for taking the time to walk through the S3 configurations while explaining them well -- it is can become pretty daunting by yourself. Cheers.


Gravatar

Chris, I was able to follow this tutorial a number of times and it's great.

As I went to integrate into a test app that I'm building, I ran into an interesting issue that seems to be a common use case / challenge for Shrine as I perused the web for the solution (not a lot of content, I might add... GoRails is pretty much the standard resource aside from the docs).

In my app, I have a Location which has_many Photos. Photo belongs_to Location. When a user creates a location, they upload photos through the location form (using fields_for method).

It seems the issue is that Shrine doesn't play particularly nicely with nested resources? (and I'm a rookie as it pertains to appropriately executing this methodology) I've left code out intentionally as I think it's beyond the scope of your support, but I might be interesting to do a video with Shrine Uploads with the model as a nested resource.

I really do believe that Shrine could well become the defacto standard, but like anything new(ish), there is a learning curve.


Gravatar

Hey so there have been some changes to AWS-SDK and the endpoints, thus resulting is some necessary changes to shrine.rb (if your using the newer versions of AWS-SDK Gem.

in the shrine.rb file to make it work it should appear like so

require "shrine/storage/s3"

s3_options = {
access_key_id: ENV["AWS_ACCESS_KEY_ID"],
secret_access_key: ENV["AWS_SECRET_KEY"],
region: ENV["AWS_REGION"], us-west-2 (or whatever your aws region is from Chris's Post above)
bucket: ENV["AWS_BUCKET"],
endpoint: ENV["AWS_HOST"], (needs to be like this (find these at http://docs.aws.amazon.com/... make sure you include the http or https like so (https://s3-us-west-2.amazon...).
}

Shrine.storages = {
cache: Shrine::Storage::S3.new(prefix: "cache", **s3_options),
store: Shrine::Storage::S3.new(prefix: "store", **s3_options),
}

Shrine.plugin :activerecord
Shrine.plugin :direct_upload
Shrine.plugin :restore_cached_data

Hope this helps if your running into an endpoint issue or permanent redirect error.

Gravatar

Thanks Shawn! 🤘

Gravatar

Ha, this is very timely. Working through some bugs. Thanks!


Gravatar

sorry I am getting this errore when I start to upload the image?
Seahorse::Client::NetworkingError in PhotosController#create

Gravatar

You're likely having a network problem, as the error says. AWS SDK will actually retry networking errors 2 times before failing, so it seems to be something permanent.

This is the list of exceptions that AWS SDK will re-raise as Seahorse::Client::NetworkingError, you can find out which one of those errors was actually raised by catching the NetworkingError, and calling `#cause` on it if you're on Ruby 2.1+, or `#original_error` if you're on lower versions.

Additionally, you can initialize Shrine::Storage::S3 with `http_wire_trace: true` to see what request is AWS SDK making and what response is it receiving, and see if you can spot the cause of the exception.

Gravatar

thank you I will look to spot the cause ;
Failed to open TCP connection to s3.louy-6.amazonaws.com:443 (getaddrinfo: Name or service not known)


Gravatar

That `public: true` hint is actually a hint! I got stuck with images reloading on every Turbolinks visit. I had images set to public on S3 end, but forgot the parameter in the app. However, any hints how I would be able to set generated thumbnail to be public, but the original image to stay private?

Gravatar

You can use the upload_options plugin to pass the :acl option to Shrine::Storage::S3#upload:


plugin :upload_options, store: ->(io, context) do
if context[:version] && context[:version] != :original
{acl: "public-read"}
else
{acl: "private"}
end
end

Gravatar

This is working well but for some uploads - I am getting an error saying 'This XML file does not appear to have any style information associated with it. The document tree is shown below.' -- Its only happening on some URLs and not all. Anyone has suggestions for this?


Gravatar
Chris is it possible to add Elastic Transcoding or an Amazon Lambda function to the direct upload process?

Login or create an account to join the conversation.