Skip to main content

Direct File Uploads to S3: Part 2 Discussion

General • Asked by Chris Oliver

Hi! help me please. error loading Image

Aws::S3::Errors::PermanentRedirect
in PhotosController#create

"The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint." ???? instead aws-sdk turn shrine-fog all works

This is a bug related to the configuration of your aws-sdk credentials. I found an issue on the aws-sdk repo that might help. Otherwise it would be good to open an issue on the aws-sdk repo.


i am getting the following error:

Aws::Errors::NoSuchEndpointError at /accounts
Encountered a `SocketError` while attempting to connect to:

https://patrolvault.s3.Oreg...

This is typically the result of an invalid `:region` option or a
poorly formatted `:endpoint` option.

resolved i had to add us-west-2

I get this error too. Does this only work if you use us-west region? I'm currently using Sydney region

For Syndey, you should have that set to "ap-southeast-2" I believe.


Hi Chris, I have 3 problems at this point.

1. At timestamp 11.55 you are showing how to enable the 'make-public' setting. I am trying to follow along, but my bucket is empty whereas yours has items listed for cache and store. How did you get those items into your bucket?

2. I have the same problem as Shawn Wilson below. I have a socket error when I attempt to save my form with the file upload. I'm trying to use the sydney region in my bucket. Is there something specific to the US west coast that needs to be worked around. It seems Shawn needed to use that region to get this working.

3. In the first video where you overview shrine, we added a hidden form field called "f.object.cached_avatar_data". I had to remove that when I was trying to render the form. Maybe that's because something we've just done with the plugins requires a different setting (I'll stay tuned to see if that comes up later.

Thanks a lot for these videos. :)

1. Those folders are automatically created in your bucket when you do a file upload. I was doing testing before the episode to make sure everything works so you might notice some differences. No problem if they aren't there, as they will be made for you.

2. I left a note and included a link in a new comment to the regions you have to specify. Each of them has a name, but then the region piece is the one you need to use here to configure. The one for Syndney looks to be "ap-southeast-2" so make sure you've got that set and that should work.

3. I didn't include the cached attachment data plugin in this episode. It's a separate module you can include: http://shrinerb.com/rdoc/cl...

Hope that helps! :)

Cool, thanks Chris! Now I have an error: The AWS Access Key Id you provided does not exist in our records. I'm using figaro on heroku so not using secrets.yml, but the key is correct. I'm going for a walk before I come back and give it another go.

Getting closer and taking a walk always helps! :)

I'd just double check in your Rails console that the ENV vars are set like you want and then make sure your S3 config is set to use them after that. Possibly one of the two is slightly misconfigured.


Here are the list of AWS regions if you are running into errors with the invalid "region" option: http://docs.aws.amazon.com/...

You can also find your region by looking at the URL when in your S3 bucket. Look for > ?region=us-west-2


You're a legend, Chris. Thanks for taking the time to walk through the S3 configurations while explaining them well -- it is can become pretty daunting by yourself. Cheers.


Chris, I was able to follow this tutorial a number of times and it's great.

As I went to integrate into a test app that I'm building, I ran into an interesting issue that seems to be a common use case / challenge for Shrine as I perused the web for the solution (not a lot of content, I might add... GoRails is pretty much the standard resource aside from the docs).

In my app, I have a Location which has_many Photos. Photo belongs_to Location. When a user creates a location, they upload photos through the location form (using fields_for method).

It seems the issue is that Shrine doesn't play particularly nicely with nested resources? (and I'm a rookie as it pertains to appropriately executing this methodology) I've left code out intentionally as I think it's beyond the scope of your support, but I might be interesting to do a video with Shrine Uploads with the model as a nested resource.

I really do believe that Shrine could well become the defacto standard, but like anything new(ish), there is a learning curve.


Hey so there have been some changes to AWS-SDK and the endpoints, thus resulting is some necessary changes to shrine.rb (if your using the newer versions of AWS-SDK Gem.

in the shrine.rb file to make it work it should appear like so

require "shrine/storage/s3"

s3_options = {
access_key_id: ENV["AWS_ACCESS_KEY_ID"],
secret_access_key: ENV["AWS_SECRET_KEY"],
region: ENV["AWS_REGION"], us-west-2 (or whatever your aws region is from Chris's Post above)
bucket: ENV["AWS_BUCKET"],
endpoint: ENV["AWS_HOST"], (needs to be like this (find these at http://docs.aws.amazon.com/... make sure you include the http or https like so (https://s3-us-west-2.amazon...).
}

Shrine.storages = {
cache: Shrine::Storage::S3.new(prefix: "cache", **s3_options),
store: Shrine::Storage::S3.new(prefix: "store", **s3_options),
}

Shrine.plugin :activerecord
Shrine.plugin :direct_upload
Shrine.plugin :restore_cached_data

Hope this helps if your running into an endpoint issue or permanent redirect error.

Thanks Shawn! 🤘

Ha, this is very timely. Working through some bugs. Thanks!


sorry I am getting this errore when I start to upload the image?
Seahorse::Client::NetworkingError in PhotosController#create

You're likely having a network problem, as the error says. AWS SDK will actually retry networking errors 2 times before failing, so it seems to be something permanent.

This is the list of exceptions that AWS SDK will re-raise as Seahorse::Client::NetworkingError, you can find out which one of those errors was actually raised by catching the NetworkingError, and calling `#cause` on it if you're on Ruby 2.1+, or `#original_error` if you're on lower versions.

Additionally, you can initialize Shrine::Storage::S3 with `http_wire_trace: true` to see what request is AWS SDK making and what response is it receiving, and see if you can spot the cause of the exception.

thank you I will look to spot the cause ;
Failed to open TCP connection to s3.louy-6.amazonaws.com:443 (getaddrinfo: Name or service not known)


That `public: true` hint is actually a hint! I got stuck with images reloading on every Turbolinks visit. I had images set to public on S3 end, but forgot the parameter in the app. However, any hints how I would be able to set generated thumbnail to be public, but the original image to stay private?

You can use the upload_options plugin to pass the :acl option to Shrine::Storage::S3#upload:


plugin :upload_options, store: ->(io, context) do
if context[:version] && context[:version] != :original
{acl: "public-read"}
else
{acl: "private"}
end
end

This is working well but for some uploads - I am getting an error saying 'This XML file does not appear to have any style information associated with it. The document tree is shown below.' -- Its only happening on some URLs and not all. Anyone has suggestions for this?


Chris is it possible to add Elastic Transcoding or an Amazon Lambda function to the direct upload process?

How can I pass the .url(public: true) to a link that is being dynamicly added to the page?


Wait I don't understand why you have to tell Shrine to be able to read the file, if the file system on the AWS S3 side has already been public. I don't quite understand that.


This gem is so tricky... Actually Shrine.plugin :direct_upload is deprecated and we would use signed plugin. But, this is not just the problem, aws console seems like if it would be updated every week and obviously this tutorial is outdated. I had a lot of problems with aws gem and console, and if you have a problem with permissions, maybe this would fix them

To manage the bucket, you need to grant the s3:* action for the bucket you designate. AWS policies designate resources by their Amazon Resource Name, or ARN and for S3 buckets, they look like: arn:aws:s3:::bucket-name-here. So to grant your user full access to your bucket, you’d paste the policy:

{
  "Statement": [
    {
      "Action": "s3:*",
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::4ormat-knowledge-base",
        "arn:aws:s3:::4ormat-knowledge-base/*"
      ]
    }
  ]
}

Now, you would think that this would be enough to enable the user to use the S3 console to manage the bucket, but you’d be wrong. Turns out the user needs one more permission to do the initial listing of the buckets in order to be able to select a bucket, and its called s3:ListAllMyBuckets. You need to add that permissions too, and it looks like this:

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListAllMyBuckets",
      "Resource": "arn:aws:s3:::*"
    }
  ]
}

http://mikeferrier.com/2011/10/27/granting-access-to-a-single-s3-bucket-using-amazon-iam/


Hey, Chris! According to the Shrine plugins page, "direct_upload is [OBSOLETE]" and they want you to "Use upload_endpoint and presign_endpoint plugins instead." I have been following along with this video and having problems getting it to work. Specifically, I get to about 9:45 and then my rails server won't start, nor, I discovered, will the rails console.

Going back through the video again, I noticed that the roda gem is for presigned endpoints, which is exactly what one of the newer Shrine plugins is supposed to be for. Could this be the problem?

It doesn't look like Roda is required anymore. You just need to include those two plugins instead of the direct_upload one.

Aside from those changes, it's probably much different. You just want your presign Javascript request to hit the presign URL and then the upload JS request needs to hit the upload_endpoint URL.


This episode is super helpful.


Hey there. Running into an issue when I first fire up the rails server. An error " A rack application must be specified (ArgumentError)" pops up. I checked my routes and everything looks to be the same as in the video.

 Rails.application.routes.draw do
   mount ImageUploader::UploadEndpoint => "/images/upload"

  resources :photos

  root to: "photos#index"
end

Is there something else I'm missing?

Update: The issue seems to be with the way the image uploader is mounted. I remove that line and the app fires up just fine, but of course can't upload an image. Is there a better way to write this or does the issue lie with something like the shrine.rb initializer?

Thanks to a kind soul on another site, I was informed that

mount ImageUploader::UploadEndpoint => "/images/upload"

must now be changed to:

mount ImageUploader.upload_endpoint(:cache), at: "/images/upload"

Hopefully this helps others with the same issue.


Login or Create An Account to join the conversation.

Subscribe to the newsletter

Join 27,623+ developers who get early access to new screencasts, articles, guides, updates, and more.

    By clicking this button, you agree to the GoRails Terms of Service and Privacy Policy.

    More of a social being? We're also on Twitter and YouTube.