Rails AWS S3: Downloading File and Storing in Temp Folder

Question

I'm trying to download a file from my AWS bucket and store it in my temp folder.

The below code returns this error: OpenURI::HTTPError (301 Moved Permanently (Invalid Location URI)):

@filename is the filename within the bucket, including the extension, as a string.

S3 =  AWS::S3.new(
  :access_key_id => ENV["AWS_ACCESS_KEY_ID"],
  :secret_access_key => ENV["AWS_SECRET_ACCESS_KEY"]
)

BUCKET = S3.buckets["name_of_bucket"]

File.open(Rails.root.join("tmp",@filename), "wb") do |file|
 file.write open(BUCKET.objects[@filename].url_for(:read)).read
end

Here's what I've found out so far:

puts BUCKET.objects['name_of_bucket']
puts BUCKET.objects['name_of_bucket'].url_for(:read)

The first BUCKET call returns the correct object, the second returns this:

https://ekohotstorage.s3-us-west-2.amazonaws.com/location_info

When you go to that url this is what is returned

The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

This (http://www.sitefinity.com/developer-network/forums/set-up-installation/amazon-s3---must-be-addressed-using-the-specified-endpoint) states "there's a limitation in S3, whereby if you create a bucket that is not "US Standard", you cannot use path-style syntax in the bucket address."

However, I've checked this bucket and it's def. region US Standard.

Edit:

I'm foolish; I had read that the AWS default was US Standard. But west isn't standard, east is so I guess it does not default to US Standard.

After fixing that foolishness with

S3 =  AWS::S3.new(
  :access_key_id => ENV["AWS_ACCESS_KEY_ID"],
  :secret_access_key => ENV["AWS_SECRET_ACCESS_KEY"],
  :region => "us-east-1"
)

I'm now receiving this error:

OpenURI::HTTPError (404 Not Found)

However, the url created from the below code does lead to the sound file.

BUCKET.objects['name_of_bucket'].url_for(:read)

So the issue must lie with

File.open(Rails.root.join("tmp",@filename), "wb") do |file|
 file.write open(BUCKET.objects[@filename].url_for(:read)).read
end

Thoughts?

Answer 1

There appear to be two things going on here. First, the US standard region is actually either Virginia or Oregon. You just need to realize that you are being redirected because Amazon has chosen Oregon for your bucket. The URL explaining the regions is currently here:

http://docs.aws.amazon.com/AmazonS3/latest/dev/LocationSelection.html

The second thing going on is redirection. You are being redirected to Oregon when you connect, because the typical S3 endpoint you will see is the closest one by latency DNS. When you later hard code in Virginia, you don't see your bucket as it is in Oregon.

Amazon has some documentation here regarding how S3 will redirect you and how to handle this.

http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTRedirect.html

Since you are using a toolkit and rather than an API, I didn't see anything specifically in the ruby forge or Amazon documentation that would tell why the redirect is not followed. It looks as though other people with the AWS ruby SDK had this problem last year. It may be fixed in the newer versions.

The documentation that I am referring to is here:

http://amazon.rubyforge.org/doc/

Amazon appears to have newer documentation here:

http://docs.aws.amazon.com/AWSRubySDK/latest/

Another thing is that I saw complaints that the redirect URL was malformed. While I was not able to duplicate that behavior, I would try to send the request to the Oregon endpoint to see if this resolves your issue.