Paperclip is reporting access_denied when trying to upload files to S3

Question

UPDATE: It works when I remove the explicit deny block from the bucket policy, but I need that in there to prevent people outside the site (including bots) from accessing the content.

--

I'm trying to figure out a way to set access control on my S3 content such that:

1. Users with HTTP referrer mydomain.com can view the files
2. Paperclip gem can upload files
3. Google bots can't crawl and index the files

Viewing existing files from the site works fine, but uploading a file gives this error in the console:

[AWS S3 403 0.094338 0 retries] put_object(:acl=>:public_read,
:bucket_name=>"mybucket",:content_length=>879394,
:content_type=>"image/jpeg",:data=>Paperclip::FileAdapter: Chrysanthemum.jpg,
:key=>"ckeditor_assets/pictures/6/original_chrysanthemum.jpg",
:referer=>"mydomain.com/")
AWS::S3::Errors::AccessDenied Access Denied

Here's the bucket policy I have:

{
"Version": "2012-10-17",
"Id": "http referer policy example",
"Statement": [
    {
        "Sid": "Allow get requests referred by mydomain.com",
        "Effect": "Allow",
        "Principal": "*",
        "Action": [
            "s3:GetObjectVersion",
            "s3:DeleteObject",
            "s3:DeleteObjectVersion",
            "s3:GetObject",
            "s3:PutObject"
        ],
        "Resource": "arn:aws:s3:::mybucket/*",
        "Condition": {
            "StringLike": {
                "aws:Referer": "mydomain.com/*"
            }
        }
    },
    {
        "Sid": "Explicit deny to ensure requests are allowed only from specific referer.",
        "Effect": "Deny",
        "Principal": "*",
        "Action": "s3:*",
        "Resource": "arn:aws:s3:::mybucket/*",
        "Condition": {
            "StringNotLike": {
                "aws:Referer": "mydomain.com/*"
            }
        }
    }
]

}

The error message is odd, because I explicitly set the referrer to mydomain.com in Paperclip settings:

production.rb:

:s3_headers => {
  'Referer' => 'mydomain.com'
}

And Paperclip does indeed use it, as shown on the second to last line of the error message.

So why does it still give Access Denied?

Answer 1

After fiddling with it for hours, I revised my approach in light of the original three requirements I listed. I'm now explicitly denying only GetObject (as opposed to everything via "*"), and I also placed a robots.txt file at the root of my bucket and made it public. Therefore:

1. Users can access bucket content only when my site is the referer (maybe this header can be spoofed, but I'm not too worried at this point). I tested this by copying a resource's link and emailing it to myself, and opening it from within the email. I got access denied, which confirmed that it cannot be hotlinked on other sites.
2. Paperclip can upload and delete files
3. Google can't index the bucket contents (hopefully robots.txt will be sufficient)

My final bucket policy for those who arrive at this via Google in the future:

{
"Version": "2012-10-17",
"Id": "http referer policy example",
"Statement": [
    {
        "Sid": "Allow get requests referred by mydomain.com",
        "Effect": "Allow",
        "Principal": "*",
        "Action": "s3:*",
        "Resource": "arn:aws:s3:::mybucket/*",
        "Condition": {
            "StringLike": {
                "aws:Referer": "https://mydomain.com/*"
            }
        }
    },
    {
        "Sid": "Explicit deny to ensure requests are allowed only from specific referer.",
        "Effect": "Deny",
        "Principal": "*",
        "Action": "s3:GetObject*",
        "Resource": "arn:aws:s3:::mybucket/*",
        "Condition": {
            "StringNotLike": {
                "aws:Referer": "https://mydomain.com/*"
            }
        }
    }
  ]
}

I do have a theory on why it didn't work. I was reading the S3 documentation on Specifying Conditions in a Policy, and I noticed this warning:

Important: Not all conditions make sense for all actions. For example, it makes sense to include an s3:LocationConstraint condition on a policy that grants the s3:PutBucket Amazon S3 permission, but not for the s3:GetObject permission. S3 can test for semantic errors of this type that involve Amazon S3–specific conditions. However, if you are creating a policy for an IAM user and you include a semantically invalid S3 condition, no error is reported, because IAM cannot validate S3 conditions.

So maybe the Referer condition did not make sense for the PutObject action. I figured I'd include this in case someone decides to pick this issue up from here and pursue it to the end. :)