Amazon S3 Signed URLs to change content type

Question

We have a large number of videos/audo/media hosted on a custom domain on S3 and have created a set of functions in order to sign the URLs and allow them to be both streamable and downloadable. The problem is that the signed URL of course never works. The error is:

The request signature we calculated does not match the signature you provided. Check your key and signing method.

Of course if we take the bytecode returned from this page and input it into the Amazon S3 Signature Tester and grab the bytecode from there it works just fine. Even if the string to sign from our function as well as the decoded byte code in the Signature Tester are identical, it never works.

It's called via a small block of PHP code:

$headers = createS3parameters($expiry, $file_type);
$request = preg_replace("/^.*?:\/\/.*\//", "/", $bucketurl);
$signature = signRequest($request, $expiry, $s3secret, $headers, "GET", $file_type);
$signed_request = "$bucketurl?AWSAccessKeyId=$s3key&Expires=$expiry&$headers&Signature=$signature";

This is the function which actually signs it.

function signRequest($request, $expiration, $s3secret, $headers = '', $type = 'GET', $content_type = 'default')
{
    if ($expiration == 0 || $expiration == null)
    {
        $expiration = time() + 315576000; // 10 years (never)
    }

    if (strcmp($content_type, 'default') == 0)
    {
        $content_type = "";
    }

    // S3 tester spits out this format
    /*$string = "$type\n".
              "\n\n".
              "$expiration\n".
              "/mybucket$request?$headers";*/

    $string = "$type\n".
              "\n".
              "$content_type\n".
              "$expiration\n".
              "$headers\n".
              "$request";


    // must be in UTF8 format
    $string = utf8_encode(trim($string));
    // encode to binary hash using sha1. require S3 bucket secret key
    $hash = hash_hmac("sha1",$string, $s3secret,false);
    // sha1 hash must be base64 encoded
    $hash = base64_encode($hash);
    // base64 encoded sha1 hash must be urlencoded
    $signature = rawurlencode($hash);

    return $signature;
}

Which then creates a URL such as:

http://mybucket.s3.amazonaws.com/My_Boring_Video.wmv?AWSAccessKeyId=AKIAIEXAMPLE6GA3WYQ&Expires=1344160808&response-content-type=application/force-download&response-expires=1344160808&Signature=OTIxOTI0YjNjMTA1NjMyNmJjYTk0MGE2YWJkMmI5OWQ3MGM2ZGY0MQ%3D%3D

Which unfortunately doesn't work. Is there an obvious problem here I've been staring at far too long to properly figure out?

Answer 1

UPDATE: Specs are specs, but are only help if they match actual practice.

Amazon's S3 specs say the signature should be formed as the following:

Signature = URL-Encode( Base64( HMAC-SHA1( YourSecretAccessKeyID, UTF-8-Encoding-Of( StringToSign ) ) ) );

StringToSign = HTTP-VERB + "\n" +
    Content-MD5 + "\n" +
    Content-Type + "\n" +
    Expires + "\n" +
    CanonicalizedAmzHeaders +
    CanonicalizedResource;    

HOWEVER the actual request needed looks like this:

 StringToSign = HTTP-VERB + "\n" +
     "\n" + 
     "\n" +
     Expires + "\n" +
     Bucket + CanonicalizedResource + "?" + CanonicalizedAmzHeaders;

Strangely enough, in PHP you also can't seem to do this:

$string = "$type\n".
          "\n".
          "\n".
          "$expiration\n".
          "/$bucket$request?$headers";

It changes the signature and ends up rejected so must all be on a single line. I haven't gone as far as checking whether this is a bug in our particular version of PHP or just a general PHP bug (or maybe intended functionality??!). You must also include the name of your bucket even if you are using the vanity URLs available such as mybucket.s3.amazonaws.com or mybucket.mydomain.com. The documentation doesn't specify what you can or can't do and I made the assumption that since we are using an S3 based vanity URL it (S3) would pick up on the domain name and translate it to the bucket.

I ended up changing my function to be the following:

function signRequest($bucket, $request, $expiration, $s3secret, $headers = '', $type = 'GET', $content_type = 'default')
{
    if ($expiration == 0 || $expiration == null)
    {
        $expiration = time() + 315576000; // 10 years (never)
    }

    if (strcmp($content_type, 'default') == 0)
    {
        $content_type = "";
    }

    $headers = trim($headers, '&');

    // This is the spec:
    /*$string = "$type\n".
        "\n".
        "$content_type\n".
        "$expiration\n".
        "$headers\n".
        "$bucket$request";*/
    // but it will only work as this
    $string = "$type\n\n\n$expiration\n/$bucket$request?$headers";

    // this could be a single line of code but left otherwise for readability
    // must be in UTF8 format
    $string = utf8_encode(trim($string));
    // encode to binary hash using sha1. require S3 bucket secret key
    $hash = hash_hmac("sha1",$string, $s3secret,true);
    // sha1 hash must be base64 encoded
    $hash = base64_encode($hash);
    // base64 encoded sha1 hash must be urlencoded
    $signature = urlencode($hash);

    return $signature;
}

Hopefully someone else finds this useful as well.

UPDATE (20180109): Adding in the function that calls this (aka let's make this blatently simple).

It helps understanding of what to pass to the signRequest() function.

    private function genS3QueryString($bucketurl)
    {
            $file_type = 'application/force-download';
            $expiry = '1831014000'; //Sun, Jan 09 2028 0700 UTC

            $headers = '&response-content-type='.$file_type.'&response-expires='.$expiry;

            $bucket = preg_replace("/^.*?:\/\/(.*)\.s3\.amazonaws\.com\/.*/", "$1", $bucketurl);
            $request = preg_replace("/^.*?:\/\/.*\//", "/", $bucketurl);
            $signature = $this->signRequest($bucket, $request, $expiry, S3_SECRET_KEY, $headers, 'GET', $file_type);

            $signed_request = '?AWSAccessKeyId='.S3_KEY.'&Expires='.$expiry.$headers.'&Signature='.$signature;

            return $signed_request;
    }

You'll note however the first function only generates the encrypted signature section required to attach to a GET request for AWS. As per the docs (see the link far above) more is required for the request, as is formed from the second function just above.

The expiry time I believe can be a rolling expiry time if you so desire but should be sufficiently in the future that the asset becomes outdated and replaced long before the actual expiry time. An arbitrary time was chosen to sufficiently outlast the current version of the website.

The second function only requires the bucket url of the protected asset. The desired response type could be added to the call or simply changed so that asset (in this case just non-displayable) documents are downloaded as a different type.

The signed_request that is then returned must then be appended back onto the bucketurl for a working URI to request from a protected S3 asset.