Looping HTTP post starts to fail for Amazon SES

Question

I originally posted this at the Amazon SES forums here: https://forums.aws.amazon.com/thread.jspa?threadID=74561&tstart=0

But since the stackoverflow community is more active, I'll post it here :)

Basically I have a forecah loop around a cURL post (see bottom of post for script snippet). It works for a couple hundred posts, but then starts to fail for all the others. Here's the last successful post followed by the first of 100's of unsuccessful posts...

* About to connect() to email.us-east-1.amazonaws.com port 443 (#0)
*   Trying 207.171.162.2... * connected
* Connected to email.us-east-1.amazonaws.com (207.171.162.2) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSL connection using RC4-MD5
* Server certificate:
*    subject: /C=US/ST=Washington/L=Seattle/O=Amazon.com Inc./CN=email.us-east-1.amazonaws.com
*    start date: 2010-10-08 00:00:00 GMT
*    expire date: 2013-10-07 23:59:59 GMT
*    subjectAltName: email.us-east-1.amazonaws.com matched
*    issuer: /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)09/CN=VeriSign Class 3 Secure Server CA - G2
* SSL certificate verify ok.

POST / HTTP/1.1
Accept: */*
Host: email.us-east-1.amazonaws.com
Content-Type: application/x-www-form-urlencoded
Date: Sat, 20 Aug 2011 18:59:56 UTC
X-Amzn-Authorization: AWS3-HTTPS AWSAccessKeyId=LKIABMH7PT8SO4YBRDQA,Algorithm=HmacSHA1,Signature=/0HFVEsTBGqUUSQGy9jvmsft2k4=
Content-Length: 5810
Expect: 100-continue

< HTTP/1.1 200 OK
< x-amzn-RequestId: 4a0f8f18-cb5f-11e0-8364-b14fdafc0888
< Content-Type: text/xml
< Content-Length: 326
< Date: Sat, 20 Aug 2011 19:04:55 GMT
< 
* Connection #0 to host email.us-east-1.amazonaws.com left intact
* Closing connection #0

The the failures start....

* About to connect() to email.us-east-1.amazonaws.com port 443 (#0)
*   Trying 207.171.162.2... * connected
* Connected to email.us-east-1.amazonaws.com (207.171.162.2) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSL connection using RC4-MD5
* Server certificate:
*    subject: /C=US/ST=Washington/L=Seattle/O=Amazon.com Inc./CN=email.us-east-1.amazonaws.com
*    start date: 2010-10-08 00:00:00 GMT
*    expire date: 2013-10-07 23:59:59 GMT
*    subjectAltName: email.us-east-1.amazonaws.com matched
*    issuer: /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)09/CN=VeriSign Class 3 Secure Server CA - G2
* SSL certificate verify ok.

POST / HTTP/1.1
Accept: */*
Host: email.us-east-1.amazonaws.com
Content-Type: application/x-www-form-urlencoded
Date: Sat, 20 Aug 2011 18:59:56 UTC
X-Amzn-Authorization: AWS3-HTTPS AWSAccessKeyId=LKIABMH7PT8SO4YBRDQA,Algorithm=HmacSHA1,Signature=/0HFVEsTBGqUUSQGy9jvmsft2k4=
Content-Length: 5806
Expect: 100-continue

< HTTP/1.1 400 Bad Request
< x-amzn-RequestId: 4b8f29db-cb5f-11e0-b9af-33e5c8fc863b
< Content-Type: text/xml
< Content-Length: 347
< Date: Sat, 20 Aug 2011 19:04:58 GMT
< 
* Connection #0 to host email.us-east-1.amazonaws.com left intact
* Closing connection #0

Here's the script snippet

foreach($JSONarray['DATABASE'] as $E) 
{
         if ((array_diff($E['LISTS'], $FILTER) != $E['LISTS']) && $E['STATUS'] == "CONF")
         {
         $MAIL = "Action=SendEmail&Source=".$FROME."&ReturnPath=".$BOUNCE."&Destination.ToAddresses.member.1=".$TOE."&Message.Subject.Data=".$SUBE."&Message.Body.Html.Data=".$BODYE;           
         //curl
         $aws = curl_init();
         curl_setopt($aws, CURLOPT_POSTFIELDS, $MAIL);
         curl_setopt($aws, CURLOPT_HTTPHEADER, $headers);
         curl_setopt($aws, CURLOPT_HEADER, false);
         curl_setopt($aws, CURLOPT_URL, $url);
         curl_setopt($aws, CURLOPT_RETURNTRANSFER, true);
         curl_setopt($aws, CURLOPT_VERBOSE, true);
         curl_setopt($aws, CURLOPT_STDERR, $SESLOG);
         curl_exec($aws);
         curl_close($aws);
         }
}

Any ideas?

Answer 1

I figured out the problem:

<ErrorResponse xmlns="http://ses.amazonaws.com/doc/2010-12-01/">
<Error>
<Type>Sender</Type>
<Code>RequestExpired</Code>
<Message>Request timestamp: Mon, 22 Aug 2011 15:49:11 UTC expired.  It must be within 300   secs/ of server time.</Message>
</Error>
<RequestId>0e3899cb-ccd7-11e0-9f09-c5d12d442026</RequestId>
</ErrorResponse>

My headers were set-up outside the forecah loop around curl. Doh! So I just moved that bit of code into the loop to solve the time-out problem.

Answer 2

Could it be a DOS prevention mecanism kicking in? How about putting some sleeps into your code? I would definitely put some safeguards against potential DOS attacks if I were Amazon..

Just a guess, though...