Calling PutBucket on a bucket name that doesn't already exist throws an exception

Question

My team is setting up a Eucalyptus server w/ Walrus that we want to use to run tests on our C# code, which uses Amazon's AWS stuff. It's nearly set up and we're getting the code pointed at Eucalyptus now, but we're running into some difficulty.

We started out by calling PutBucket to create the bucket that we'll be using. The bucket did not exist previously, so there shouldn't be any problems. However, we're getting a null reference exception with the following stacktrace:

at System.Xml.XmlReader.CalcBufferSize(Stream input)  
at System.Xml.XmlTextReaderImpl.InitStreamInput(Uri baseUri, String baseUriStr, Stream stream, Byte[] bytes, Int32 byteCount, Encoding encoding)  
at System.Xml.XmlTextReaderImpl..ctor(String url, Stream input, XmlNameTable nt)  
at System.Xml.XmlTextReader..ctor(Stream input)  
at Amazon.S3.AmazonS3Client.transform(String responseBody, String actionName, Type t)  
at Amazon.S3.AmazonS3Client.processRequestResponse[T](HttpWebResponse httpResponse, S3Request request, Type t, T& response, Exception& cause)  
at Amazon.S3.AmazonS3Client.handleHttpResponse[T](S3Request userRequest, HttpWebRequest request, HttpWebResponse httpResponse, Int32 retries, Int64 lengthOfRequest, T& response, Exception& cause, HttpStatusCode& statusCode)  
at Amazon.S3.AmazonS3Client.getResponseCallback[T](IAsyncResult result)  
at Amazon.S3.AmazonS3Client.endOperation[T](IAsyncResult result)  
at Amazon.S3.AmazonS3Client.EndPutBucket(IAsyncResult asyncResult)  
at Amazon.S3.AmazonS3Client.PutBucket(PutBucketRequest request)  
[at application library]

Now here's where it gets weird. When we checked the server, the bucket was successfully created! The server processed the request correctly and made the bucket that we wanted. The response is coming back invalid, but the request was processed correctly. And so far the problem only seems to occur when calling PutBucket, because when we delete the bucket it is successfully deleted off of the server and no exceptions are thrown.

So far we're at a loss as to what's happening. We suspect that there's a problem on Walrus's end. Perhaps AWS is expecting one thing and Walrus is doing another. Maybe we didn't configure Eucalyptus and/or Walrus correctly. We're not sure at all.

That's why I'm posting the question here. I was curious if anyone has had any experience with this before, or perhaps can offer some insight. What could cause the server to correctly process a request but send back an invalid response? Why would Eucalyptus make the bucket but AWS think that the operation failed?

Answer 1

The issue is likely that the Walrus PUT bucket response includes a body that element that is no longer included in responses from S3 (it used to be the case, but was removed).

So the operation is succeeding on Walrus, but the client is failing to parse the response properly. This is often dependent upon the specific clients as some are more strict than others (some, for instance, will not even try to parse a response body if it isn't expected to exist for a particular response code such as HTTP 200 OK).

I would recommend looking at what is being sent/received over the wire. Wireshark/tcpdump should show the full response coming from Walrus. Likely there is an XML difference that is enough that the client is failing. We at Eucalyptus are working to fix these issues as they arise, but because the S3 API (REST in particular) changes without warning or a version bump, sometimes things change and the SDK is updated before we can make the change in Walrus.

Feel free to ask on engage.eucalyptus.com for help or catch us on IRC on Freenode in #eucalyptus or #eucalyptus-devel