We have finally solved this problem. As mentioned previously, we noticed that the IIS logs contained a sc-win32-status 64
error when we experienced the Service Unavailable
problem in the browser when (and only when) our site was using the Load Balancer
.
To help look into this further, we did a network capture of the traffic on the Load Balancer
while testing. We reproduced the random Service Unavailable
problem, saw the associated win32-status 64
error in the IIS logs, and identified the specific packet of traffic on the network capture for this event.
Using Wireshark
, we followed the TCP stream and noticed that the TCP connection was reset by the Load Balancer
immediately after this packet. We reproduced the problem three times and every time there was a TCP reset immediately afterwards.
Walking backwards through the TCP stream, we noticed in all three instances a packet for HTTP/1.1 200 (accplication/octet-stream)
and prior to that a request to download a document (ie. .pdf or .xlsx or .docx) from one of our sites. The server that contains all our documents is not a web server and does not have the IIS role active. The document server does not have a way to define the content/media type for the document that is being downloaded. Hence the generic (application/octet-stream) packet in the network capture. The Load Balancer
treated the request for a document as potentially malicious and decided to reset the TCP connection if another request is made. To fix the problem, we added a content type library function to our application using this post as a guide. Sorted!
In Summary:
- A document was requested from our document server via our web application
- The document was sent back to the user with a generic content type =
application/octet-stream
- The Load Balancer flagged this activity to be potentially malicious
- Another request within this TCP connection was made
- The Load Balancer reset the TCP connection
- This results in a Service Unavailable
Lesson Learned:
Always define your content/media types if you are serving content from a non web server or a web server running an IIS version less than 7 (Heaven forbid).