How to isolate IIS Thread Pool Crashing

Question

This question is related to a connection reset question I asked recently. I am also pursuing the info in this SO article and this SO article. What I want to know is the best way to track down the cause of an IIS Thread Pool crash. I am also pursuing this problem with our vendor. Here are the details.

I have an IIS application. It uses a very simple ASP page, modeled on a sample supplied by our vendor, when we purchased our first COM toolkit quite a few years ago. This application runs on IIS on Windows Server 2003. The application is a separate web service, not a virtual root, and it's at a different port number than 80. In other words, we're not using another NIC or virtual IP address. The site has little traffic, but was configured to have a DMZ. The whole configuration looks a bit funky to me.

Address verification requests are sent to the IIS application from browser-less applications that use http protocol. Street number, street name, city, and state are sent with a few other pieces of identification, and a response is sent back. One of the applications is written in C; the other is written in Clojure.

Sending "single-shot" requests to the newly built IIS application works just fine from both applications. Sending a lot of requests, and it's been hard to pin down whether that's over 25 or some other number, causes the IIS thread pool to crash. That's more or less what the error logs say.

This application has run without incident for years on a W2K/IIS server. The ASP page talks to an address verification engine by means of a COM toolkit that we purchased from the vendor. We had to upgrade to a new server (and 2012 COM toolkit) only because the newest COM toolkit would not install on W2K, and the new COM toolkit contained support for the new USPS address verification regulations. We'll validate fewer addresses without the new toolkit, and the only way you can barcode addresses and get that discount is to validate addresses. (When you send out 29,000 motor vehicle excise bills, the discount helps.)

What is the best way to track down this problem, so I can find the culprit? I'm looking for links to good information as much as answers. I am sorry to be vague; I know the rules of SO, and endeavor to provide as much detail as possible. I can re-edit this post and provide log entries, if someone wants to see those. The bottom line is my Clojure client (handles the batch requests) starts getting resets, and crashes, when the IIS thread pool crashes.

Epilogue:

We believe the rapid opening and closing of the COM object is the issue, and our writing the ASP page this way was sanctioned by our vendor. To get around this problem and the fact we also needed to install MS Access for additional address verification functions, we wound up writing an ActiveState Perl program to solve multiple needs.

First, we have purchased an ActiveState product that will the our Perl program run as a service. The service will start with parameters and a port number that allow the http-based requests to be sent, just like the were before. In this case the COM object will be opened once when the Perl program starts, and close when the Perl program exits.

The other functions provided by the Perl program are not germane to this post, but the writing of this Perl program obviates the need for IIS and MS Access, which were required with the old configuration.

So, if you want to take something from this post, it would be open a COM object on opening of a program and close on program close, at least for W2K Server 2003 and higher.

Answer 1

We had an issue at work with respect to application slowness. We did an application pool memory dump and then analysed the dump.

http://support.microsoft.com/kb/919790