Winforms App becomes unresponsive after a period of time when making Asynchronous WCF calls via netTcpBinding

Question

Scenario

I am supporting/enhancing a Windows Form App that communicates via a WCF service hosted in IIS using a ServiceHostFactory class.

For the problem I am trying to solve, I am dealing with these 2 methods in the ServiceContract. The ServiceContract is marked as PerCall.

    [OperationContract(IsOneWay = true)]
    void RunJob(int jobId);

    [OperationContract]
    byte[] GetUserJobs(int userID);

The user will submit a job, via the RunJob method in a fire and forget fashion. This job will make a WCF call to another service, get some data back, do a lot of stuff with it and store it in the database. This particular job takes roughly 62 minutes to complete.

While the job is in a Running state, the client calls the GetUserJobs method asynchronously every 10 seconds to check job status and update the GUI accordingly. The client is communicating with the WCF service via netTcpBinding.

Problem

Right around the 1 hour mark, the GUI becomes completely unresponsive. Asynchronous calls are still being made, but the completed event is never being called. It seems to me something is deadlocked or blocking, and I can't figure exactly why this is happening. The GUI can start becoming unresponsive before the RunJob (OneWay) is actually finished on the server, but the job itself always finishes and data gets saved to the database.

So even though the GUI is unusable, the Server is still working, except it won't respond to any WCF calls.

If I edit the web.config on the server to recycle IIS, the GUI becomes responsive again.

I am by no means a WCF expert, and I am really struggling with coming up with a solution to this problem. I am pretty sure it is happening because of the asynchronous calls, but I can't pinpoint the exact cause yet. The Asynchronous calls are getting closed, according to the code below.

According to my trace logs, I see the following errors right around when the GUI becomes unresponsive:

<ExceptionType>System.ServiceModel.CommunicationException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>There was an error writing to the pipe: The pipe is being closed. (232, 0xe8).</Message>

Then for Activity http://tempuri.org/IConnectionRegister/ValidateUriRoute I will see:

<DataItem> <TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Warning"> <TraceIdentifier>http://msdn.microsoft.com/en-US/library/System.ServiceModel.CommunicationObjectFaulted.aspx</TraceIdentifier> <Description>Faulted System.ServiceModel.Channels.ClientFramingDuplexSessionChannel</Description> <AppDomain>/LM/W3SVC/2/ROOT/Service_mt-4-130395564562210673</AppDomain> <Source>System.ServiceModel.Channels.ClientFramingDuplexSessionChannel/56837067</Source> </TraceRecord> </DataItem>

<DataItem> <TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Warning"> <TraceIdentifier>http://msdn.microsoft.com/en-US/library/System.ServiceModel.CommunicationObjectFaulted.aspx</TraceIdentifier> <Description>Faulted System.ServiceModel.Channels.ServiceChannel</Description> <AppDomain>/LM/W3SVC/2/ROOT/Service_mt-4-130395564562210673</AppDomain> <Source>System.ServiceModel.Channels.ServiceChannel/3537205</Source> </TraceRecord> </DataItem>

Relevant Code/Config (Sanitized a bit)

Client Binding Config

I have tried to set all the timeouts to "infinite" for now, just to rule out that there was some strange timeout behavior. I have tried various timeout settings, but nothing seems to work.

    <binding name="MyEndpoint" closeTimeout="infinite"
                openTimeout="infinite" receiveTimeout="infinite" sendTimeout="infinite"
                transactionFlow="false" transferMode="Buffered" transactionProtocol="OleTransactions"
                hostNameComparisonMode="StrongWildcard" listenBacklog="10"
                maxBufferPoolSize="2147483647" maxBufferSize="2147483647"
                maxConnections="100" maxReceivedMessageSize="2147483647">
          <readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647"
                  maxArrayLength="2147483647" maxBytesPerRead="2147483647"
                  maxNameTableCharCount="2147483647" />
          <reliableSession ordered="true" inactivityTimeout="infinite"
                  enabled="false" />
          <security mode="Transport">
            <transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />
            <message clientCredentialType="Windows" />
          </security>
</binding>

Server Binding Config

      <netTcpBinding>
    <binding name="MyBinding" portSharingEnabled="true" transferMode="Buffered" closeTimeout="infinite" openTimeout="infinite" receiveTimeout="infinite" sendTimeout="infinite" maxBufferPoolSize="524288" maxBufferSize="2147483647" maxConnections="10" listenBacklog="200" maxReceivedMessageSize="2147483647">
      <security mode="Transport">
        <transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />
        <message clientCredentialType="Windows" />
      </security>
      <reliableSession ordered="true"
             inactivityTimeout="infinite"
             enabled="false" />
      <readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
    </binding>
<netTcpBinding>

 <serviceBehaviors>
    <behavior name="MyServiceBehavior">
      <serviceMetadata httpGetEnabled="true" />
      <serviceDebug includeExceptionDetailInFaults="true" />
      <ServiceErrorHandler />
      <dataContractSerializer maxItemsInObjectGraph="2147483647" />          
    </behavior>
    <behavior name="">
      <serviceMetadata httpGetEnabled="true" />
      <serviceDebug includeExceptionDetailInFaults="true" />
    </behavior>
  </serviceBehaviors>

The GUI does something similar to the following to asynchronously call the WCF service:

 public void RefreshJobs()
 {
    //When the GUI becomes unresponsive, a lot of these log statements start piling up in the log file, until service is restarted
    Logger.GetInstance().Info("Begin RefreshJobs");
    ServiceClient svc = new ServiceClient("MyEndpoint", url);
    try
    {
        svc.GetUserJobsCompleted += new EventHandler<GetUserJobsCompletedEventArgs>(svc_GetJobsCompleted);
    }
    catch (Exception e)
    {
        throw e;
    }

    svc.GetUserJobsAsync(SomeSingleton().CurrentUser.UserID, false, svc);
    Logger.GetInstance().Info("End RefreshJobs");
}        

private void svc_GetJobsCompleted(object sender, GetUserJobsCompletedEventArgs e)
{
    Logger.GetInstance().Info("Start GetJobsCompleted");
    if (e.Result != null)
    {
        //Do Stuff to Update GUI
    }

    //Close the connection
    if (e.UserState is ServiceClient)
    {
        ((ServiceClient)e.UserState).Close();
    }
    Logger.GetInstance().Info("End GetJobsCompleted");
}

Answer 1

The problem ended up being the CPU being overloaded, which caused the Smsvchost.exe to stop responding.

A fix from Microsoft is available