WCF throughput lower than expected

Question

I apologize for the post length, but there are lots of things that can cause my situation and I've tried to include all of the setting changes I've made based on other posts. In short, My WCF service seems to be throttled to 3 or 4 concurrent client requests at a time. If I set the application pool Maximum Worker Processes higher (around 10) or set the service behavior ConcurrencyMode to Multiple, I get much better throughput (several times faster). However, these seem like work-arounds to the real issue, bringing with them their own issues. Am I mistaken, or should IIS be able to bring up many instances (dozens or more) of my WCF service inside one worker process to handle the load? I just know I'm missing a setting somewhere, but I cannot find it.

Edit: In trying out the suggestions so far, I realized my math was off for throughput. With the ForEach loop, I am getting an estimated concurrent processing on the server in the low 20s (per task duration * number of tasks / total run time). This still seems low for the actual work being done (sleep 10 seconds), but no longer ridiculously low.

Second Edit: I marked @Pablo's comment as the answer because his answer plus a link of his gave me the information to significantly boost performance (I think about 3 fold). However, I would like to pose the follow-up question- What is a reasonable expectation for processing concurrent requests in WCF / IIS? Assuming CPU, memory, and IO are not bottlenecks, what is a practical limit / expectation (per CPU) for handling requests? What I'm looking for is a rule-of-thumb that tells me that I'm probably not going to get any more big boosts without adding CPUs (or worker processes). Thanks again.

(On Windows 2008 Server, hosted by IIS, 1 processor)
WCF Service Config (Abbreviated):

<?xml version="1.0"?>
<configuration>
  <configSections>
  <system.serviceModel>
    <serviceHostingEnvironment multipleSiteBindingsEnabled="true"/>
    <services>
      <service name="FMHIRWCFSvc.IRService" behaviorConfiguration="IRServiceBehavior">
        <endpoint address="" binding="basicHttpBinding" bindingConfiguration="Binding1" contract="FMHIRWCFSvc.IIRService" />
      </service>
    </services>
    <bindings>
      <basicHttpBinding>
        <binding name="Binding1" maxReceivedMessageSize="104857600">
          <readerQuotas maxArrayLength="104857600"/>
          <security mode="Transport">
            <transport clientCredentialType="None"/>
          </security>
        </binding>
      </basicHttpBinding>
    </bindings>
    <behaviors>
      <serviceBehaviors>
        <behavior name="IRServiceBehavior">
          <serviceMetadata httpsGetEnabled="true"/>
          <serviceDebug includeExceptionDetailInFaults="true"/>
          <serviceThrottling
              maxConcurrentCalls = "500"
              maxConcurrentSessions = "500"
              maxConcurrentInstances = "500"
            />
        </behavior>
      </serviceBehaviors>
    </behaviors>
  </system.serviceModel>
  <applicationSettings>
    <FMHIRWCFSvc.Properties.Settings>
      <setting name="FMHIRWCFSvc_ir_dev_websvc_IRWebService40" serializeAs="String">
        <value>http://ir-dev-websvc/imageright.webservice/IRWebService40.asmx</value>
      </setting>
    </FMHIRWCFSvc.Properties.Settings>
  </applicationSettings>
  <system.net>
    <connectionManagement>
      <add address="*" maxconnection="500"/>
    </connectionManagement>
  </system.net>
</configuration>

Client Config (Abbreviated):

<?xml version="1.0"?>
<configuration>
  <startup><supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/></startup>
  <system.serviceModel>
    <bindings>
      <basicHttpBinding>
        <binding name="BasicHttpBinding_IIRService" closeTimeout="00:01:00"
          openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
          allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
          maxBufferSize="65536" maxBufferPoolSize="524288" maxReceivedMessageSize="65536"
          messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered"
          useDefaultWebProxy="true">
          <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
            maxBytesPerRead="4096" maxNameTableCharCount="16384" />
          <security mode="TransportCredentialOnly">
            <transport clientCredentialType="Windows" />
          </security>
        </binding>
      </basicHttpBinding>
    </bindings>
    <client>
      <endpoint address="https://{myserveraddress}/FMHIRWCFSvc/IRService.svc"
        binding="basicHttpBinding" bindingConfiguration="BasicHttpBinding_IIRService"
        contract="wcf_local.IIRService" name="BasicHttpBinding_IIRService" />
    </client>
  </system.serviceModel>
  <system.net>
    <connectionManagement>
      <add address="*" maxconnection="500"/>
    </connectionManagement>
  </system.net>
</configuration>

Client Method Call:

static void WCFTesting()
{
    ConcurrentQueue<Exception> exceptionList = new ConcurrentQueue<Exception>();
    int[] taskList = new int[250];
    Parallel.ForEach(taskList, theTask => 
    {
        try
        {
            // Create the WCF client
            BasicHttpBinding binding = new BasicHttpBinding {
                Security = { Mode = BasicHttpSecurityMode.Transport },
                SendTimeout = TimeSpan.FromSeconds(20)
            };
            EndpointAddress endpointAddress = new EndpointAddress("https://{myserveraddress}/FMHIRWCFSvc/IRService.svc");
            IRServiceClient wcfClient = new IRServiceClient(binding, endpointAddress);

            // Call wcf service method that sleeps 10 seconds and returns
            wcfClient.TestCall();
        }
        catch (Exception exception) {
            // Store off exceptions for later processing
            exceptionList.Enqueue(exception);
        }
    });
}

WCF Service Code:

[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]
...
public string TestCall()
{
    Thread.Sleep(10000);
    return null;
}

Thanks for any insight or suggestions!

Answer 1

The testing mechanism you are using might not be entirely correct.

Using Parallel.For() doesn't mean that it will create 250 workers in parallel. In this case it seems you are limiting your benchmark to what would be optimal to your client's processor configuration, and not actually testing what the server can handle.

If you really want to invoke 250 parallel threads and see how it reacts you can manually create all the threads. Such as:

        var exceptionList = new ConcurrentQueue<Exception>();
        const int max = 250;
        int numberOfTasks = max;
        var signal = new ManualResetEvent(false);
        for (var i = 0; i < max; i++)
        {
            var thread = new Thread(() =>
            {
                try
                {
                    // Create the WCF client
                    BasicHttpBinding binding = new BasicHttpBinding
                    {
                        Security = { Mode = BasicHttpSecurityMode.Transport },
                        SendTimeout = TimeSpan.FromSeconds(20)
                    };
                    EndpointAddress endpointAddress = new EndpointAddress("https://{myserveraddress}/FMHIRWCFSvc/IRService.svc");
                    IRServiceClient wcfClient = new IRServiceClient(binding, endpointAddress);

                    // Call wcf service method that sleeps 10 seconds and returns
                    wcfClient.TestCall();
                }
                catch (Exception exception)
                {
                    // Store off exceptions for later processing
                    exceptionList.Enqueue(exception);
                }

                if (Interlocked.Decrement(ref numberOfTasks) == 0) signal.Set();
            });
            thread.Start();
        }
        signal.WaitOne();

Answer 2

Parallel.ForEach and Task.Start end up in approximately the same code. Tasks are not guaranteed to run concurrently.

Better to test with async calls to the WCF service link.