Kerberos authentication on a self-hosted WCF Data Service

Question

We have a WCF Data Service which is self-hosted under a Windows service (not using IIS) which we are currently working to secure using SSL and Windows Authentication.

After some time playing around with netsh and server certificates, we now have the service secured with SSL and we have also enabled Windows Authentication on the webHttpBinding in our app.config - however we are now seeing some strange behaviour when attempting to authenticate certain users - some can log in fine, others have their credentials rejected and are prompted with HTTP 400 errors.

After some testing and digging around it would appear that we might be running into this problem, where the authentication header used by Kerberos may be greater than the maximum permitted header length (which I believe is 16k) for certain users - and although there is a documented workaround for IIS, there does not appear to be an equivalent setting we can use for a self-hosted service, or in our app.config - unless I'm missing something? We tried setting the maxReceivedMessageSize and maxBufferSize fields to their maximum values to see if that would make any difference, but apparently not.

Binding config:

  <webHttpBinding>
    <binding name="DataServicesBinding"
             maxReceivedMessageSize="2147483647"
             maxBufferSize="2147483647">
      <security mode="Transport">
        <transport clientCredentialType="Windows" />
      </security>
    </binding>
  </webHttpBinding>

We've managed to work around this issue temporarily by setting the clientCredentialType in our binding to use Ntlm instead, but we'd like to get Kerberos working if possible for obvious reasons.

Answer 1

So, as it turns out, this was caused by our service not being configured with a SPN (Service Principal Name). This can be done using the setspn tool with Windows Server. (See this MSDN article for more information.)

Once the SPN was applied, Kerberos authentication started to work as expected.

Answer 2

Use wireshark to see what the client sends. Make sure that this input is correct and then come back.