WCF client SecurityNegotiationException

Question

I have a WCF service that uses a server certificate. The certificate is signed by our internal certificate authority. The root certificate is self-signed and is distributed to all domain computers via AD. We have built and deployed many programs that attach to this service (around 50 applications over about 200 PCs), and they all work as expected, except for one server (I'll refer to this as BROKENHOST). On this specific server (Windows Server 2003 standard server), whenever a program attempts to connect to the service, it throws a SecurityNegotiationExeption, specifically (names changed to protect the company):

System.ServiceModel.Security.SecurityNegotiationException: The X.509 certificate CN=server.sub.domain.com, OU=sub, O=domain.com, L=Somewhere, S=California, C=US chain building failed. The certificate that was used has a trust chain that cannot be verified. Replace the certificate or change the certificateValidationMode. The revocation function was unable to check revocation because the revocation server was offline.

---> System.IdentityModel.Tokens.SecurityTokenValidationException: The X.509 certificate CN=server.sub.domain.com, OU=sub, O=domain.com, L=Somewhere, S=California, C=US chain building failed. The certificate that was used has a trust chain that cannot be verified. Replace the certificate or change the certificateValidationMode. The revocation function was unable to check revocation because the revocation server was offline.

The certificate used by the WCF service is also used by IIS. The strange part here is that if I open IE on BROKENHOST and put in https://server.sub.domain.com, the page opens just fine, and the certificate chain says OK. So, I'm not sure why IE is fine with the cert, but the windows apps fail.

The client connections themselves are created via a channelfactory like this:

System.ServiceModel.Channels.AddressHeader[] addressHeader = new System.ServiceModel.Channels.AddressHeader[] {
            System.ServiceModel.Channels.AddressHeader.CreateAddressHeader("CUSTOM_USER_ID", "CUSTOM", m_user_id),
            System.ServiceModel.Channels.AddressHeader.CreateAddressHeader("CUSTOM_SOURCE_NAME", "CUSTOM", m_source_name)
};

string strUri = string.Format("net.tcp://{0}:{1}/{2}", m_strServer[strRegKey], m_strPort[strRegKey], strAddress);
System.ServiceModel.EndpointAddress epa = new System.ServiceModel.EndpointAddress(new Uri(strUri), EndpointIdentity.CreateDnsIdentity(m_strServer[strRegKey]), addressHeader);

NetTcpBinding binding = new NetTcpBinding();
binding.OpenTimeout = TimeSpan.FromSeconds(15);
//binding.OpenTimeout = TimeSpan.FromTicks(1);
binding.ReaderQuotas.MaxArrayLength = Int32.MaxValue; // ChunkSizeSingleton.DOWNLOAD_CHUNK_MAX_SIZE;
binding.ReaderQuotas.MaxBytesPerRead = Int32.MaxValue;
binding.ReaderQuotas.MaxDepth = Int32.MaxValue;
binding.ReaderQuotas.MaxNameTableCharCount = Int32.MaxValue;
binding.ReaderQuotas.MaxStringContentLength = Int32.MaxValue;
binding.Security.Mode = SecurityMode.TransportWithMessageCredential;
binding.Security.Transport.ProtectionLevel = System.Net.Security.ProtectionLevel.EncryptAndSign;
binding.Security.Message.ClientCredentialType = MessageCredentialType.UserName;
binding.ReliableSession.Enabled = true;
binding.ReliableSession.Ordered = true;
binding.ReliableSession.InactivityTimeout = TimeSpan.FromMinutes(10);

binding.MaxReceivedMessageSize = Int32.MaxValue; // (long)(ChunkSizeSingleton.DOWNLOAD_CHUNK_MAX_SIZE * 1.3);

ChannelFactory<T> channelFactory = new ChannelFactory<T>(binding, epa);

channelFactory.Credentials.UserName.UserName = m_user_name;
channelFactory.Credentials.UserName.Password = m_user_password;
channelFactory.Credentials.SupportInteractive = false;

foreach (OperationDescription od in channelFactory.Endpoint.Contract.Operations)
{
    DataContractSerializerOperationBehavior ser = od.Behaviors.Find<DataContractSerializerOperationBehavior>();

    ser.MaxItemsInObjectGraph = Int32.MaxValue;
}

Again, we're confident that the server setup is correct, and that there's something wrong with BROKENHOST. I've tried running the code with the remote debugger, but that didn't yield me anything we didn't already know.

If anybody has any other suggestions for me to try, please let me know.

P.S. in case it helps, here is the stacktrace:

at System.IdentityModel.Selectors.X509CertificateChain.Build(X509Certificate2 certificate)
at System.IdentityModel.Selectors.X509CertificateValidator.ChainTrustValidator.Validate(X509Certificate2 certificate)
at System.IdentityModel.Selectors.X509SecurityTokenAuthenticator.ValidateTokenCore(SecurityToken token)
at System.IdentityModel.Selectors.SecurityTokenAuthenticator.ValidateToken(SecurityToken token)
at System.ServiceModel.Channels.SslStreamSecurityUpgradeInitiator.ValidateRemoteCertificate(Object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors)
at System.Net.Security.SslStream.userCertValidationCallbackWrapper(String hostName, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors)
at System.Net.Security.SecureChannel.VerifyRemoteCertificate(RemoteCertValidationCallback remoteCertValidationCallback)
at System.Net.Security.SslState.CompleteHandshake()
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
at System.ServiceModel.Channels.SslStreamSecurityUpgradeInitiator.OnInitiateUpgrade(Stream stream, SecurityMessageProperty& remoteSecurity)
   --- End of inner exception stack trace ---

Server stack trace:

at System.ServiceModel.Channels.SslStreamSecurityUpgradeInitiator.OnInitiateUpgrade(Stream stream, SecurityMessageProperty& remoteSecurity)
at System.ServiceModel.Channels.StreamSecurityUpgradeInitiatorBase.InitiateUpgrade(Stream stream)
at System.ServiceModel.Channels.ConnectionUpgradeHelper.InitiateUpgrade(StreamUpgradeInitiator upgradeInitiator, IConnection& connection, ClientFramingDecoder decoder, IDefaultCommunicationTimeouts defaultTimeouts, TimeoutHelper& timeoutHelper)
at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.SendPreamble(IConnection connection, ArraySegment`1 preamble, TimeoutHelper& timeoutHelper)
at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.DuplexConnectionPoolHelper.AcceptPooledConnection(IConnection connection, TimeoutHelper& timeoutHelper)
at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at System.ServiceModel.ICommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.SecurityChannelFactory`1.ClientSecurityChannel`1.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.DoOperation(SecuritySessionOperation operation, EndpointAddress target, Uri via, SecurityToken currentToken, TimeSpan timeout)
at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.GetTokenCore(TimeSpan timeout)
at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout)
at System.ServiceModel.Security.SecuritySessionClientSettings`1.ClientSecuritySessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ReliableChannelBinder`1.ChannelSynchronizer.SyncWaiter.TryGetChannel()
at System.ServiceModel.Channels.ReliableChannelBinder`1.ChannelSynchronizer.SyncWaiter.TryWait(TChannel& channel)
at System.ServiceModel.Channels.ReliableChannelBinder`1.ChannelSynchronizer.TryGetChannel(Boolean canGetChannel, Boolean canCauseFault, TimeSpan timeout, MaskingMode maskingMode, TChannel& channel)
at System.ServiceModel.Channels.ReliableChannelBinder`1.Send(Message message, TimeSpan timeout, MaskingMode maskingMode)
at System.ServiceModel.Channels.SendReceiveReliableRequestor.OnRequest(Message request, TimeSpan timeout, Boolean last)
at System.ServiceModel.Channels.ReliableRequestor.Request(TimeSpan timeout)
at System.ServiceModel.Channels.ClientReliableSession.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ClientReliableDuplexSessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open()