What causes “Tcp channel protocol violation expecting preamble” in .NET Remoting?

Question

I have a distributed application that uses .NET Remoting on an in-house gigabit network. There is a single server, and over a dozen clients that connect to the server. The clients run multiple threads, and there can be up to 10 concurrent requests from each client.

This application works very well most of the time. The server stays up for months at a time. From time to time I get an exception on a client, and I can't figure out what's causing the exception. The exception and stack trace are:

System.Runtime.Remoting.RemotingException: Tcp channel protocol violation: expecting preamble.

Server stack trace: 
   at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadAndMatchPreamble()
   at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadVersionAndOperation(UInt16& operation)
   at System.Runtime.Remoting.Channels.Tcp.TcpClientSocketHandler.ReadHeaders()
   at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
   at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

That's the stack trace back to where I make the remoting call.

I've looked and can find nothing out of the ordinary in the calls I'm making, or in the data that the server is returning.

Google searches for this error are not very fruitful. Most of the errors I've seen revolve around somebody converting from HTTP to TCP and not changing everything, so they get the exception when they try to connect. In my case, the client will run for days before I get this error.

One other data point: the server does get a lot of requests. Most of the clients are Web crawlers that each make more than 2,000 requests to the server every minute. So the server is processing upwards of 500 requests per second, with bursts of higher traffic. In any case, the server seems to handle the traffic okay, and I would expect a much different error if the server became overloaded.

Any ideas what's causing this error?

Answer 1

This error usually happens when a message received with wrong headers. You can repeat this error with creating a telnet connection to your server and type something. In most cases it's a network error.

I highly recommend the check your firewall. Some firewalls drops the network packets due wrong attack alert.

Load balancing is another possible reason. Load balancer splits packets to different servers.

Answer 2

I've see this issue when MS DNS server used too lot of open port, so outbound port can't be created. Yes, it's very funny.