SocketException: An existing connection was forcibly closed under IIS 7, but OK under IIS 6

Question

We have had an .ashx application running under IIS 6/Windows Server 2003 for a few years now. We've recently been trying to get some new servers up and running and need to migrate to Win 2k8 R2 and IIS 7.5, however I'm having a bear of a time getting the application to run well on the new servers.

The .ashx is a web interpreter for a custom message queuing engine we have that talks via TCP. So, inside the .ashx we create a socket to talk to the message queue server. Nothing about this architecture has changed, but the error that I get on the Win 2k8 and IIS 7.5 setup, which runs .Net 4, is

System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host at System.Net.Sockets.Socket.Receive(Byte[] buffer, SocketFlags socketFlags)

We don't get this error on the older setup - same code, though it's running the .Net 3.5 runtime.

Please keep in mind that this error is not being thrown from an HTTP-facing socket, but rather from the socket that is trying to talk to the message queue server. I have verified that I do not believe I have a firewall issue by running a "raw" telnet session from the web server to the message queue and am able to successfully interact with the message queue.

Here is how we set up the socket:

            this.m_sock = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.IP);
            System.Net.IPAddress ip = System.Net.IPAddress.None;

            // Before bothering the DNS server, try to parse numeric IP
            try
            {
                ip = System.Net.IPAddress.Parse(host);
            }
            catch (FormatException)
            {
                // Not a numeric IP.  Do the DNS lookup.
                // (BTW: DNS lookup will convert the IP, but only after a failing lookup, which is slow)
                IPHostEntry ipadd = System.Net.Dns.GetHostEntry(host);
                // (Note: That lookup might fail.  It'll throw a SocketException.)
                ip = ipadd.AddressList[0];
            }
            try
            {
                this.m_sock.Connect(ip, port);
                this.m_sock.SetSocketOption(SocketOptionLevel.IP, SocketOptionName.ReuseAddress, true);
                this.m_sock.SetSocketOption(SocketOptionLevel.IP, SocketOptionName.NoDelay, true);
                this.m_sock.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, 1);
                this.m_sock.ReceiveTimeout = 60 + this.m_timeout;
                this.m_sock.Blocking = true;
                this.m_sock.ReceiveBufferSize = 65536;
                this.m_sock.SendBufferSize = 65536;
                if (ok != "" && !this.m_isweb)
                {
                    Console.WriteLine(ok);
                }
                break;
            }
            catch (SocketException)
            {
                this.m_sock.Close();
                this.m_sock = null;
            }

And now a snippet of the "ReadSome" method that blows up on us at the this.m_sock_Receive:

            if (this.m_sock.Poll(1000000, SelectMode.SelectRead))
            {
                n = this.m_sock.Available;
                if (n == 0)
                    n = 1;      // Presume at least one byte readable
                if (max > 0 && n > max)
                    n = max;
                byte[] buf = new byte[n];
                int rcvd = this.m_sock.Receive(buf, SocketFlags.None);
                if (rcvd == 0)
                    throw new knetmq_Exception(this, "Read EOF",
                        knetmq_Exception.codes.EOF);

                return buf;
            }

Before the "Receive" happens, there is a "Send" that occurs without exception, however I do not see the payload on the other side.

Any help is appreciated!

Answer 1

Oh Joy! I fixed it!

The short of it: I hadn't set my TTL, and on the new server TTL was getting consistently exceeded.

More in-depth: The new server happened to be on a slightly different network. Because we didn't set TTL, the TTL defaulted to 1. A wireshark discovered that the TTL was being exceeded repeatedly and so a TCP RST was issued, which caused the error to be thrown. That's why the messages were never received by the remote host.

I banged my head on this one for 2 days, so I hope anyone who ends up getting that SocketException can see this and look for TTL issues!