WSASYSCALLFAILURE with overlapped IO on Windows XP

Question

I hit a bug in my code which uses WSARecv and WSAGetOverlapped result on an overlapped socket. Under heavy load, WSAGetOverlapped returns with WSASYSCALLFAILURE ('A system call that should never fail has failed') and my TCP stream is out of sync afterwards, causing mayhem in the upper levels of my program.

So far I have not been able to isolate it to a given set of hardware or drivers. Has somebody hit this issue as well, and found a solution or workaround?

Answer 1

It seems I have solved this issue by sleeping 1ms and retrying the WSAGetOverlapped result when it reports a WSASYSCALLFAILURE.

I had another issue related to overlapped events firing, even though there is no data, which I also had to solve first. The test is now running for over an hour, with a few WSASYSCALLFAILURE handled correctly. Hopefully the overnight test will succeed as well.

@Len: thanks again for your help.

EDIT: The overnight test was successful. My bug was caused by two interdependent issues:

Issue 1: WaitForMultipleObjects in ConnectionSet::select occasionally signals data on an empty socket, causing SocketConnection::readSync to deadlock. Fix: Do a non-blocking read on the first byte of each packet. Reset ConnectionSet if socket was empty

Issue 2: WSAGetOverlappedResult returns occasionally WSASYSCALLFAILURE, causing out-of-sync on the TCP stream. Fix: Retry WSAGetOverlappedResult after a small sleep period.

http://equalizer.svn.sourceforge.net/viewvc/equalizer?view=revision&revision=4649

Answer 2

How many connections, how many pending recvs, how many outsanding sends? What does perfmon or task manager say about the amount of non-paged pool used? How much memory in the box? Does it go away if you run the program on Vista or above? Do you have any LSPs installed?

You could be exhausting non-paged pool and causing a badly written driver to misbehave when it fails to allocate memory. This issue is less likely to bite on Vista or later as the amount of non-paged pool available has increased dramatically (see http://www.lenholgate.com/blog/2009/03/excellent-article-on-non-paged-pool.html for details). Alternatively you might be hitting the "locked pages" limit (you can only lock a fixed number of pages in memory on the OS and each pending I/O operation locks one or more pages depending on buffer size and allocation alignment).