what happens to incoming data, when there is no wsarecv at the moment

Question

I work on multithreaded IOCP Server in windows. To avoid handling with many pending reads, I have always only one wsarecv operation on each connected socket. In brief my design looks like:

1. after client connecting, wsarecv is posted on connected socket
2. when wsarecv is executed, data is proccessed and wsasend is posted on the same socket
3. when wsasend from step 2 completes (GQCS gets notification), wsarecv is posted again.

It means there is a little amount of time between step 2 and 3, when there is no pending wsarecv waiting for client data, which can appears in any time.

Is this situation, I should worry about or I can assume that if data would arrive at this particular little amount of time, it will be stored in some kind of internal buffer and taken from it at the moment wsarecv from step 3 is posted ?

thx for help.

Answer 1

As long as you have not disabled the network stack's buffering (using SO_RCVBUF and setting the buffer size to 0) then you will have some buffer space within the networking stack and this will be used if you don't have a WSARecv() pending.

If you are using TCP then you don't even have to worry when you fill this buffer space as that will cause a Zero Window and the sender will, hopefully, stop sending (see here for why it might not actually stop sending), but even if it doesn't then your stack will simply throw away subsequent datagrams and TCP will eventually resend them.

With UDP it's a little different. If you fill the recv buffer then you will start to drop datagrams. By default the stack will drop the newest datagrams, you can change this by setting SIO_ENABLE_CIRCULAR_QUEUEING which will cause the oldest datagrams to be dropped instead.

You COULD opt to always have at least one WSARecv() pending on a connection by a) posting more than one to start with and b) posting a new one as the first thing you do when one completes. This works well for UDP but with TCP the problem with this approach is that you then have to allow for the fact that multiple recvs could complete "at the same time" and then you have to ensure that your I/O threads work together to keep the TCP data stream in sync (see here for the issues).

It may be more performant to disable the stack's recv buffer and always have enough WSARecv()'s pending on a connection as this will remove a memory copy from the inbound data path.