Is it possible to not receive a completion for a WSASend call?

Question

Like the title says, is it possible for a successful WSASend call on a socket associated with an I/O completion port to not post a completion for any reason other than the thread ending?

I have a strange situation where it looks like a completion isn't being posted for a WSASend, which results in a socket leak; the application thinks a send is still pending for the socket and refuses to release it.

The send code is as follows:

void CSocketServer::Write(
    Socket *pSocket,
    CIOBuffer *pBuffer) const
{
    pSocket->AddRef();

    pBuffer->SetOperation(IO_Write_Completed);
    pBuffer->SetupWrite();
    pBuffer->AddRef();

    DWORD dwFlags = 0;
    DWORD dwSendNumBytes = 0;

    if (SOCKET_ERROR == ::WSASend(
        pSocket->m_socket,
        pBuffer->GetWSABUF(), 
        1, 
        &dwSendNumBytes,
        dwFlags,
        pBuffer, 
        NULL))
    {
        DWORD lastError = ::WSAGetLastError();

        if (ERROR_IO_PENDING != lastError)
        {
            pSocket->OnConnectionError(WriteError, pBuffer, lastError);

            pSocket->WriteCompleted();  // this pending write will never complete...

            pSocket->Release();
            pBuffer->Release();
        }
    }
    // Note: even if WSASend returns SUCCESS an IO Completion Packet is 
    // queued to the IOCP the same as if ERROR_IO_PENDING was returned.
    // Thus we need no special handling for the non error return case.
    // See http://support.microsoft.com/default.aspx?scid=kb;en-us;Q192800
    // for details.
}

Answer 1

Are you using any of the funky new features, like turning off completions for successful calls using FILE_SKIP_COMPLETION_PORT_ON_SUCCESS?

Are you doing any form of flow control on your sends or are you just sending whenever you feel like it and as often as you want? What you MIGHT be seeing is simply a SLOW completion due to the fact that the TCP stack is doing congestion control and cant send your data yet. If you keep sending data in an uncontrolled manner you can often get into a situation where completions start taking longer and longer to occur. Especially if you are sending data at a faster rate than the TCP connection is successfully getting it to the other side and especially if the TCP window isn't that big. See here: http://www.lenholgate.com/blog/2008/07/write-completion-flow-control.html for some more information.

Of course it could simply be a bug in your send logic, could you post some code?

Note that there's a known bug with WSARecv() and UDP (so nothing to do with your question at all) when using FILE_SKIP_COMPLETION_PORT_ON_SUCCESS which gives the situation that you describe if the datagram is bigger than the buffer that you supply and the WSARecv() call would have generated an WSAEMOREDATA; see here: http://www.lenholgate.com/blog/2010/01/file-skip-completion-port-on-success-and-datagram-socket-read-errors.html

Answer 2

It's possible to prevent a completion port from posting by setting the low bit of an OVERLAPPED structure's valid hEvent:

From documentation for GetQueuedCompletionStatus:

Even if you have passed the function a file handle associated with a completion port and a valid OVERLAPPED structure, an application can prevent completion port notification. This is done by specifying a valid event handle for the hEvent member of the OVERLAPPED structure, and setting its low-order bit. A valid event handle whose low-order bit is set keeps I/O completion from being queued to the completion port.