Why does WSASend return 0 but still call the completion routine?

Question

The docs clearly say that if WSASend completes immediately, that you'll get WSA_IO_PENDING, but that just doesn't ever happen. I always get 0, and dwBytesTransferred always matches the bytes that I sent. However, it seems like sometimes my completion routine is called and sometimes, it isn't. I allocate the buffer for the send, so I need to free the buffer if the completion routine won't be called.

I have three impromptu counters, m_dwAsyncSend, m_dwSyncSend, and m_dwCompletions. m_dwAsycSend is always zero and m_dwSyncSend and m_dwCompletions are always very far apart, as one m_dwSyncSend may be 750 and m_dwCompletions is 2. There are plenty of times that the thread is alertable, so I don't think I'm starving it that way.

This is making me crazy! It's after midnight and I've been at this all day. I blame that if any of this is incoherent!

Here's the code, I don't think you need the class file to see what I'm doing.

void
CALLBACK
SendCompletion(
    DWORD dwError,
    DWORD cbTransferred,
    LPOVERLAPPED pOvl,
    DWORD dwFlags
    )
{
    LPWSABUF pBuffer = (LPWSABUF)((DWORD_PTR)pOvl + sizeof(OVERLAPPED));
    CNetAsyncSocket *pSock = (CNetAsyncSocket *)pOvl->hEvent;
    pSock->m_dwCompletions++;
    if(dwError != NO_ERROR)
    {
        // If things didn't go as planned, ring the bell and disconnect.
        pSock->Disconnect();
        tracef(TT_REGULAR, 1,
            "SOCKET_ERROR in CNetAsyncSocket::Send(), disconnecting, error code: %ld, on socket: %s:%ld",
            dwError, pSock->GetIP(), pSock->GetPort());
        free(pOvl);
    }
    else
    {
        // If we sent less than we meant to, queue up the rest.
        if(cbTransferred < pBuffer->len)
        {
            DWORD dwRemaining = pBuffer->len - cbTransferred;
            memmove(pBuffer->buf, (PVOID)((DWORD_PTR)pBuffer->buf + dwRemaining), dwRemaining);
            pBuffer->len = dwRemaining;
        }
        else
        {
            free(pOvl);
        }
    }
}
void CNetAsyncSocket::SendAsync(PBYTE pData, DWORD dwLength)
{
    // We want to minimize heap churn, so let's do this in one allocation.
    // Also, having this in one chunk of memory makes it easier to interpret
    // it on the other side.
    DWORD dwAllocSize =
        sizeof(OVERLAPPED) +        // The OVERLAPPED struct.
        sizeof(WSABUF) +            // The buffer info.
        dwLength;                   // The actual buffer we're copying.
    LPOVERLAPPED pOvl = (LPOVERLAPPED)malloc(dwAllocSize);

    if(pOvl == NULL)
    {
        // Out of memory.
    }

    // Initialize the allocation.
    ZeroMemory(pOvl, dwAllocSize);


    // Build the pointers.
    LPWSABUF pBuffer = (LPWSABUF)((DWORD_PTR)pOvl + sizeof(OVERLAPPED));
    pBuffer->len = dwLength;
    assert(pBuffer->len < 1000000);
    pBuffer->buf = (PCHAR)((DWORD_PTR)pBuffer + sizeof(WSABUF));
    // When you have a completion routine, the hEvent member is ignored, so we
    // can use it to pass our this pointer to the completion routine.
    pOvl->hEvent = (PVOID)this;

    // Copy the data to the buffer.
    CopyMemory(pBuffer->buf, pData, dwLength);

    // Send the data.
    DWORD dwSent = 0;
    int iResult = WSASend(
        m_hSocket,          // The socket.
        pBuffer,            // The WSABUF struct.
        1,                  // Number of structs (1).
        &dwSent,            // Bytes sent. Updated if it happens synchronously.
        0,                  // No flags.
        pOvl,               // The overlapped struct.
        SendCompletion);    // Completion routine.

    if(iResult == NO_ERROR)
    {
        // If the send happened synchronously, we can go ahead and delete the
        // memory that we allocated.

        // TODO: look at bytes transferred, and if they're less than the total
        // then issue another send with the remainder.

        if(HasOverlappedIoCompleted(pOvl))
        {
            // If I actually free this here, the completion routine gets garbage.
            //free(pOvl);
            m_dwSyncSend++;
        }
        else
        {
            m_dwAsyncSend++;
        }
    }
    else
    {
        // If we got WSA_IO_PENDING, then that just means the completion routine
        // will take care of it.
        if(iResult != WSA_IO_PENDING)
        {
            Disconnect();
            tracef(TT_REGULAR, 1,
                "SOCKET_ERROR in CNetAsyncSocket::Send(), disconnecting, error code: %ld, on socket: %s:%ld",
                iResult, GetIP(), GetPort());
            // Don't need the payload anymore.
            free(pOvl);
        }
        else
        {
            m_dwAsyncSend++;
        }
    }
}

Answer 1

The documentation says that you'll get 0 if the operation could be completed immediately and SOCKET_ERROR (with a last error code of WSA_IO_PENDING) if it couldn't. Either way, a call to the completion routine will be queued.

So the behaviour you're describing is as-expected, and the only case when you should free the buffer is if an error other than WSA_IO_PENDING occurred.