How to pass user-defined data to a worker thread using IOCP?

Question

Hey... I created a small test server using I/O completion ports and winsock. I can successfully connect and associate a socket handle with the completion port. But I don´t know how to pass user-defined data-structures into the wroker thread...

What I´ve tried so far was passing a user-structure as (ULONG_PTR)&structure as the Completion Key in the association-call of CreateIoCompletionPort() But that did not work.

Now I tried defining my own OVERLAPPED-structure and using CONTAINING_RECORD() as described here http://msdn.microsoft.com/en-us/magazine/cc302334.aspx and http://msdn.microsoft.com/en-us/magazine/bb985148.aspx. But that does not work, too. (I get freaky values for the contents of pHelper)

So my Question is: How can I pass data to the worker thread using WSARecv(), GetQueuedCompletionStatus() and the Completion packet or the OVERLAPPED-strucutre?

EDIT: How can I successfully transmit "per-connection-data"?... It seems like I got the art of doing it (like explained in the two links above) wrong.

Here goes my code: (Yes, its ugly and its only TEST-code)

struct helper
    {
        SOCKET m_sock;
        unsigned int m_key;
        OVERLAPPED over;
    };


///////

SOCKET newSock = INVALID_SOCKET;
    WSABUF wsabuffer;
    char cbuf[250];
    wsabuffer.buf = cbuf;
    wsabuffer.len = 250;
    DWORD flags, bytesrecvd;


    while(true)
    {
        newSock = accept(AcceptorSock, NULL, NULL);
        if(newSock == INVALID_SOCKET)
            ErrorAbort("could not accept a connection");

        //associate socket with the CP
        if(CreateIoCompletionPort((HANDLE)newSock, hCompletionPort, 3,0) != hCompletionPort)
            ErrorAbort("Wrong port associated with the connection");
        else
            cout << "New Connection made and associated\n";

        helper* pHelper = new helper;
        pHelper->m_key = 3;
        pHelper->m_sock = newSock;
        memset(&(pHelper->over), 0, sizeof(OVERLAPPED));
        flags = 0;
        bytesrecvd = 0;

        if(WSARecv(newSock, &wsabuffer, 1, NULL, &flags, (OVERLAPPED*)pHelper, NULL) != 0)
        {
            if(WSAGetLastError() != WSA_IO_PENDING)
                ErrorAbort("WSARecv didnt work");
        }
    }

    //Cleanup
    CloseHandle(hCompletionPort);
    cin.get();
    return 0;
}

DWORD WINAPI ThreadProc(HANDLE h)
{
    DWORD dwNumberOfBytes = 0;
    OVERLAPPED* pOver = nullptr;
    helper* pHelper = nullptr;
    WSABUF RecvBuf;
    char cBuffer[250];
    RecvBuf.buf = cBuffer;
    RecvBuf.len = 250;
    DWORD dwRecvBytes = 0;
    DWORD dwFlags = 0;
    ULONG_PTR Key = 0;

    GetQueuedCompletionStatus(h, &dwNumberOfBytes, &Key, &pOver, INFINITE);

    //Extract helper
    pHelper = (helper*)CONTAINING_RECORD(pOver, helper, over);


    cout << "Received Overlapped item" << endl;
    if(WSARecv(pHelper->m_sock, &RecvBuf, 1, &dwRecvBytes, &dwFlags, pOver, NULL) != 0)
        cout << "Could not receive data\n";
    else
        cout << "Data Received: " << RecvBuf.buf << endl;

    ExitThread(0);
}

Answer 1

You can send special-purpose data of your own to the completion port via PostQueuedCompletionStatus.

The I/O completion packet will satisfy an outstanding call to the GetQueuedCompletionStatus function. This function returns with the three values passed as the second, third, and fourth parameters of the call to PostQueuedCompletionStatus. The system does not use or validate these values. In particular, the lpOverlapped parameter need not point to an OVERLAPPED structure.

Answer 2

If you pass your struct like this it should work just fine:

helper* pHelper = new helper;
CreateIoCompletionPort((HANDLE)newSock, hCompletionPort, (ULONG_PTR)pHelper,0);
...


helper* pHelper=NULL;
GetQueuedCompletionStatus(h, &dwNumberOfBytes, (PULONG_PTR)&pHelper, &pOver, INFINITE);

---

Edit to add per IO data:

One of the frequently abused features of the asynchronous apis is they don't copy the OVERLAPPED struct, they simply use the provided one - hence the overlapped struct returned from GetQueuedCompletionStatus points to the originally provided struct. So:

struct helper {
  OVERLAPPED m_over;
  SOCKET     m_socket;
  UINT       m_key;
};

if(WSARecv(newSock, &wsabuffer, 1, NULL, &flags, &pHelper->m_over, NULL) != 0)

Notice that, again, in your original sample, you were getting your casting wrong. (OVERLAPPED*)pHelper was passing a pointer to the START of the helper struct, but the OVERLAPPED part was declared last. I changed it to pass the address of the actual overlapped part, which means that the code compiles without a cast, which lets us know we are doing the correct thing. I also moved the overlapped struct to be the first member of the struct.

To catch the data on the other side:

OVERLAPPED* pOver;
ULONG_PTR key;
if(GetQueuedCompletionStatus(h,&dw,&key,&pOver,INFINITE))
{
  // c cast
  helper* pConnData = (helper*)pOver;

On this side it is particularly important that the overlapped struct is the first member of the helper struct, as that makes it easy to cast back from the OVERLAPPED* the api gives us, and the helper* we actually want.

Answer 3

I use the standard socket routines (socket, closesocket, bind, accept, connect ...) for creating/destroying and ReadFile/WriteFile for I/O as they allow use of the OVERLAPPED structure.

After your socket has accepted or connected you should associate it with the session context that it services. Then you associate your socket to an IOCP and (in the third parameter) provide it with a reference to the session context. The IOCP does not know what this reference is and doesn't care either for that matter. The reference is for YOUR use so that when you get an IOC through GetQueuedCompletionStatus the variable pointed to by parameter 3 will be filled in with the reference so that you immediately find the context associated with the socket event and can begin servicing the event. I usually use an indexed structure containing (among other things) the socket declaration, the overlapped structure as well as other session-specific data. The reference I pass to CreateIoCompletionPort in parameter 3 will be the index to the structure member containing the socket.

You need to check if GetQueuedCompletionStatus returned a completion or a timeout. With a timeout you can run through your indexed structure and see (for example) if one of them has timed out or something else and take appropriate house-keeping actions.

The overlapped structure also needs to be checked to see that the I/O completed correctly.

The function servicing the IOCP should be a separate, multi-threaded entity. Use the same number of threads that you have cores in your system, or at least no more than that as it wastes system resources (you don't have more resources for servicing the event than the number of cores in your system, right?).

IOCPs really are the best of all worlds (too good to be true) and anyone who says "one thread per socket" or "wait on multiple-socket list in one function" don't know what they are talking about. The former stresses your scheduler and the latter is polling and polling is ALWAYS extremely wasteful.