IOCP multiple socket completionports in same container

Question

For the past couple of days I have been thinking about how to solve one of my problems I am facing, and I have tried to research the topic but don't really know what I can do.

I have 2 sockets in the same struct that both have the same completionport. Problem is, they both use different protocols. Is there a way that I can find out which socket got triggered? Their called game_socket, and client_socket

Example code would be something like...

while (true) {
error = GetQueuedCompletionStatus(CompletionPort, &BytesTransfered, (PULONG_PTR)&Key, &lpOverlapped, 0);
srvc = CONTAINING_RECORD ( lpOverlapped, client , ol );
if ( error == TRUE ) {
cout << endl << "SOCKET: [" << srvc->client_socket << "] TRIGGERED - WORKER THREAD" << endl;
cout << endl << "BytesTransfered: [" << BytesTransfered << "]" << endl;

if ( srvc->game_client triggered ) {
// .. this code
} else {

// .. this code
}

Any ideas or help would be appreciated :)

Answer 1

The overlapped extension structure you're using is ideally suited for a specific connection instance. That structure, for example should not only contain the socket the operation is being performed on, but the action as well. For example, an IO completion on a socket that could be reading or writing should reflect which state was present at the time the IO request was started. Most examples of IOCP-based client/server code will attest to this general ideology.

In your case, you should NOT be using the same OVERLAPPED structure for two different sockets with two potentially independent actions. If the sockets are 'related' in some sense then that should be tracked and maintained outside of your OVERLAPPED structure. Each should have its own distinct structure for its own distinct IO completions. Anything else is honestly asking for a headache.

All that being said, it doesn't appear your use case can be far off from transforming your current scheme to accommodate what I'm describing. I'm always reluctant to give "you shouldn't do it that way" answers, as i totally hate hearing them myself, but it is none-the-less likely warranted in this case.

All the best.

Answer 2

When using IOCP you have two items of user data that you can use per asynchronous operation.

The first is the completion key which is "per connection" data and is set when you associate the file handle or socket with the completion port by calling CreateIoCompletionPort() with the socket/file handle and an existing completion port. This value is returned on every call to GetQueuedCompletionStatus() for a given connection.

The second is the 'extended overlapped' structure which is "per operation" data. Each concurrent operation MUST have a unique overlapped structure assigned for it.

In your design above you should be using the "per connection" data to identify your connection. You have two connections (sockets) that are linked, so I would have two linked classes, one for each side of the connection. The class (or struct if you're using C rather than C++) would hold the socket, a pointer to the other half of the connection and a flag that indicates which type of connection it is. If using C++ these two classes would derive from a common base class so that you could call a member function to determine which type of connection the completion key represents. If using C use a discriminated union.

You then simply cast your completion key to the base, then determine which connection type you have and cast to the correct class/struct. You now know which connection the operation was performed on. The connection class can store all of the state for the state machine that runs its protocol.

As Criag mentions, each overlapped structure is "unique per operation" and in my IOCP code this tends to allow access to the data buffer for the operation and a flag that tells me what the operation was (read/write/accept/connect, etc).

One potential piece of complexity is keeping the per operation and per connection data alive until all outstanding operations that use it are complete. I personally favour reference counting of both the per connection and per operation data, but I'm sure there are other ways.

I have some example code for IOCP that uses these concepts (though doesn't provide a 'two connection' example), you can download it from here: http://www.serverframework.com/products---the-free-framework.html