SetFileCompletionNotificationModes seems to not work properly

Question

I'm using SetFileCompletionNotificationModes() API to optimize my I/O completion ports loops, but it seems to not work properly. Even if I set FILE_SKIP_COMPLETION_PORT_ON_SUCCESS for sockets and HANDLEs, an I/O port's completion callback is still called also if ReadFile() WriteFile() WSARecv() WSASend() do return synchronosly.

I'm sure that the 3 conditions that MSDN says must be true (A completion port is associated with the file handle, The file is opened for asynchronous I/O, A request returns success immediately without returning ERROR_PENDING) are met, and are all true, so why I still receive I/O completion calls?

When i call SetFileCompletionNotificationModes() it returns success, so no errors or whatsoever, and the system is Windows 7.

How I can replicate a scenario when after I have activated SetFileCompletionNotificationModes() on my socket/HANDLEs, I can clearly see that the I/O completion callback won't be called?

I guessed that it happens when I write few bytes on a socket, since the socket's buffer is quite bigger, I didn't full it, so another write of another few bytes shouldn't block, since there are still a lot of room in the buffer, so it should return immediately, but not with ERROR_IO_PENDING, just in a synchronous way, right? (more or less, in a similar way of unix EWOULDBLOCK/EAGAIN: when i call write() for few bytes it returns immediately, and doesn't return EAGAIN, because there is still space in the write buffer).

Well it doesn't do that. Even for writing multiple times few bytes on a socket, it still calls the I/O completion callback, avoding the benefits of setting FILE_SKIP_COMPLETION_PORT_ON_SUCCESS

Am I missing something important? Any clue about that?

NOTE: I know that this wont work if the socket is only compatible with Layered Service Providers (LSP) that return Installable File Systems (IFS) handles, but that's not my case, it should work. Btw I'm trying this also with pipes and files.

Shouldn't files never call I/O completion callbacks because they never block, just like in unix read() and write() calls on local files never returns EWOULDBLOCK/EAGAIN, so ReadFile() and WriteFile() with an handle set with FILE_SKIP_COMPLETION_PORT_ON_SUCCESS should return immediately?

Answer 1

A network write completion is only generated when the data buffer that you have provided is no longer required by the network stack. It's somewhat hard to reason about when this will be and it's also somewhat irrelevant and not something to worry about. When you issue an overlapped write with FILE_SKIP_COMPLETION_PORT_ON_SUCCESS set then your write operation will return 0 if and only if the completion has occurred synchronously. Write code to handle that case correctly (it's the same code that you need for the recv side) and forget about it. You'll get the performance and context switching advantages when it's possible and your code with work correctly when it isn't.

As for file system access, that depends on the file system drivers and, likely the actual hardware. See here for some information about how hard it can be to get some hardware to do async file writes at all, ever. Then note that when I switched the test that I talk about there from a workstation with 'normal' SATA disks to a server with hardware raid everything was different and all of the writes were always fully async...

Answer 2

Make sure you validate the 4th condition mentioned in the documentation:

When the FileHandle parameter is a socket, this mode is only compatible with Layered Service Providers (LSP) that return Installable File Systems (IFS) handles. To detect whether a non-IFS LSP is installed, use the WSAEnumProtocols function and examine the dwServiceFlag1 member in each returned WSAPROTOCOL_INFO structure. If the XP1_IFS_HANDLES (0x20000) bit is cleared then the specified LSP is not an IFS LSP. Vendors that have non-IFS LSPs are encouraged to migrate to the Windows Filtering Platform (WFP).

Also read this MSDN Support ticket:

SetFileCompletionNotificationModes API causes an I/O completion port not to work correctly if a non-IFS LSP is installed