Is it possible to change HANDLE that has been opened for synchronous I/O to be opened for asynchronous I/O during its lifetime?

Question

Most of my daily programming work in Windows is nowadays around I/O operations of all kind (pipes, consoles, files, sockets, ...). I am well aware of different methods of reading and writing from/to different kinds of handles (Synchronous, asynchronous waiting for completion on events, waiting on file HANDLEs, I/O completion ports, and alertable I/O). We use many of those.

For some of our applications it would be very useful to have only one way to treat all handles. I mean, the program may not know, what kind of handle it has received and we would like to use, let's say, I/O completion ports for all.

So first I would ask:

Let's assume I have a handle:

HANDLE h;

which has been received by my process for I/O from somewhere. Is there any easy and reliable way to find out what flags it has been created with? The main flag in question is FILE_FLAG_OVERLAPPED.

The only way known to me so far, is to try to register such handle into I/O completion port (using CreateIoCompletionPort()). If that succeeds the handle has been created with FILE_FLAG_OVERLAPPED. But then only I/O completion port must be used, as the handle can not be unregistered from it without closing the HANDLE h itself.

Providing there is an easy a way to determine presence of FILE_FLAG_OVERLAPPED, there would come my second question:

Is there any way how to add such flag to already existing handle? That would make a handle that has been originally open for synchronous operations to be open for asynchronous. Would there be a way how to create opposite (remove the FILE_FLAG_OVERLAPPED to create synchronous handle from asynchronous)?

I have not found any direct way after reading through MSDN and googling a lot. Would there be at least some trick that could do the same? Like re-creating the handle in same way using CreateFile() function or something similar? Something even partly documented or not documented at all?

The main place where I would need this, is to determine the the way (or change the way) process should read/write from handles sent to it by third party applications. We can not control how third party products create their handles.

Dear Windows gurus: help please!

With regards

Martin

Answer 1

3 years passed and Windows 8 has been released. Thanks to the regression introduced in implementation of console in Windows 8 I got to do something about the issue which triggered this question. So I have finally tried to use ReOpenFile() function call.

In one sentence: for my purposes it is useless.

The ReOpenFile() API is used for “taking an existing file handle and getting another handle that has a different set of access rights”. At least that is stated in the original article.

I tried to use The ReOpenFile() on Console input handle:

  stdin_in = GetStdHandle(STD_INPUT_HANDLE);
  stdin_in_operlapped = ReOpenFile(stdin_in, GENERIC_READ | GENERIC_WRITE,
                                   FILE_SHARE_READ, FILE_FLAG_OVERLAPPED);
  if (stdin_in_operlapped ==  INVALID_HANDLE_VALUE)
    {
      my_debug("failed to ReOpen stdin handle with OVERLAPPED flag: %d", GetLastError());
      exit(1);
    }

And what I get is: error 1168: “Element Not Found”. “Thank you Microsoft”. I will not even try to use it for anonymous pipes, since the documentation states:

“Asynchronous (overlapped) read and write operations are not supported by anonymous pipes. This means that you cannot use the ReadFileEx and WriteFileEx functions with anonymous pipes. In addition, the lpOverlapped parameter of ReadFile and WriteFile is ignored when these functions are used with anonymous pipes.”

Thank you, people, all for your suggestions. When reading from handle asynchronously, one has to be ready also for the fact that the operation may complete synchronously. That I know. The main reason I was asking this question is:

when a synchronous read has been issued on some objects (at least anonymous pipes, and console input in Windows 8) then calling CloseHandle() from another thread on the same handle will either fail, or hang, until the ReadFile() completes; that means it will hang indefinitely in many cases. That is the reason why I wanted to replace the synchronous handle with asynchronous.

Now it is clear to me that it is simply not possible in Windows operating systems to cancel some read operations in straightforward way. When reading from synchronous handles, one just has to exit from application even if ReadFile() is still reading from a handle in some thread, because it is simply impossible to reliably wake up such a read operation. In know... In newer OS it is possible to cancel that operation. However, there is no way to know weather the thread is in the ReadFile() call already, or not yet. If ReadFile() is not called yet then there is no operation to cancel and subsequent read will hang. The only way would be to close the handle, but that operation hangs, or fails on some objects and on some operating systems. The only proper solution to this is asynchronous I/O. But, as I mentioned in the beginning, our APP is started by third party apps and we can't force them all to always create named pipes with overlapped flag set for the stdio.

I am giving up and going to implement nasty ugly hacks... we will have to keep reading without OVERLAPPED structure from HANDLEs that have been created with OVERLAPPED flag, and leaking handles and threads....

Answer 2

I see I was a bad reader of MSDN :/ I totally missed function ReOpenFile() which has been introduced probably as early as in June 2003 in Windows Server 2003 (according to this article). To defend myself at least a little: I would expect the CreateFile() description to cross reference to ReOpenFile() description. There is a reference on ReOpenFile() page to CreateFile() page, but not the other way round.

This function seems to enable precisely what I need: adding or removing of FILE_FLAG_OVELRAPPED to/from already existing handles by creating new handle with desired properties! :-D I have not tested it yet, though. It is, unfortunately, available only on Windows 2003 Server, Windows Vista and onwards. Question about the previous OS versions has been answered here. The function does not exist in public API in OS prior to Windows 2003 Server. It is used by the underlying implementation, but it is not available to developers on those systems (Not supported).

That practically means that there is no hope for me at least for next few years, until we drop support for older Windows platforms. It also means that the situation regarding the I/O has been REALLY bad on OS older then Windows Vista. Other painful part that has been missing altogether was the possibility to cancel synchronous and asynchronous I/O on those older systems.

Furthermore, I still miss one part of the answer: can the presence of flags be tested by any means? I have not found function for doing that. That means that if we want to guarantee presence of some flag in file object, then the file always has to be re-opened.

Answer 3

If I understand what you're after, I would like to suggest that you don't care if was opened with the overlapped flag or not. I believe that you can safely pass in an OVERLAPPED structure in both the synchronous and asynchronous cases. Your code needs to be able to handle ReadFile() returning false and GetLastError() returning ERROR_IO_PENDING. You'll also need to add appropriate calls to GetOverlappedResult(), WaitForSingleObject(), etc.

The MSDN article on ReadFile() has some good info on this under "Considerations for working with synchronous file handles", and "Considerations for working with asynchronous file handles" in the "Synchronization and File Position" section.

Answer 4

Testing handle flags should probably be done the same way as testing the permissions the handle was created with. Try it. If the API fails, try the fallback. If that fails, return an error.

I think the really telling thing is the way the documentation for ReadFile says "If hFile is opened with FILE_FLAG_OVERLAPPED, ... function can incorrectly report that the read operation is complete."

My interpretation of the error is, (and the question you need to Ask yourself is): if it was possible to check the overlapped status of a file handle, why would ReadFile not do that check and then validate the OVERLAPPED structure accordingly, to explicitly fail if called in a non overlapped way with an overlapped handle?

Answer 5

I don't know a way to determine the flag of an handle and the side-effects of using ReOpen api, but since your goal was

it would be very useful to have only one way to treat all handles

If you desire synchronous behavior (i mean using synchronous api for non overlapped handles and async api fed with OVERLAPPED structure with subsequent wait on the overlapped's event) you can always use the async api also if the handle was opened in non overlapped mode as already stated by @Brett
I can confirm that this works (AT LEAST for named pipes) es:

void connectSynchronous(HANDLE hPipeThatWeDontKnowItsFlag){
    ...
    BOOL bRet = ::ConnectNamedPipe(hPipeThatWeDontKnowItsFlag, pOverlapped);

    if(bRet == FALSE){
        DWORD dwLastErr = ::GetLastError();

        if(dwLastErr == ERROR_IO_PENDING){
            //The handle was opened for asynchronous IO so we have to wait for the operation
            ...waitFor on the overlapped hEvent;

        }else if(dwLastErr == ERROR_PIPE_CONNECTED){
            //The handle was opened for synchronous IO and the client was already connected before this call: that's OK!
            return;
        }else{
            throw Error(dwLastErr);
        }
    }/*else{
        //The handle was opened for synchronous IO and the client has connected: all OK
    }*/
}

Answer 6

An alternative to the CreateIoCompletionPort hack is to do a zero byte ReadFile with a NULL lpOverlapped. If it fails with ERROR_INVALID_PARAMETER assume it was opened with FILE_FLAG_OVERLAPPED.