set named pipe message mode

Question

I have the following piece of code:

//...
SafeFileHandle handle = NativeMethods.CreateFile(pipeName, FileAccess.ReadWrite, FileShare.None, IntPtr.Zero, FileMode.Open, FileAttributes.Normal, IntPtr.Zero);
if (handle.IsInvalid)
    continue;
uint mode = (uint)PipeMode.ReadModeMessage; // mode==2
bool result = NativeMethods.SetNamedPipeHandleState(handle, ref mode, IntPtr.Zero, IntPtr.Zero);
//...

The problem is that the call to 'SetNamedPipeHandleState' fails: result is false and GetLastError() returns 5 (ERROR_ACCESS_DENIED). Other than that the pipe works just fine - I can read and write data. Except, of course, that it is not working in message mode - eg. contents of two WriteFile message calls gets read by single ReadFile call. What am I doing wrong here?

Answer 1

CreateFile opens the client end of a named pipe already created by something else acting as the pipe server. You don't tell us in the question how your pipe is created. The pipe will only work in message mode if the PipeMode was specified as PIPE_TYPE_MESSAGE when the pipe was created by the pipe server.

If the pipe is created in message mode, then the pipe client can choose whether to read in message mode or in byte mode. If the pipe isn't in message mode, no attempt by a pipe client to set the read mode to message (PIPE_READMODE_MESSAGE) will have any effect, as it won't change the pipe mode.

You also don't show us how you have implemented your NativeMethods, but if your CreateFile parameters are mapped directly to the arguments of the Win32 CreateFile function, you are only requesting FILE_READ_DATA and FILE_WRITE_DATA access rights for your pipe handle. These rights are not sufficient to allow you to call SetNamedPipeHandleState, which explains the access denied error. See the Win32 API documentation:

The handle must have GENERIC_WRITE access to the named pipe for a write-only or read/write pipe, or it must have GENERIC_READ and FILE_WRITE_ATTRIBUTES access for a read-only pipe.