How to do local IPC without leaking handles (cross platform)?

Question

How can I initiate IPC with a child process, without letting it inherit all handles? To make it more interesting, this shoud work on windows as well as unix.

The background: I am writing a library that interfaces with a 3rparty shared library (let's just call it IT) which in turn contains global data (that really should be objects!). I want to have multiple instances of this global data. As far as I understand, I have two options to solve this:

1. create a cython module that links against a static variant of IT, then copy and import the module whenever I want a new instance. Analogously, I could copy IT but that's even more work to create a ctypes interface.

2. spawn a subprocess that loads IT and establish an IPC connection to it.

There are a few reasons to use (2):

• I am not sure, if (1) is reliable in any way and it feels like a bad idea (what happens with all the extra modules, when the application exits in an uncontrolled way?).

• boxing IT into a separate process might actually be a good idea anyway for security considerations: IT deals with potentially unsafe input and IT's code quality isn't overly good. So, I'd rather not have any secure resources open when running it.

• there is probably lot's of need for this kind of IPC in future applications

So what are my options? I have already looked into:

• multiprocessing.Process at first looked nice, until I realized that the new process gets a copy of all my handles. Needless to say that this is quite problematic, since now resources cannot be reliably freed by closing them in the parent process + the security issues mentioned earlier.

• Use os.closerange within a multiprocessing.Process to close to all handles manually - except for the Pipe I'm interested in. Does os.closerange close only files or does it take care of other types of resources as well? If so: how can I determine the range, given the Pipe object?

• subprocess.Popen(.., close_fds=True, stdin=PIPE, stdout=PIPE) works fine on unix but isn't possible on win32.

• Named pipes are very different on win32 and unix. Are their any libraries that their usage?

• Sockets. Promising, especially since their are handy RPC libraries that can work with sockets. On the other hand, I fear that this may cause a whole bunch of security issues. Are sockets that I have determined to be of local origin (sock.getpeername()[0] == '127.0.0.1') secure against tempering?

Are there any possibilities that I have overlooked?

To round up: the main question is how to establish a secure IPC with a child process on windows+unix? But please don't hesitate to answer if you know any answers to only partial problems.

Thanks for taking the time to read it!

Answer 1

It seems on python>=3.4 subprocess.Popen(..., stdin=PIPE, stdout=PIPE, close_fds=False) is a possible option. This is due to a patch that makes all opened file descriptors non-inheritable by default. To be more precise, they will be automatically closed on execv (so still can't use multiprocessing.Process), see PEP 446.

This is also a valid option for other python versions:

• on windows, HANDLEs are created non-inheritable by default, so you will leak only handles that were made inheritable explicitly
• on POSIX/python<=3.3 you can still use os.closerange to close open file descriptors after spawning the subprocess

for a corresponding example see:

https://github.com/coldfix/python-ipc-test

The most useful combinations are:

1. stdio:pickle

   • pro: completely cross-platform in my tests
   • pro: fastest option (with 2)
   • con: stdin/stdout can not be redirected independently

2. inherit_unidir:pickle

   • pro: you can redirect STDIO streams independently
   • pro: fastest option together with stdio:pickle
   • con: very low level platform specific code

3. socket:sockpipe

   • pro: cross-platform with little effort
   • con: there is a short period when "attackers" may connect to the port, you could require a pass-phrase or something to prevent that from happening
   • con: slightly slower than alternatives on windows (factor 1.6 in my measurements)
   • when not using AF_UNIX there are unpredictable performance hits on linux