Python MultiProcess, Logging, Various Classes

Question

I am currently trying to log to a single file from multiple processes but I am having a lot of trouble with it. I have spend countless hours looking online -- stackoverflow and Google, but have come up with nothing concrete.

I have read: How should I log while using multiprocessing in Python?

I've been trying to use zzzeek's code but it does not write to the file for me. I don't have a specific way I'm doing it -- I've just been trying every way I can.

Have any of you got it to work and have sample code, or do you have an alternative way of doing it. I need to log multiple processes to the same file. I would also like to log any errors from various class to the same file. I, however, am satisfied with simple getting the multiprocess one to work.

Thanks

Answer 1

Look at these posts:

1. Using logging with multiprocessing
2. Improved QueueHandler, QueueListener: dealing with handlers that block
3. logutils: Using recent logging features with older Python versions

Answer 2

Here's some sample code that works with zzzeek's handler:

mtlog = MultiProcessingLog('foo.log', 'a', 0, 0)
logging.getLogger().addHandler(mtlog)

def task(_):
    logging.error('Hi from {}'.format(os.getpid()))

p = multiprocessing.Pool()
p.map(task, range(4))

Here's my running it:

$ python mtlog.py
$ cat foo.log
Hi from 6179
Hi from 6180
Hi from 6181
Hi from 6182

In fact, any trivial test I come up with works just fine. So clearly, you're doing something wrong, probably the same thing, in all of your attempts.

---

My first guess is that you're trying to use it on Windows. As Noah Yetter's comment says:

Unfortunately this approach doesn't work on Windows. From docs.python.org/library/multiprocessing.html 16.6.2.12 "Note that on Windows child processes will only inherit the level of the parent process’s logger – any other customization of the logger will not be inherited." Subprocesses won't inherit the handler, and you can't pass it explicitly because it's not pickleable.

Although zzzeek replies that he thinks it'll work, I'm 90% sure he's wrong:

I think that only refers to the logger that's hardwired into the multiprocessing module. This recipe isn't making any usage of that nor should it care about propagation of loglevels, it just shuttles data from child to parent using normal multiprocessing channels.

That's exactly backward. Propagation of log levels does work; propagation of addHandler does not.

To make this work, you'd need to pass the queue explicitly to the children, and build the child-side logger out of that.