Logging worker processes with Parallel Python

Question

I've inherited the maintenance of some scientific computing using Parallel Python on a cluster. With Parallel Python, jobs are submitted to a ppserver, which (in this case) talks to already-running ppserver processes on other computers, dishing tasks out to ppworkers processes.

I'd like to use the standard library logging module to log errors and debugging information from the functions that get submitted to a ppserver. Since these ppworkers run as separate processes (on separate computers) I'm not sure how to properly structure the logging. Must I log to a separate file for each process? Maybe there's a log handler that would make it all better?

Also, I want reports on what process on what computer has hit an error, but the code I'm writing the logging in probably isn't aware of these things; maybe that should be happening at the ppserver level?

(Version of the question cross-posted on Parallel Python Forums, I'll post an answer here if I get something there about this from a non SO user)

Answer 1

One way to solve your problem is to do the following:

1. In each worker process, use a logging.handlers.SocketHandler to send events from the worker to a dedicated logger process.
2. Create a dedicated logger process which listens for logging events on a socket, based on the working example given in the docs at https://docs.python.org/3/howto/logging-cookbook.html#sending-and-receiving-logging-events-across-a-network
3. Profit ;-)

If you catch exceptions in your worker functions and log them, then you should be able to get visibility of errors across all workers in one place.

Answer 2

I'd use Python's logging and socket APIs. Just follow the example here.

Simply start a ppworker dedicated to logging somewhere, and create a new logging.Logger in each of the other workers with a logging.SocketHandler specifying the hostname and port of the machine running the logging ppworker.

If you have a syslog server running, you can also use Python's syslog module, which is documented here.