How can you profile a parallelized Python script?

Question

Suppose I have a python script called my_parallel_script.py that involves using multiprocessing to parallelize several things and I run it with the following command:

python -m cProfile my_parallel_script.py

This generates profiling output for the parent process only. Calls made in child processes are not recorded at all. Is it possible to profile the child processes as well?

If the only option is to modify the source, what would be the simplest way to do this?

Answer 1

cProfile only works within a single process, so you will not automatically get the child process profiled.

I would recommend that you tweak the child process code so that you can invoke it separately as a single process. Then run it under the profiler. You probably don't need to run your system multi-process while profiling, and it will simplify the job to have only a single child running.