Control a separate process in python using multiprocessing module

Question

I am wondering what is the easiest way to be able to control a process started by the main function in python.

For example, in my main loop I call a function that will be a separate process and its only purpose is to collect data to a buffer. Then whenever I want, I will indicate to that process to stop collecting data and to store it in a text file. Then, when it completes writing to file I would like it to wait for another signal (coming from main) before it starts the same loop again, that is, collecting new data to a buffer. The process will repeat indefinitely, though it would be awesome if I had the ability to actually stop the process until I want new data. I tried using multiprocessing.Event() but for some reason when I event.set() or event.clear() the message sometimes isn't received in time and thus the data formatting is screwed up.

Example:

def separateProcess():
    datBuffer = []
    while True:
        datBuffer.append(collectData(sample))
        if signal.recv == 'TimeToWriteToFile':
            #Write the data buffer to file.
            while True:
                if signal.recv == 'NewData':
                    #Signal to begin recording new data has been received
                    datBuffer = [] #clear the buffer for new data.
                    break
        else:
            #Continue recording Data.
            pass

def main():
    #This code will do some stuff regarding the experiment.
    p = mp.Process(target=separateProcess)
    p.start()


    #Based on a particular event I will send the signal when needed.
    if experiment == 'Success':
        sendToProc('TimeToWriteToFile') #Theoretical signal to the other process.
        sleep(10) #Wait for X amount of seconds then begin recording new data.

        sendToProc('NewData')

I can provide the code sample of my failed attempt at creating such script if needed. But basically I wish to know a method to achieve what I have up there, it would also be awesome if that method worked using global variables as signals. I know I can't since a new process does not share global state...

That's all.

Answer 1

Your code looks pretty good. I suggest creating a Queue in the parent process, then sending anything to your worker, which will output the data. When the parent proc wants the worker to die, send a None.

source

import multiprocessing, Queue

def myproc(arg):
    return arg*2

def worker(inqueue):
    for num in iter(inqueue.get, None):
        print myproc(num)


inq = multiprocessing.Queue()
# prefill with 3 jobs
for num in range(3):
    inq.put(num)
# signal end of jobs
inq.put(None)

worker_p = multiprocessing.Process(
    target=worker, args=(inq,),
)
worker_p.start()

worker_p.join()

output

0
2
4