Multiprocess with Serial Object as Parameter

Question

I'm having a problem with Python and passing a serial object as a parameter to a separate process. The program is being run in Windows 8, so using global variables isn't an option.

from multiprocessing import Queue
from multiprocessing import Process
import os
import serial
from serial.tools import list_ports
from time import sleep

displayMessages = Queue()
modemPort = None

def processDisplayMessages(displayMessages):
    while True:
        msg = displayMessages.get()  #should halt until message in queue
        print msg

def processIncomingSerialMessages(modemPort, displayMessages):
    while True:
        line = modemPort.readline()
        displayMessages.put(line)

def main():
    print "Serial Send Test"
    Process(target=processDisplayMessages, args = (displayMessages,)).start()
    modemPort = serial.Serial('COM5', 57600, timeout=0.9)  # open first serial port
    Process(target=processIncomingSerialMessages, args = (modemPort, displayMessages)).start()
    print "Back from launch"

    sleep(0.1)

 if __name__ == '__main__':
    main()

When the program runs I get the following error:

Process Process-2:
Traceback (most recent call last):
  File "c:\python27\lib\multiprocessing\process.py", line 258, in _bootstrap
    self.run()
  File "c:\python27\lib\multiprocessing\process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\matthew.parets\Documents\..Development\RaspberryPi\windows\seri
alRecvPrototype.py", line 27, in processIncomingSerialMessages
    line = modemPort.readline()
  File "c:\python27\lib\site-packages\serial\serialwin32.py", line 246, in read
    if not self.hComPort: raise portNotOpenError
AttributeError: 'Serial' object has no attribute 'hComPort'

If I place the open for the serial port (modemPort) as the first line of processIncomingSerialMessages the program works fine. The issue is that I need to decouple input and output from the modem so I need to pass the serial object as a parameter. And Python doesn't seem to like that.

Can anyone see the mistake I am making, or can anyone suggest an alternative?

Answer 1

I can't run this code, but I'd be amazed if it worked: an argument passed across processes works by pickling the argument object on the sending side, sending the pickle string across processes via a pipe or a socket, and unpickling that string on the receiving side. I know of no case of any open I/O-kind-of-object for which this can possibly work (files, sockets, pipes ...). I/O-kinds-of-objects don't just have internal data state, they're also connected to resources that Python itself doesn't implement. A pickle is just a stream of raw bytes.

You already figured out that you have to open the serial port in the worker process. Alas, I don't know what "I need to decouple input and output from the modem" means, so it's hard to suggest a workaround. But I'm sure you'll figure it out if you accept what you've already discovered the hard way: passing an open serial object across processes is never going to work.

That said, you could dig into the various pickling protocols and build your own class with custom pickling/unpickling code that (re)opened a serial object when unpickled. That would be an elaborate way to hide what would otherwise be straightforward code to (re)open a serial object in worker processes.

EDIT: Q&A

And again windows doesn't offer the easy way out of global variables, so I'm stuck with a single process handing both send and receive.

A "global variable" probably wouldn't help you. I assume you have fork() in mind, but nothing is shared across fork() either: child processes see copies of the parent process's address space. I/O gimmicks often fail to work correctly then too.

Does Python provide a way to pass a value by reference, or a reference value to a process? I have tried add the serial object to lists and sets with the same results. Again, does Python provide something like an Object or an Array from Java where I could get a reference through without it being "Pickled"?

There's a very high wall between processes in all modern OSes. To get anything truly shared across processes you need to use types built from the ground up for that, or use the OS's "shared memory" facilities.

You can read the docs for multiprocessing.sharedctypes, which supplies ways to use shared (across processes) memory. But, as the docs warn:

Note Although it is possible to store a pointer in shared memory remember that this will refer to a location in the address space of a specific process. However, the pointer is quite likely to be invalid in the context of a second process and trying to dereference the pointer from the second process may cause a crash.

It's not going to work. This isn't a programming language issue, it's an OS issue. sharedctypes is useful for things like arrays of ints and floats.

For this application to work (live telemetry) the receive process has to remain live at all times.

Sorry, I'm not following that in context. You said in your question: "If I place the open for the serial port (modemPort) as the first line of processIncomingSerialMessages the program works fine.". processIncomingSerialMessages has an infinite loop following that. In what sense does that violate "the receive process has to remain live at all times"?

In the code you showed, it doesn't appear to make a lick of difference whether the serial port is opened in a worker process or in the main process (if the latter actually worked), and indeed you said it worked fine if you did it the former way. So what exactly is wrong with doing it that way? Indeed, why use a worker process at all for that? The main program you showed does nothing (except sleep for a tenth of a second) after starting both workers, so why not let the main program be "the receive process"?