Is it possible for 2 processes to access the same list?

Question

Im design a python module that I want to run 2 methods(method1 and method2) in 2 different processes is it possible to have a global list that both processes read and write to? Or will this cause issues later on in my project?

Heres an example:

from multiprocessing import Process, Queue
data =[]

def method1():
  global data
  data += [10,14,5]

def method2():
  global data
  data = [1,3,4]
  proc = Process(target=method1)
  proc.start()
  print data 

if __name__ == '__main__':
  method2()

This needs to be cross platform, windows, linux and OS X, in Python 2.7.

Answer 1

If this needs to be cross-platform, then no, you cannot share a list like this. (On Unix, if you're careful, you sometimes can, but never on Windows.)

If you read Sharing state between processes in the documentation, it explains what you can share and how, but basically, it's just simple values, Arrays of other shareable types, and anything you can define as a ctypes.Structure, and that's it.

The Programming guidelines, especially the Windows section, explains why this is true, and how you can deal with it. But basically, the problem is that, on Windows, the new process is not a fork of the old one, it's a brand-new process.

There are many alternatives.

The best is usually to redesign your algorithm around passing immutable values to the child and having it return new immutable values, instead of mutating shared values.

If you can't go that far, you can usually rewrite things in terms of passing messages over, e.g., a Queue or a Pipe. In the worst case, the messages can be (list.append, data, [10, 14, 5]), but usually you can come up with something higher-level and more meaningful than that.

Answer 2

From help docs: http://docs.python.org/2/library/multiprocessing.html#exchanging-objects-between-processes

from multiprocessing import Process, Queue

def f(q):
    q.put([42, None, 'hello'])

if __name__ == '__main__':
    q = Queue()
    p = Process(target=f, args=(q,))
    p.start()
    print q.get()    # prints "[42, None, 'hello']"
    p.join()

You're already importing Queue. Now you just have to use it :)

put adds items, get gets items, from the queue.