numpy matrix to shared array for multiprocessing

Question

the matrix Z is defined like this:

N = 15
Z = np.zeros((N, N))

and I want different threads to fill it so I'm need to create a shared memory matrix or array. I'm doing this:

Z_shared = Array('d', Z)

But I get this error:

TypeError: only length-1 arrays can be converted to Python scalars

Which I suppose means that Array doesn't accept matrices. Is there another function that does what I need or should I convert Z to a vector first and then pass it to Array. If so is there a function that does this for me. I don't want to loop over Z since N can be very large.

I'm fill Z_shared in another function that is called like this:

from multiprocessing import Pool, Array
pool = Pool(processes=1)

def foo(data):
    x, y, val = data
    Z_shared[x][y] = val
pool.map(fooe, DATA)

'data' is a tuple of indexes and values and 'DATA' is a list a of tuples.

thank you

Answer 1

Strictly just answering the question, if the object constructed by Array acts like a list, you can copy everything like so:

Z_shared = Array('d', Z.size)
Z_shared[:] = Z.reshape((-1)) # copy entrywise

That seems bad, though. For one thing, the shared array won't have ndarray semantics, and will be a copy. Does it really need to be a single shared variable?

---

You're splitting into N*N processes in your program, where N can be large. Making a new process can be expensive. You should only split into as many processes as you have cores, I think.

So the idea is, split the data based on the x coordinate, and have one process to deal with the top half of the matrix and the other to deal with the bottom half. Both would be separate arrays in shared memory.

---

Reading into your question, you should not be playing with multiprocessing in the first place (premature optimization!). Instead, check out sparse matrices: http://docs.scipy.org/doc/scipy/reference/sparse.html