Python Multiprocessing Pool passing function arguments

Question

I've got 4 nested loops, where I loop though lists of lists, and a function where I pass some arguments I get from the function, specifically:

def method(var1, var2 ....., var 16):
  #computation 

pool = Pool()

for values1 in list_of_values1:
  #assign 4 variables from the list 
  for values2 in list_of_values2:
    #assign 4 variables from list 
    for values3 in list_of_values3:
      #assign 4 variables from  list 
      for values4 in list_of_values4:
        #assign 4 variables from list
        method(var1, ..., var16)

I have tried using the

pool.map(method, [var1,..., var16]) in order to paralelize the whole process but it throws an error saying "method() takes exactly 16 arguments, 1 given"

I have also tried to use Threads but it does not improve much.

Grateful for the help!

Cheers

Answer 1

Pool.map is designed to call method on each value in the iterator you pass as the second argument. So in your example, it will try to do the following in parallel:

method(var1)
method(var2)
method(var3)
...
method(var16)

I think what you want to do is:

for values1 in list_of_values1:
  #assign 4 variables from the list 
  for values2 in list_of_values2:
    #assign 4 variables from list 
    for values3 in list_of_values3:
      #assign 4 variables from  list 
      for values4 in list_of_values4:
        #assign 4 variables from list
        pool.apply_async(method, (var1, ..., var16))
pool.close()
pool.join()

This is just calling method with the 16 arguments it expects, but doing it in a background process.

It's also possible you can do something a little nicer than the multiple nested for loops, by using itertools to build a list of (var1, ..., var16) tuples, which could then be passed to pool.map. But it depends on what #assign 4 variables from list* actually does.