Working with parallel python and classes

Question

I was shocked to learn how little tutorials and guides there is to be found on the internet regarding parallel python (PP) and handling classes. I've ran into a problem where I want to initiate a couple of instances of the same class and after that retreive some variables (for instances reading 5 datafiles in parallel, and then retreive their data). Here's a simple piece of code to illustrate my problem:

import pp

class TestClass:
    def __init__(self, i):
        self.i = i

    def doSomething(self):
        print "\nI'm being executed!, i = "+str(self.i)
        self.j = 2*self.i
        print "self.j is supposed to be "+str(self.j)
        return self.i

class parallelClass:
    def __init__(self):
        job_server = pp.Server()
        job_list = []
        self.instances = [] # for storage of the class objects
        for i in xrange(3):
            TC = TestClass(i) # initiate a new instance of the TestClass
            self.instances.append(TC) # store the instance
            job_list.append(job_server.submit(TC.doSomething, (), ())) # add some jobs to the job_list
        results = [job() for job in job_list] # execute order 66...

        print "\nIf all went well there's a nice bunch of objects in here:"
        print self.instances
        print "\nAccessing an object's i works ok, but accessing j does not"
        print "i = "+str(self.instances[2].i)
        print "j = "+str(self.instances[2].j)

if __name__ == '__main__' :
    parallelClass() # initiate the program

I've added comments for your convenience. What am I doing wrong here?

Answer 1

You should use callbacks

A callbacks is a function that you pass to the submit call. That function will be called with the result of the job as argument (have a look at the API for more arcane usage).

In your case

Set up a callback:

class TestClass:
    def doSomething(self):
         j = 2 * self.i
         return j # It's REQUIRED that you return j here.

    def set_j(self, j):
        self.j = j

Add the callback to the job submit call

 class parallellClass:
      def __init__(self):
          #your code...
          job_list.append(job_server.submit(TC.doSomething, callback=TC.set_j))

And you're done.

I made some improvements to the code to avoid using self.j in the doSomething call, and only use a local jvariable.

As mentioned in the comments, in pp, you only communicate the result of your job. That's why you have to return this variable, it will be passed to the callback.