Python Multiprocessing: Example not raising a PicklingError

Question

Can anyone explain to me why the last example (Example 3. Multiprocess wrapper for Net-SNMP) in the following page: https://www.ibm.com/developerworks/aix/library/au-multiprocessing/ does not raise a PicklingError ?

I have tried it with my own bound method that updates and returns an instance attribute(similar to the example which updates and returns an attribute of the instance) and it raises the following error:

 Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
PicklingError: Can't pickle <type 'thread.lock'>: attribute lookup thread.lock failed

Here is my code:

from multiprocessing import Process, Queue
import requests

class MyClass(object):

    def do_request(self, url):
        try:
            self.response = requests.get(url)
        except:
            self.response = None
        return self.response

def make_request(url):
    s = MyClass()
    return s.do_request(url)

# Function run by worker processes
def worker(input, output):
    for func in iter(input.get, 'STOP'):
        result = make_request(func)
        output.put(result)

def main():
    """Runs everything"""

    #clients
    urls = ['http://www.google.com', 'http://www.amazon.com']
    NUMBER_OF_PROCESSES = len(urls)

    # Create queues
    task_queue = Queue()
    done_queue = Queue()

    #submit tasks
    for url in urls:
        task_queue.put(url)

    #Start worker processes
    for i in range(NUMBER_OF_PROCESSES):
        Process(target=worker, args=(task_queue, done_queue)).start()

     # Get and print results
    print 'Unordered results:'
    for i in range(len(urls)):
        print '\t', done_queue.get()

    # Tell child processes to stop
    for i in range(NUMBER_OF_PROCESSES):
        task_queue.put('STOP')
        print "Stopping Process #%s" % i

if __name__ == "__main__":
    main()

Answer 1

The problem is that the return from requests.get() is not a pickable object. You'll need to extract the information you want and return that to the parent object. Personally, I like to keep to simple types plus lists and dicts for this type of thing - it keeps the number of bad things that can happen to a minimum.

Here's an experiment that threw a much more messy exception on my linux + python 2.7 machine, but gives you an idea of the problem:

>>> import requests
>>> import pickle
>>> r=requests.get('http://google.com')
>>> pickle.dumps(r)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)

  ... many lines removed

  File "/usr/lib/python2.7/copy_reg.py", line 77, in _reduce_ex
    raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled

You can test whether I'm right or not by replacing self.response = requests.get(url) with self.response = "hello".