PyCurl. 'Can't pickle Curl object' error on Windows x64

Question

import multiprocessing
import time
import pycurl

class Bot(multiprocessing.Process):
    def __init__(self):
        multiprocessing.Process.__init__(self)       
        self.c = pycurl.Curl() 

    def run(self):
        pass

if __name__ == '__main__': 
    Bot().start()    

This code works good on Ubuntu 13.04 x64, but fails on Windows 7 x64/Server 2008 x64. I use pyCurl 7.19.0 and Python 2.7. The error is:

Z:\bot>python test.py
Traceback (most recent call last):
  File "test.py", line 74, in <module>
    Bot().start()
  File "C:\Python27\lib\multiprocessing\process.py", line 104, in start
    self._popen = Popen(self)
  File "C:\Python27\lib\multiprocessing\forking.py", line 244, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python27\lib\multiprocessing\forking.py", line 167, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 313, in save
    (t.__name__, obj))
pickle.PicklingError: Can't pickle 'Curl' object: <pycurl.Curl object at 0x00000
00002360478>

Z:\bot>Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python27\lib\multiprocessing\forking.py", line 347, in main
    self = load(from_parent)
  File "C:\Python27\lib\pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "C:\Python27\lib\pickle.py", line 858, in load
    dispatch[key](self)
  File "C:\Python27\lib\pickle.py", line 880, in load_eof
    raise EOFError
EOFError

Why this happens?

Answer 1

On posix systems, multiprocessing works by using the fork() syscall, which clones the running process, and all of its state (including things like instances of complex classes, such as pycurl.Curl).

Windows does not have fork(), or anything like it, so multiprocessing fires up a new python interpreter for each child process, with a special stub function that listens to the parent process and recreates the state so that it looks much like it would had fork() been used. The key technique used by multiprocessing for this is that it recreates each object in the child process, from the parent process, as they are used. This works by converting the objects to a bytecode representation (using the pickle module), sending them over a pipe to the child, and converting them back to python objects once there.

For most kinds of python objects, this works perfectly, and is transparent to you. There are a number of obvious kinds of things where this cannot possibly work; open files cannot be passed around in this way; nor can the state of non-python library objects, which don't know anything about the pickle system. Unfortunately, pycurl is a bit of both of these things.