Could a C extension for multithreaded Python boost performance?

Question

I have heard of Python's GIL problem, which states that there can only be one Python thread executing the Python bytecode at one time in a multicore machine. So a multithreaded Python program is not a good idea.

I am wondering if I can write a C extension that uses pthread to potentially boost the performance of my program. I'd like to create a thread in my C extension and make it runs in parallel with Python's main thread.

My assumption is that the Python main thread will do things more related to IO, while the pthread in my C extension will spend most of its time doing computation. The Python main thread communicates with the thread in the C extension using a queue (like a producer consumer model).

Is there any difference between multithread with Python and C extension?

Answer 1

To answer your original question:

Yes, C extensions can be immune from the GIL, provided they do not call any Python API functions without the GIL held. So, if you need to communicate with the Python app, you'd need to acquire the GIL to do so. If you don't want to get your hands too dirty with the C API, you can use ctypes to call a C library (which can just use pthreads as usual), or Cython to write your C extension in a Python-like syntax.

Answer 2

The Python interpreter is not aware of C launched threads in any way, so they can happily churn their own CPU time.

However I doubt this is a correct solution for your performance problems. First try using multiple processes with multiprocess module. If interprocess IO is too much after that you can result trickery like C threads. This would make your program an order of magnitude more complex so avoid it if possible.

Answer 3

Provided one thread runs CPU bound while the other runs IO bound, I don't see a problem.

The IO bound thread will call IO routines which usually release the GIL while doing their stuff, effectively allowing the other thread to run.

So give the "simple" solution a try and switch only if it really doesn't work the way you want it to.