Is there such a thing as working threading in python?

Question

I wrote a beautiful multithreaded script, and when I ran it, it performed worse with 25 threads than with just direct invocation of the thread handler.

Then I discovered the global interpreter lock. I want to ask, before I discard python for this script and rewrite the thing in something else, is there any way to do actual working multithreading in python?

Answer 1

The other approach is to abandon threads and instead use the Multiprocessing module (Python 2.6+), which gets around the GIL, and has an API that's at least similar to the one in the threading module.

Answer 2

Now this is an interesting question - I don't think there's an escaping the GIL in straight CPython.

Stackless Python is supposed to have improved "concurrency" performance over CPython with its use of microthreads, but I dont't think it escapes the GIL.

Additionally, according to the GIL page on python.org, Jython and IronPython don't have a GIL.

Answer 3

The correct answer depends heavily on what you're doing.

Heavily CPU-bound (and blocking IO-bound) tasks, like compression and image rendering, are usually done with native code, and native libraries normally release the GIL while they work, which allows concurrency. When you can isolate the CPU-intensive work to a narrow native call, you get concurrency, native performance where it counts, and the convenience of writing most of the code in Python.

Not all code has those small, isolatable blocks of computational code that can be neatly implemented in a native library, but a whole lot do.

Answer 4

In CPython (the implementation of Python in C), no, according to my knowledge. (If I'm wrong, please correct me, I'd sure like to know if a solution exists!).

You might want to be interested in IronPython (.NET) or JPython (Java/JWM). I haven't used them but I believe at least one of them supports threads native to the given execution environment.

Answer 5

You could try the multiprocessing module, if it's applicable to your problem.