Why does Python's GIL enforce strict ordering of processing?

Question

from concurrent.futures import ThreadPoolExecutor, as_completed

def main():
    with ThreadPoolExecutor(max_workers=16) as producersPool:
        for i in [1,2,3,4,5,6,7,8,9,0]:
            producersPool.submit((lambda i : print(i))(i))


if __name__ == "__main__":
    main()

Running with python3:

1
2
3
4
5
6
7
8
9
0

Always the same.

Now I'd like you to get me right - I don't necessarily wish for those tasks to be reordered, but I'm just wondering why does the ordering happen at all? I mean, one could expect to get things done in deterministic way inside one thread, but the strict queueing of the threads seems a bit odd to me.

Anyway, how do I get the real concurrency in Python 3? (Jython and IronPython support only 2.x, as far as I know).

Answer 1

All jobs are executed before they are submitted, and none of them do more than write to an output stream. You submit the result of calling each lambda.

Even if you used:

producersPool.submit(lambda i: print(i), (i,))

there is not enough in each thread to be preempted by another thread, so they all complete before any other thread can but in.

Answer 2

to answer the second part of your question:

"how do I get the real concurrency in Python 3?"

you can use the multiprocessing module:

"multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads."

Since your example used ThreadPoolExecutor from concurrent.futures, have a look at ProcessPoolExecutor in the same package. It is similar to ThreadPoolExecutor, but uses processes (instead of threads), so it is not hampered by the GIL.