Passing a multiprocessing queue/dictionary/etc.. to green threads

Question

Is it safe to pass a multiprocessing object (queue, dictionary, etc...) to multiple gevent threads? Since they're not actually running concurrently, I don't think there's a problem. However, I know that gevent isn't supposed to be specifically compatible with multiprocessing.

Answer 1

The benefits would likely be lost, standard threaded queue implements locks where a green thread would likely be slowed down. Thankfully, gevent often has its own but similar constructs. Check out gevent.queue

Answer 2

Unfortunately, it seems like, at the moment, gevent is not compatible with objects from multiprocessing:

It is dangerous. The mp.Queue and other mp data structures utilize things like Semaphores internally: https://github.com/python/cpython/blob/master/Lib/multiprocessing/queues.py#L48

Semaphores under Linux are not fd-based and would require threaded wrappers to unblock the main loop thread. Generally speaking, if things go south, it's possible to completely block the main thread with a semaphore waiting infinitely for some event to happen.

(Quote from GitHub issues https://github.com/gevent/gevent/issues/1443)

Answer 3

I would say that it is a thread safe object then it is not dangerous, but you should always think hard about it. If it isn't thread safe you need to worry about reentrancy of the methods and the consequence of the different object operations not being atomic. Some objects are stateful and they need to complete certain operations before another thread comes in.