Python deadlock related to packaging and mayavi?

Question

I have a curious problem that I hope someone can shed some light on.

I have a complex piece of code, that started as a directory full of scripts, that I decided to rework into a package. This code change appears to have been the trigger for some strange deadlocks to appear.

Below is an attempt at canonical reproduction of the problem; which fails, in the sense that this code runs as expected. Actually reproducing the issue may require a lot of code; but I cannot for the life of my imagine what is different for the offending code snippet in-context.

import numpy as np
from scipy.sparse import csr_matrix
from threading import Thread

def dummy():
    print 'this is printed'
    I = np.eye(3)
    print 'all is still fine'
    csr_matrix(I)
    print 'this is never printed; csr_matrix appears to be a trigger for deadlock'
    print np.ones(4)
    print 'same problem; somehow, printing ndarrays is no longer cool either'

thr = Thread(target=dummy)
thr.start()

Perhaps this terse comment in the docs is related? I am not sure I fully appreciate what is being said here

http://docs.python.org/2/library/threading#importing-in-threaded-code

Firstly, other than in the main module, an import should not have the side effect of spawning a new thread and then waiting for that thread in any way. Failing to abide by this restriction can lead to a deadlock if the spawned thread directly or indirectly attempts to import a module.

Some context: I am using python 2.7, numpy 1.8, where I try to spawn this new thread from within a mayavi/traitsui thread (which I dont see why it should be relevant, and which worked fine before the package structure, but ok). Also, there is a boatload of numpy/scipy code in my spawned thread that executes perfectly fine; its just printing ndarrays and creating sparse matrices which so far have proven to be triggers for deadlock.

I am suspecting some funky interaction with mayavi, since closing the mayavi window causes all deadlocked threads to start running again. Perhaps these specific statement trigger the python thread to yield back to the mayavi thread, but they somehow fail to gain focus again?

Any hints that lead to further narrowing down on this mystery are much appreciated!

Answer 1

From your comments, it looks like you start up the UI event loop at the top level of one of your subsidiary modules. This is not a good idea because it causes exactly the same problems that the documentation alludes to. import foo should never start a UI event loop. The problem is that the main thread grabs the import lock to handle the import of the module. This module starts up the UI event loop before completing the import. This is essentially the same situation as the waiting for the other threads to finish; you are waiting for the UI loop to finish. If your UI starts up other threads, the code that is running in the other threads will not be able to import anything (both csr_matrix() and ndarray.__repr__() import other modules) because the main thread is still holding onto the import lock.