I recently got hit by the same problem, and I also tracked it down to this exact block of code in the threading
module.
It sucks.
Can you explain what's the purpose of this? Are Python developers assume no Python user will care about such time lengths?
Beats me...
Do you recommend me to overload the threading module?
Either overload the threading module, or migrate to python3
, where this part of the implementation has been fixed.
In my case, migrating to python3 would have been a huge effort, so I chose the former. What I did was:
- I created a quick
.so
file (usingcython
) with an interface topthread
. It includes python functions which invoke the correspondingpthread_mutex_*
functions, and links againstlibpthread
. Specifically, the function most relevant to the task we're interested in is pthread_mutex_timedlock. - I created a new
threading2
module, (and replaced allimport threading
lines in my codebase withimport threading2
). Inthreading2
, I re-defined all the relevant classes fromthreading
(Lock
,Condition
,Event
), and also ones fromQueue
which I use a lot (Queue
andPriorityQueue
). TheLock
class was completely re-implemented usingpthread_mutex_*
functions, but the rest were much easier -- I simply subclassed the original (e.g.threading.Event
), and overridden__init__
to create my newLock
type. The rest just worked.
The implementation of the new Lock
type was very similar to the original implementation in threading
, but I based the new implemenation of acquire
on the code I found in python3
's threading
module (which, naturally, is much simpler than the abovementioned "balancing act" block). This part was fairly easy.
(Btw, the result in my case was 30% speedup of my massively-multithreaded process. Even more than I expected.)
I hope this helps.