How to avoid deadlock with class attributes initialization with locks, inheritance and threading in Python?

Question

I am trying to implement thread safe code but encounter some simple problem. I searched and not found solution.

Let me show abstract code to describe problem:

import threading

class A(object):
  sharedLock = threading.Lock()
  shared = 0

  @classmethod
  def getIncremented(cls):
    with cls.sharedLock:
      cls.shared += 1
      return cls.shared

class B(A):
  pass

class C(A):
  @classmethod
  def getIncremented(cls):
    with cls.sharedLock:
      cls.shared += B.getIncremented()
      return cls.shared

I want to define class A to inherit many child classes for example for enumerations or lazy variables - specific use no matter. I am already done single thread version now want update multi thread.

This code will give such results as should do:

id(A.sharedLock) 11694384
id(B.sharedLock) 11694384
id(C.sharedLock) 11694384

I means that lock in class A is lock in class B so it is bad since first entry into class B will lock also class A and class C. If C will use B it will lead to dedlock.

I can use RLock but it is invalid programming pattern and not sure if it not produce more serious deadlock.

How can I change sharedLock value during initialization of class to new lock to make id(A.sharedLock) != id(B.sharedLock) and same for A and C and B and C?

How can I hook class initialization in python in generic to change some class variables?

That question is not too complex but I do not know what to do with it.

Answer 1

Here is solution - this allow separate lock per each class since it done on class constructor level (metaclass). Thank you for all hints and help to achieve this code it looks very nice.

I can be also mangled variable but need to use hardcode '_A__lock' what can be problematic and not tested by me.

import threading

class MetaA(type):
  def __new__(self, name, bases, clsDict):
    # change <type> behavior
    clsDict['_lock'] = threading.Lock()
    return super(MetaA, self).__new__(self, name, bases, clsDict)

class A(object):
  __metaclass__ = MetaA

  @classmethod
  def getLock(cls):
    return cls._lock

class B(A):
  pass

print 'id(A.getLock())', id(A.getLock())
print 'id(B.getLock())', id(B.getLock())
print A.getLock() == B.getLock()

Answer 2

I want inherit parent share variables except shared parent locks

You must not do this. It makes access to "share variables" not thread-safe.

---

sharedLock protects shared variable. If the same shared variable can be modified in a recursive call then you need RLock(). Here shared means shared among all subclasses.

It looks like you want a standalone function (or a static method) instead of the classmethod:

def getIncremented(_lock=Lock(), _shared=[0]):
    with _lock:
      _shared[0] += 1
      return _shared[0]

Thus all classes use the same shared variable (and the corresponding lock).

If you want each class to have its own shared variable (here shared means shared among instances of this particular class) then don't use cls.shared that may traverse ancestors to get it.

To hint that subclasses shouldn't use a variable directly, you could use the syntax for a private variable:

class A:
   __shared = 0
   __lock = Lock()

If a subclass overrides a method that uses __shared then it won't use A.__shared by accident in the code directly.

Answer 3

As you noticed, if you expose shared locks as class attributes, the locks are shared by subclasses.

You could hack around this by redefining the lock on each subclass:

class B(A):
  sharedLock = threading.Lock()

You could even use metaclasses to achieve this (please don't). It seems to me that you're approaching the program from the wrong angle.

This task is easier if you assign locks explicitly to instances (not classes).

class A(object):
    def __init__(self, lock):
        this.sharedLock= lock

my_lock= threading.Lock()
a= A(my_lock)

Of course, you run into the "problem" of having to explicitly pass the lock for each instance. This is traditionally solved using a factory pattern, but in python you can simply use functions properly:

from functools import partial
A_with_mylock= partial(A, my_lock)
a2= A_with_mylock()