TypeError when using super method with class decorator for a derived class

Question

Firstly, apologies for the long explanation.

Version #1 - Code: Class decorator for a Class

class A(object):
    def __init__(self, klass):
        print "A::__init__()"
        self._klass = klass

    def __call__(self):
        print "A::__call__()"
        return self._klass()

    def __del__(self):
        print "A::__del__()"

@A
class B(object):
    def __init__(self):
        print "B::__init__()"

def main():
    b = B()

if __name__ == "__main__":
    main()

Version #1 - Output:

A::__init__()
A::__call__()
B::__init__()
A::__del__()

Version #2 - Code: Class decorator for a Derived Class which explicitly initialises the Base Classes.

class A(object):
    def __init__(self, klass):
        print "A::__init__()"
        self._klass = klass

    def __call__(self):
        print "A::__call__()"
        return self._klass()

    def __del__(self):
        print "A::__del__()"

class Parent1(object):
    def __init__(self):
        print "Parent1:: __init__()"
        super(Parent1, self).__init__()

class Parent2(object):
    def __init__(self):
        print "Parent2:: __init__()"
        super(Parent2, self).__init__()    

@A
class B(Parent1, Parent2):
    def __init__(self):
        print "B::__init__()"
#        super(B, self).__init__()
        Parent1.__init__(self)
        Parent2.__init__(self)

def main():
    b = B()

if __name__ == "__main__":
    main()

Version #2 - Output:

A::__init__()
A::__call__()
B::__init__()
Parent1:: __init__()
Parent2:: __init__()
Parent2:: __init__()
A::__del__()

Version #3 - Code: Class decorator for a Derived Class with super()

class A(object):
    def __init__(self, klass):
        print "A::__init__()"
        self._klass = klass

    def __call__(self):
        print "A::__call__()"
        return self._klass()

    def __del__(self):
        print "A::__del__()"   

class Parent1(object):
    def __init__(self):
        print "Parent1:: __init__()"
        super(Parent1, self).__init__()

class Parent2(object):
    def __init__(self):
        print "Parent2:: __init__()"
        super(Parent2, self).__init__()

@A
class B(Parent1, Parent2):
    def __init__(self):
        print "B::__init__()"
        super(B, self).__init__()

def main():
    b = B()

if __name__ == "__main__":
    main()

Version #3 - Output:

A::__init__()
A::__call__()
B::__init__()
Traceback (most recent call last):
  File "so.py", line 40, in <module>
    main()
  File "so.py", line 36, in main
    b = B()
  File "so.py", line 10, in __call__
    return self._klass()
  File "so.py", line 32, in __init__
    super(B, self).__init__()
TypeError: must be type, not A
A::__del__()

Question:

Version #1 is just for a reference. It explains what I am trying to do i.e. capture the creation and deletion of objects of class B.

In version #2, I have tried the same for objects of a class B which is derived from Parent1 and Parent2 which are initialised explicitly using Parent1.__init__(self) and Parent2.__init__(self) which works fine as expected.

But in version #3, I have tried the same with super() method. But I get the following error - TypeError: must be type, not A. I think this is because the __init__() method for all the parent classes in the MRO chain is not properly called - Why? And, how do I fix this?

Answer 1

I took sometime to understand why it is hard to capture the object instantiation and object deletion using __call__ and __del__ methods respectively. The following are some of the useful references

• http://eli.thegreenplace.net/2009/06/12/safely-using-destructors-in-python/
• I don't understand this python __del__ behaviour
• the same topic has also been discussed in the python-list mailer - check out the thread here - https://mail.python.org/pipermail/python-list/2014-February/667592.html

There are excellent hacks available to do this using __del__ method but they are have side effects! For example, the answer given by @user2357112 is a good hack but it does not work when we make a circular reference because the garbage collector cannot determine which __del__ among the circular reference to call first! However this can be avoided by using a weak ref; but its still a hack!

One of the suggestions that was made was to create a context manager which can create and delete the objects of a particular class.

I have the following example which sort of emulates that. Please take a closer look at the Controller decorator.

class Parent1(object):
    def __init__(self):
        #print "Parent1::__init__()"
        super(Parent1, self).__init__()

class Parent2(object):
    def __init__(self):
        #print "Parent2::__init__()"
        super(Parent2, self).__init__()

def Controller(_cls):
    class Wrapper(_cls):
        def create(self, name):
            ret = _cls.create(self, name)
            print "Added to Database! :: ", name
            # Database add here!
            return ret

        def remove(self, name):
            ret = _cls.remove(self, name)
            print "Deleted from Database! :: ", name
            # Database delete here!
            return ret
    return Wrapper

@Controller
class Manager(object):
    def __init__(self):
        #print "Manager::__init__()"
        self._repo = []

    def create(self, name):
        a = A(name)
        print "Object created :: ", name
        self._repo.append(a)

    def remove(self, name):
        for i, item in enumerate(self._repo):
            if item._name == name:
                del self._repo[i]
                print "Object removed :: ", name

    def display(self):
        for item in self._repo:
            print item

class A(Parent1, Parent2):
    def __init__(self, name):
        #print "A::__init__()"
        self._name = name
        super(A, self).__init__()

    def __repr__(self):
        return self._name

def main():
    m1 = Manager()
    m1.create("apples")
    m1.create("oranges")
    m1.create("grapes")
    #m1.display()
    m1.remove("apples")
    #m1.display()

if __name__ == "__main__":
    main()

When executed, it produces the following result:

Object created ::  apples
Added to Database! ::  apples
Object created ::  oranges
Added to Database! ::  oranges
Object created ::  grapes
Added to Database! ::  grapes
Object removed ::  apples
Deleted from Database! ::  apples

This is the safest solution that I could come up for my problem. Suggestions welcome!

Answer 2

The main problem is that super's first argument needs to be the actual class, but in version 3, in

super(B, self)

B isn't the class you created. It's the A instance wrapping the class. You would need to do something like

class _B(Parent1, Parent2):
    def __init__(self):
        print "B::__init__()"
        super(_B, self).__init__()
B = A(_B)

or instead of wrapping B in an A instance, use a decorator that replaces B's __init__ and __del__ methods with wrappers without replacing the whole B class.

Also, if you want to track the deletion of B instances, a __del__ method on A won't do that. It'll track the deletion of the class, not individual instances.

---

Here's a decorator that should do what you want, without the many problems that come from wrapping the class in something that isn't a class:

def track_creation_and_deletion(klass):
    original_init = klass.__init__
    try:
        original_del = klass.__del__
    except AttributeError:
        def original_del(self):
            pass

    def new_init(self, *args, **kwargs):
        print '{}.{}.__init__'.format(klass.__module__, klass.__name__)
        return original_init(self, *args, **kwargs)
    def new_del(self):
        print '{}.{}.__del__'.format(klass.__module__, klass.__name__)
        return original_del(self)

    # functools.wraps doesn't play nicely with built-in methods,
    # so we handle it ourselves
    new_init.__name__ = '__init__'
    new_init.__doc__ = original_init.__doc__
    new_init.__module__ = klass.__module__
    new_init.__dict__.update(getattr(original_init, '__dict__', {}))

    new_del.__name__ = '__del__'
    new_del.__doc__ = original_del.__doc__
    new_del.__module__ = klass.__module__
    new_del.__dict__.update(getattr(original_del, '__dict__', {}))

    klass.__init__ = new_init
    klass.__del__ = new_del

    return klass

About half of this is error-handling and copying some metadata to make the new methods look like they were defined by the caller. The key parts are that we define new __init__ and __del__ methods wrapping and replacing the class's old ones. When an instance of the decorated class is created, the __init__ method we gave it will invoke logging code of our choice. When an instance of the decorated class is garbage collected, the __del__ method we gave it will invoke other logging code. Since we didn't replace the class object itself, references to the class by name in super calls will refer to the class they need to refer to.

One limitation of this approach is that it's difficult to inspect the instance itself in our __init__, because it may not be fully constructed, even after the wrapped __init__ returns. For example, if we try to print the instance, we may trigger a subclass's __str__ method relying on subclass attributes that aren't ready yet, causing an AttributeError.