“MetaClass”, “__new__”, “cls” and “super” - what is the mechanism exactly?

Question

I have read posts like these:

1. What is a metaclass in Python?
2. What are your (concrete) use-cases for metaclasses in Python?
3. Python's Super is nifty, but you can't use it

But somehow I got confused. Many confusions like:

When and why would I have to do something like the following?

# Refer link1
return super(MyType, cls).__new__(cls, name, bases, newattrs)

or

# Refer link2
return super(MetaSingleton, cls).__call__(*args, **kw)

or

# Refer link2
return type(self.__name__ + other.__name__, (self, other), {})

How does super work exactly?

What is class registry and unregistry in link1 and how exactly does it work? (I thought it has something to do with singleton. I may be wrong, being from C background. My coding style is still a mix of functional and OO).

What is the flow of class instantiation (subclass, metaclass, super, type) and method invocation (

metaclass->__new__, metaclass->__init__, super->__new__, subclass->__init__ inherited from metaclass

) with well-commented working code (though the first link is quite close, but it does not talk about cls keyword and super(..) and registry). Preferably an example with multiple inheritance.

P.S.: I made the last part as code because Stack Overflow formatting was converting the text metaclass->__new__ to metaclass->new

Answer 1

OK, you've thrown quite a few concepts into the mix here! I'm going to pull out a few of the specific questions you have.

In general, understanding super, the MRO and metclasses is made much more complicated because there have been lots of changes in this tricky area over the last few versions of Python.

Python's own documentation is a very good reference, and completely up to date. There is an IBM developerWorks article which is fine as an introduction and takes a more tutorial-based approach, but note that it's five years old, and spends a lot of time talking about the older-style approaches to meta-classes.

super is how you access an object's super-classes. It's more complex than (for example) Java's super keyword, mainly because of multiple inheritance in Python. As Super Considered Harmful explains, using super() can result in you implicitly using a chain of super-classes, the order of which is defined by the Method Resolution Order (MRO).

You can see the MRO for a class easily by invoking mro() on the class (not on an instance). Note that meta-classes are not in an object's super-class hierarchy.

Thomas' description of meta-classes here is excellent:

A metaclass is the class of a class. Like a class defines how an instance of the class behaves, a metaclass defines how a class behaves. A class is an instance of a metaclass.

In the examples you give, here's what's going on:

1. The call to __new__ is being bubbled up to the next thing in the MRO. In this case, super(MyType, cls) would resolve to type; calling type.__new__ lets Python complete it's normal instance creation steps.

2. This example is using meta-classes to enforce a singleton. He's overriding __call__ in the metaclass so that whenever a class instance is created, he intercepts that, and can bypass instance creation if there already is one (stored in cls.instance). Note that overriding __new__ in the metaclass won't be good enough, because that's only called when creating the class. Overriding __new__ on the class would work, however.

3. This shows a way to dynamically create a class. Here's he's appending the supplied class's name to the created class name, and adding it to the class hierarchy too.

I'm not exactly sure what sort of code example you're looking for, but here's a brief one showing meta-classes, inheritance and method resolution:

class MyMeta(type):
    def __new__(cls, name, bases, dct):
        print "meta: creating %s %s" % (name, bases)
        return type.__new__(cls, name, bases, dct)

    def meta_meth(cls):
        print "MyMeta.meta_meth"

    __repr__ = lambda c: c.__name__

class A(object):
    __metaclass__ = MyMeta
    def __init__(self):
        super(A, self).__init__()
        print "A init"

    def meth(self):
        print "A.meth"

class B(object):
    __metaclass__ = MyMeta
    def __init__(self):
        super(B, self).__init__()
        print "B init"

    def meth(self):
        print "B.meth"

class C(A, B):
    __metaclass__ = MyMeta
    def __init__(self):
        super(C, self).__init__()
        print "C init"

>>> c_obj = C()
meta: creating A (<type 'object'>,)
meta: creating B (<type 'object'>,)
meta: creating C (A, B)
B init
A init
C init
>>> c_obj.meth()
A.meth
>>> C.meta_meth()
MyMeta.meta_meth
>>> c_obj.meta_meth()
Traceback (most recent call last):
  File "mro.py", line 38, in <module>
    c_obj.meta_meth()
AttributeError: 'C' object has no attribute 'meta_meth'

Answer 2

Here's the more pragmatic answer.

It rarely matters

1. "What is a metaclass in Python". Bottom line, type is the metaclass of all classes. You have almost no practical use for this.

   class X(object):
       pass
   type(X) == type
   

2. "What are your (concrete) use cases for metaclasses in Python?". Bottom line. None.

3. "Python's Super is nifty, but you can't use it". Interesting note, but little practical value. You'll never have a need for resolving complex multiple inheritance networks. It's easy to prevent this problem from arising by using an explicity Strategy design instead of multiple inheritance.

Here's my experience over the last 7 years of Python programming.

1. A class has 1 or more superclasses forming a simple chain from my class to object.

2. The concept of "class" is defined by a metaclass named type. I might want to extend the concept of "class", but so far, it's never come up in practice. Not once. type always does the right thing.

3. Using super works out really well in practice. It allows a subclass to defer to it's superclass. It happens to show up in these metaclass examples because they're extending the built-in metaclass, type.

   However, in all subclass situations, you'll make use of super to extend a superclass.

Metaclasses

The metaclass issue is this:

• Every object has a reference to it's type definition, or "class".

• A class is, itself, also an object.

• Therefore a object of type class has a reference to it's type or "class". The "class" of a "class" is a metaclass.

Since a "class" isn't a C++ run-time object, this doesn't happen in C++. It does happen in Java, Smalltalk and Python.

A metaclass defines the behavior of a class object.

• 90% of your interaction with a class is to ask the class to create a new object.

• 10% of the time, you'll be using class methods or class variables ("static" in C++ or Java parlance.)

I have found a few use cases for class-level methods. I have almost no use cases for class variables. I've never had a situation to change the way object construction works.