Create a Python type from C that implements a __dict__?

Question

• How is a type created to have a __dict__ as per a "normal" class would have were it defined in Python?
• Are there any examples of non-dynamic types with __dict__s?
• Do types defined via Python's PyTypeObject pass through type_new?

There is a tp_dict member of PyTypeObject, but I can find no information on how it's used. There also seems to be something going on in typeobject.c's type_new but I can't decipher it clearly.

Here is some related information I've found:

• __dict__ in class inherited from C extension module
• How is __slots__ implemented in Python?

Answer 1

The following code will generate a class that implements a __dict__ in Python 2.x:

typedef struct {
  PyObject_HEAD
  PyObject* dict;
} BarObject;

static PyTypeObject BarObject_Type = {
  PyObject_HEAD_INIT(NULL)
};

PyMODINIT_FUNC
initFoo(void)
{
  PyObject *m;

  m = Py_InitModule("Foo", NULL);
  if (m == NULL)
    return;

  BarObject_Type.tp_new = PyType_GenericNew;
  BarObject_Type.tp_name = "Foo.Bar";
  BarObject_Type.tp_basicsize = sizeof(BarObject);
  BarObject_Type.tp_getattro = PyObject_GenericGetAttr;
  BarObject_Type.tp_setattro = PyObject_GenericSetAttr;
  BarObject_Type.tp_flags = Py_TPFLAGS_DEFAULT;
  BarObject_Type.tp_dictoffset = offsetof(BarObject,dict);
  BarObject_Type.tp_doc = "Doc string for class Bar in module Foo.";
  if (PyType_Ready(&BarObject_Type) < 0)
    return;

  Py_INCREF(&BarObject_Type);
  PyModule_AddObject(m, "Bar", (PyObject*)&BarObject_Type);
}

The important bit is the tp_dictoffset member of the PyTypeObject struct (http://docs.python.org/c-api/typeobj.html):

If the instances of this type have a dictionary containing instance variables, this field is non-zero and contains the offset in the instances of the type of the instance variable dictionary; this offset is used by PyObject_GenericGetAttr().

Do not confuse this field with tp_dict; that is the dictionary for attributes of the type object itself.

Answer 2

To answer the last question first: No, type_new is only used for "heap types" that are dynamically defined at runtime (e.g. via a class statement). Statically defined types are initialised using PyType_Ready() instead.

To answer your first question: to create an extension type with a __dict__ descriptor, you need to dynamically allocate the type the same way the interpreter does for a class definition.

One way to get examples for that is to do as John suggests and generate some examples of your own with Cython.

For CPython 2.x you can look at the build_class method in the CPython source code (http://svn.python.org/view/python/trunk/Python/ceval.c?view=markup) to get an idea of the steps involved in a fully general solution.

If you're using Python 3 though, then this question may be of interest: What does Python's builtin __build_class__ do?

That is, as a CPython 3.x specific solution, the simplest thing to do is call builtins.__build_class__ with appropriate arguments via the C API.

Answer 3

I haven't done this, I'm embarrasingly bad on using the C-API, but according to the docs it should be enough with using PyObject_GenericGetAttr and PyObject_GenericSetAttr for the getattro and setattro methods. You also need to have a PyObject attribute called __dict__ of course.

Have you tried that?

Answer 4

How about writing some Python code defining a class and a method or two, compiling it with Cython and inspecting the generated C code?