Access methods not exposed in the method mapping table with ctypes

Question

The method mapping table is an array of PyMethodDef structures,

struct PyMethodDef {
   char *ml_name;
   PyCFunction ml_meth;
   int ml_flags;
   char *ml_doc;
};

where the ml_name is the name of the functions accessed in the interpreter (i.e. gc.collect()), ml_meth is the address of the function described in the previous section, ml_flags indicates which of the signatures ml_meth is using and ml_doc is the docstring for the function.

Now, say I want to access from within the gcmodule.c gc_list_append which is not exposed by the gc interface.

I've tried the following:

pythonapi.PyImport_ImportModule("gcmodule")

or

pythonapi.PyImport_ImportModule(ctypes.c_wchar("gcmodule"))

Hoping that the reference returned can be used to access unexposed methods, but in both cases I get errors:

ImportError: No module named 'g'

or

TypeError: one character unicode string expected

Any idea how an unexposed function or data structures can be accessed via ctypes/pythonapi?

Answer 1

PyImport_ImportModule imports a module as named by the const char * argument. It calls PyImport_Import, which in turn calls __builtin__.__import__ in PY2. Calling it with the C string "gc" returns a reference to sys.modules['gc'] if it exists, else the interpreter creates the module by calling initgc , which it finds in _PyImport_Inittab. The gc module's init function calls Py_InitModule4, which loops over the GcMethods table to add methods to the module dict.

Every global definition you see in gcmodule.c that's marked static is only visible within that compilation unit at the system level, i.e. it doesn't have external linkage. Commonly an extension module only exports the name of the PyMODINIT_FUNC, but several of the built-in modules in CPython also export public APIs such as PyObject_GC_Track.

On Windows, exporting a symbol from a DLL by name or ordinal requires the extra step of declaring it in a .def file or with the modifier __declspec(dllexport). In CPython that gets applied via the macros PyAPI_FUNC (pyport.h), PyAPI_DATA, and PyMODINIT_FUNC. For example, objimpl.h declares PyAPI_FUNC(void) PyObject_GC_Track(void *).

Declaring a symbol static is namespace management. However, if you can get the address of a function somehow, you can certainly call it with a ctypes function pointer. Finding the address is the issue. ctypes uses the standard POSIX dlsym and Windows GetProcAddress functions to look up exported symbols, which won't help you here. You may be able to access function pointers referenced directly or indirectly by CPython objects, since id returns the object base address.

---

The following isn't directly related to your problem, but I'm including it for general interest.

On a POSIX system that loads extension modules with dlopen, you can specify RTLD_GLOBAL to merge the dynamic symbols into the global namespace. For example, the _ctypes extension module is linked from multiple object files and needs several extern symbols, such as PyCData_set. You can tweak the import to have PyCData_set accessible globally (e.g. via ctypes.pythonapi):

>>> import sys, DLFCN
>>> sys.setdlopenflags(DLFCN.RTLD_GLOBAL | DLFCN.RTLD_NOW)
>>> import ctypes
>>> hasattr(ctypes.pythonapi, 'PyCData_set')
True

Of course, that won't work on Windows. The proper way for an extension module to make a C API available is by using a PyCapsule, such as datetime.datetime_CAPI (datetimemodule.c).