ctypes: Correctly subclass c_void_p for passing and returning custom data types, by example

Question

I am working with ctypes and cannot seem to figure out how to work with custom data types. The hope is to have a Python interface to the public methods of a C++ cell class and a C++ cellComplex class.

My current problem is working with the C function called get_elementsAtSpecifiedDim(), defined below under extern "C" {.... This function has been written to return a void * which is really a std:vector< cell<double>* >, a vector of pointers to C++ cell objects.

My Client code below shows an example of a call to this function (via the Python CellComplex method elemen()) and this appears to me to be working fine. You will notice that my Python implementation of elemen() declares the return type lib.elemen.restype to be an array of (Python) Cell objects. That's what I get, >>>print cmplx.elemen() yields

[<cellComplex_python.Cell_Array_8 object at 0x10d5f59e0>, <cellComplex_python.Cell_Array_12 object at 0x10d5f5830>, <cellComplex_python.Cell_Array_6 object at 0x10d5f58c0>, <cellComplex_python.Cell_Array_1 object at 0x10d5f5950>]

But here's the problem:

Now I want to call my C functions on one of the Cell objects in an array in this list. For example, cmplx.elemen()[0][0] is a <Cell object at 0x10d5f5b00>, so in my mind I should be able to do this:

cmplx.elemen()[0][0].dim(), but this segfaults.

My suspicion is that I am not creating the custom Python classes Cell and CellComplex correctly. In particular, in Python class Cell, method dim(self): I have the line lib.dim.argtypes = [Cell], which must be absolutely bogus. As well, I have this silly Python class c_cellComplex which does nothing except allow me to indicate to myself what a particular ctypes.c_void_p is supposed to point to. In fact, I claim that my definitions of these Python classes are entirely bogus and I am being tricked into thinking I am on the right track by the miracle that this runs at all (up until I try to call a Cell method on a supposed Cell instance...

Client code:

p = [[0,1],[0,1]] 
cmplx = cellComplex(p)
e = cmplx.elemen()

e[0][0].dim() # segfault

begin EDIT eryksun's answer below provides an example of how to subclass c_void_p, and addresses a few other conceptual issues - start there if you have the same questions I had.

The segfault issue comes from the fact that get_elementsAtSpecifiedDim() defined in extern C { ... returns a memory address to a std::vector<cell<double>* >, a datatype that cannot be parsed back in Python. In this case I can just grab the pointers in the vector and return them, like so:

extern "C" {

  void * get_elementAtSpecifiedDimAndLoc(void *ptr, int dim, int nr) {
    cellComplex<double>* cmplx = static_cast<cellComplex<double>* >(ptr);
    cell<double>* c = cmplx->elemen()[dim][nr];
    return c;
  }
}

and can be called like so:

def elemen(self):    
     el = []
     for i in range(self.dim):
          size  = lib.size_elementsAtSpecifiedDim(self,i)               
          cells = []
          for j in range(size):
               cells.append(lib.get_elementAtSpecifiedDimAndLoc(self,i,j))            
          el.append(cells)
     return el
# put this somewhere
lib.get_elementAtSpecifiedDimAndLoc.restype  = Cell 
lib.get_elementAtSpecifiedDimAndLoc.argtypes = [CellComplex,c_int,c_int]

The client code now works.

end EDIT

Here is the magnificent folly:

# cellComplex_python.py 
lib = ctypes.cdll.LoadLibrary('./cellComplex_lib.so')

class c_cellComplex(ctypes.c_void_p):
     pass

class Cell(ctypes.c_void_p):

     def dim(self):
          lib.dim.restype  = ctypes.c_int
          lib.dim.argtypes = [Cell]
          self.dimension = lib.dim(self)
          return self.dimension

class CellComplex(ctypes.c_void_p):

     def __init__(self,p):   
          self.dimension = len(p)
          lib.new_cellComplex.restype  = c_cellComplex
          lib.new_cellComplex.argtypes = [(ctypes.c_double*2)*self.dimension, 
                                           ctypes.c_size_t]
          e = [(ctypes.c_double*2)(p[i][0],p[i][1]) for i in range(self.dimension)]
          point = ((ctypes.c_double*2)*self.dimension)(*e)
          self.cmplx = lib.new_cellComplex(point,self.dimension)


     def elemen(self):
          lib.size_elementsAtSpecifiedDim.restype  = ctypes.c_int
          lib.size_elementsAtSpecifiedDim.argtypes = [c_cellComplex,
                                                      ctypes.c_int]
          lib.get_elementsAtSpecifiedDim.argtypes  = [c_cellComplex,ctypes.c_int]

          self.sizeAtDim = []
          self.elements  = []
          for i in range(self.dimension+1):

               self.sizeAtDim.append(lib.size_elementsAtSpecifiedDim(self.cmplx,i))
               lib.get_elementsAtSpecifiedDim.restype = Cell*self.sizeAtDim[i]
               self.elements.append(lib.get_elementsAtSpecifiedDim(self.cmplx,i))

          return self.elements

C code:

// cellComplex_extern.cpp
#include<"cell.hpp">
#include<"cellComplex.hpp"> 

extern "C" {

  void * new_cellComplex(double p[][2], size_t dim) {
    std::vector< std::pair<double,double> > point;
    for (size_t i=0; i<dim; ++i) {
      point.push_back( std::make_pair(p[i][0],p[i][1]));
    }   
    cellComplex<double>* cmplx = new cellComplex<double>(point);
    return cmplx;
  }

  void * get_elementsAtSpecifiedDim(void *ptr, int dim) {
    cellComplex<double>* cmplx = static_cast<cellComplex<double>* >(ptr);
    std::vector<std::vector<cell<double>* > >* e = &cmplx->elemen();
    return &e[dim];
  }

  int size_elementsAtSpecifiedDim(void *ptr, int dim) {
    cellComplex<double>* cmplx = static_cast<cellComplex<double>* >(ptr);
    return cmplx->elemen()[dim].size();
  }  

  int dim(void *ptr) {
    cell<double>* ref = static_cast<cell<double>* >(ptr);
    return ref->dim();
  }

}

Answer 1

Instead of subclassing c_void_p, you can define the class method from_param and instance attribute _as_parameter_. You may not need either option if you're just proxying a C++ object that's referenced in a private attribute such as _obj. That said, a subclass of c_void_p can be used directly with ctypes pointers, arrays, and structs, which may be convenient in your overall design.

The following example may help:

from ctypes import *

__all__ = ['CellComplex']

class Cell(c_void_p):

    def __new__(cls, *args, **kwds):
        raise TypeError("cannot create %r instances" % cls.__name__)

    @property
    def dimension(self):
        return lib.dim(self)

class CellComplex(c_void_p):

    def __init__(self, p):
        pair = c_double * 2
        point = (pair * len(p))(*(pair(*q[:2]) for q in p))
        self.value = lib.new_cellComplex(point, len(p)).value

    @property
    def dimension(self):
        """Wrap a function that returns size_t."""
        return lib.????????(self)

    def get_elements(self):
        el = []
        for i in range(self.dimension):
            size = lib.size_elementsAtSpecifiedDim(self, i)
            cells = lib.get_elementsAtSpecifiedDim(self, i)
            el.append(cells[:size])
        return el

Function pointer definitions:

lib = CDLL('./cellComplex_lib.so')

lib.dim.restype = c_int
lib.dim.argtypes = [Cell]

lib.new_cellComplex.restype = CellComplex
lib.new_cellComplex.argtypes = [POINTER(c_double * 2), c_size_t]

lib.size_elementsAtSpecifiedDim.restype = c_int
lib.size_elementsAtSpecifiedDim.argtypes = [CellComplex, c_int]

lib.get_elementsAtSpecifiedDim.restype = POINTER(Cell)
lib.get_elementsAtSpecifiedDim.argtypes = [CellComplex, c_int]

I separated out the function pointer definitions from the class method definitions. There's no need to redefine a function pointer's restype and argtypes every time a method is called. If the function returns a variably sized array, you're better off setting it to a pointer type. You can slice the result to a list or cast to an array type.

CellComplex is initialized by a sequence p of floating point pairs such as [[0.1, 0.2], [0.3, 0.4], [0.5, 0.6]].

I eliminated the c_cellComplex class. You can just set the instance's value.

lib.new_cellComplex returns an instance of CellComplex, but ctypes bypasses __new__ and __init__ when CellComplex is used as a restype, so that's not an issue. It would be less twisted to instead override __new__, but you'd still have to override c_void_p.__init__.

The dimension attribute needs to be a property that calls an exported function, instead of relying on static data in the Python object.