Pass array of structs from Python to C

Question

[Update: Problem solved! See bottom of the post]

I need to allow python developers to pass an array of packed data (in this case vertices) into my API, which is a series of C++ interfaces exposed manually through the Python C API. My initial impression with this is to use the ctypes Structure class to allow for an interface like this:

class Vertex(Structure):
_fields_ = [
    ('x', c_float),
    ('y', c_float),
    ('z', c_float),
    ('u', c_float),
    ('v', c_float),
    ('color', c_int)
] 

verts = (Vertex * 3)()
verts[0] = Vertex(0.0, 0.5, 0.0, 0.0, 0.5, 0xFF0000FF)
verts[1] = Vertex(0.5, -0.5, 0.0, 0.5, -0.5, 0x00FF00FF)
verts[2] = Vertex(-0.5, -0.5, 0.0, -0.5, -0.5, 0x0000FFFF)

device.ReadVertices(verts, 3) # This is the interfaces to the C++ object

Where the function I'm trying to pass to has the following signature:

void Device::ReadVertices(Vertex* verts, int count);

And the Python wrapper looks something like this:

static PyObject* Device_ReadVertices(Py_Device* self, PyObject* args)
{
    PyObject* py_verts;
    int count;

    if(!PyArg_ParseTuple(args, "Oi", &py_verts, &count)) 
        return NULL;

    // This Doesn't Work!
    Vertex* verts = static_cast<Vertex*>(PyCObject_AsVoidPtr(py_verts));

    self->device->ReadVertices(verts, count);

    Py_RETURN_NONE;
}

Of course, the biggest issue I have is this: I can retrieve the PyObject for the struct, but I have no idea how I would cast it to the correct type. The above code fails miserably. So how exactly would I go about allowing the user to pass me this kind of data from Python?

Now, a couple of things to consider: First is that I already have quite a bit of my Python/C++ layer written, and am perfectly happy with it (I moved away from SWIG so I could have more flexibility). I don't want to re-code it, so I would prefer a solution that works with the C API natively. Second, I do intend to have the Vertex structure be pre-defined in my C++ code, so I would prefer to not have the user need to re-define it in the Python (cuts down on errors that way), but I'm not sure how to expose a contiguous structure like that. Third, I have no reason for trying the ctypes structure aside from not knowing another way to do it. Any suggestions are welcome. Finally, since this is (as you may have guessed) for a graphics app I would prefer a faster method over a convenient one, even if the faster method takes a little bit more work.

Thanks for any help! I'm still feeling my way around python extensions, so it's a great help to get community input on some of the stickier parts.

[SOLUTION]

So first off, thanks to everyone who pitched in their ideas. It was a lot of little tidbits that added up to the eventual answer. In the end here is what I found: Sam's suggestion of using struct.pack ended up being right on the money. Seeing that I'm using Python 3, I had to tweak it ever so slightly, but when all was said and done this actually got a triangle showing up on my screen:

verts = bytes()
verts += struct.pack("fffffI", 0.0, 0.5, 0.0, 0.0, 0.5, 0xFF0000FF)
verts += struct.pack("fffffI", 0.5, -0.5, 0.0, 0.5, -0.5, 0x00FF00FF)
verts += struct.pack("fffffI", -0.5, -0.5, 0.0, -0.5, -0.5, 0x0000FFFF)

device.ReadVertices(verts, 3)

With my tuple parsing now looking like this:

static PyObject* Device_ReadVertices(Py_Device* self, PyObject* args)
{
    void* py_verts;
    int len, count;

    if(!PyArg_ParseTuple(args, "y#i", &py_verts, &len, &count)) 
        return NULL;

    // Works now!
    Vertex* verts = static_cast<Vertex*>(py_verts);

    self->device->ReadVertices(verts, count);

    Py_RETURN_NONE;
}

Note that even though I don't use the len variable in this example (though I will in the final product) I need to parse the tuple using 'y#' instead of just 'y' or else it will stop at the first NULL (according to the documentation). Also to be considered: void* casts like this are quite dangerous, so please do loads more error checking than I show here!

So, job well done, happy day, pack up and go home, yes?

Wait! Not so fast! There's MORE!

Feeling good about how that all worked out I decided, on a whim, to see if my previous attempt still blew up on me and reverted back to the first snippet of python in this post. (Using the new C code, of course) and... it worked! The results were identical to the struct.pack version! Wow!

So this means your users have a choice in how they're going to provide this kind of data, and your code can handle either with no changes. Personally I'm going to encourage the ctype.Structure method, since I think it makes for easier readability, but really it's whatever the user is comfortable with. (Heck, they could manually type out a string of bytes in hex if they wanted to. It works. I tried.)

Honestly I think this is the best possible outcome, so I'm ecstatic. Thank you all again, and good luck to anyone else who runs into this problem!

Answer 1

Not tested but you should give this a try and let us know if its fast enough for your needs.

On the python side, pack the vertices into a string instead of an object.

str = "" # byte stream for encoding data
str += struct.pack("5f i", vert1.x, vert1.y, vert1.z, vert1.u, vert1.v, vert1.color) # 5 floats and an int
# same for other vertices

device. ReadVertices( verts, 3) # send vertices to C library

On the C library/python wrapper, modify your PyArgs_ParseTuple to use the format string "si". This will convert your python string into a C string (char*) which you can then typecast as a pointer to your vector struct. At this point the C string is a stream of bytes/words/floats and should be what you're looking for.

Good luck!

Answer 2

The easiest thing I can see to do would be to just avoid the issue altogether and expose a Device_ReadVertex that takes in x, y, z, u, v and color as arguments. This has obvious drawbacks, like making the Python programmers feed it vertices one by one.

If that's not good enough (seems likely it isn't), then you could try defining a new Python type as described here. It's a bit more code but I think this is the "more architecturally sound" method, because you ensure your Python developers are using the same type definition as you are in the C code. It also allows for a bit more flexibility than a simple struct (it's really a class, with the potential to add methods, etc), which I'm not sure you actually need but it might come in handy later.