Using Py_buffer and PyMemoryView_FromBuffer with different itemsizes

Question

This question is related to a previous question I asked. Namely this one if anyone is interested. Basically, what I want to do is to expose a C array to Python using a Py_buffer wrapped in a memoryview-object. I've gotten it to work using PyBuffer_FillInfo (work = I can manipulate the data in Python and write it to stdout in C), but if I try to roll my own buffer I get a segfault after the C function returns.

I need to create my own buffer because PyBuffer_FillInfo assumes that the format is char, making the itemsize field 1. I need to be able to provide items of size 1, 2, 4 and 8.

Some code, this is a working example:

Py_buffer *buf = (Py_buffer *) malloc(sizeof(*buf));
int r = PyBuffer_FillInfo(buf, NULL, malloc(sizeof(char) * 4), 4, 0, PyBUF_CONTIG);
PyObject *mv = PyMemoryView_FromBuffer(buf);
//Pack the memoryview object into an argument list and call the Python function
for (blah)
  printf("%c\n", *buf->buf++); //this prints the values i set in the Python function

Looking at the implementation of PyBuffer_FillInfo, which is really simple, I rolled my own function to be able to provide custom itemsizes:

//buffer creation function
Py_buffer *getReadWriteBuffer(int nitems, int itemsize, char *fmt) {
  Py_buffer *buf = (Py_buffer *) malloc(sizeof(*buf));
  buf->obj = NULL
  buf->buf = malloc(nitems * itemsize);
  buf->len = nitems * itemsize;
  buf->readonly = 0;
  buf->itemsize = itemsize;
  buf->format = fmt;
  buf->ndim = 1;
  buf->shape = NULL;
  buf->strides = NULL;
  buf->suboffsets = NULL;
  buf->internal = NULL;
  return buf;
}

How i use it:

Py_buffer *buf = getReadWriteBuffer(32, 2, "h");
PyObject *mv = PyMemoryView_FromBuffer(buf);
// pack the memoryview into an argument list and call the Python function as before

for (blah)
  printf("%d\n", *buf->buf); //this prints all zeroes even though i modify the array in Python

return 0;
//the segfault happens somewhere after here

The result of using my own buffer object is a segfault after the C function returns. I really don't understand why this happens at all. Any help would be most appreciated.

EDIT According to this question, which I failed to find before, itemsize > 1 might not even be supported at all. Which makes this question even more interesting. Maybe I could use PyBuffer_FillInfo with a large enough block of memory to hold what I want (32 C floats for example). In that case, the question is more about how to assign Python floats to the memoryview object in the Python function. Questions questions.

Answer 1

So, in lack of answers I decided to take another approach than the one I originally intended. Leaving this here in case someone else hits the same snag.

Basically, instead of creating a buffer (or bytearray, equiv.) in C and passing it to Python for the extension user to modify. I simply redesigned the code slightly, so that the user returns a bytearray (or any type that supports the buffer interface) from the Python callback function. This way I need not even worry about the size of the items since, in my case, all the C code does with the returned object is to extract its buffer and copy it to another buffer with a simple memcpy.

Code:

PYGILSTATE_ACQUIRE; //a macro i made
PyObject *result = PyEval_CallObject(python_callback, NULL);
if (!PyObject_CheckBuffer(result))
  ; //raise exception

Py_buffer *view = (Py_buffer *) malloc(sizeof(*view));
int error = PyObject_GetBuffer(result, view, PyBUF_SIMPLE);
if (error)
  ; //raise exception

memcpy(my_other_buffer, view->buf, view->len);

PyBuffer_Release(view);
Py_DECREF(result);
PYGILSTATE_RELEASE; //another macro

I hope this helps someone.