Passing io._BufferedReader to c-function

Question

This question is a follow-up to my previous question, where I tried to compile python-yenc for Python3. After being told there wasn't a quick fix, I decided to take up the challange and rewrite it completely.

The only thing I can't figure out is how to use PyArg_ParseTupleAndKeywords with io-objects. Here is the relevant code:

PyObject *in_file, *out_file;
static char *kwlist[] = { "infile", "outfile", NULL };

if(!PyArg_ParseTupleAndKeywords(args, kwds, "O!O!|l", kwlist,\
                     &PyBytes_Type, &in_file,\
                     &PyBytes_Type, &out_file,\
                     &bytes)) return NULL;

which obviously yields

Traceback (most recent call last):
  File "test.py", line 21, in <module>
    print(_yenc.decode_file(b), outfile=o)
TypeError: argument 1 must be bytes, not _io.BufferedReader

How can I pass _io.BufferedReader-objects to my function?

Thanks, Martijn

Answer 1

Mzialla, you should not be using "O!" in PyArg_ParseTupleAndKeywords. Doing so means that you cannot pass any object other than the specified type. I believe the typical approach to interfacing with files in Python 3.2 extensions is not to assume any particular type, but program against the "protocol".

So you should do something like:

if(!PyArg_ParseTupleAndKeywords(args, kwds, "OO|l", kwlist,\
             &in_file, &out_file, &bytes))
    ...

And after that, you have two options: either you interact with the stream object using its Python methods, or you obtain the stream's file descriptor (PyObject_AsFileDescriptor) and operate on that using the OS-level read/write functions (or equivalent).

For the Python-method approach, you should obtain the "read" method, and invoke that instead of fread. Something along the following lines (untested):

PyObject *read = PyObject_GetAttrString(in_file, "read");
if (!read) handle_error;
while(encoded < bytes || bytes == 0) {
    PyObject *bytes_obj= PyObject_CallFunction(read, "i", 1);
    if (!bytes || !PyBytes_Check(bytes_obj)) handle_error;
    char *s = PyBytes_AS_STRING(bytes_obj);
    ...
}

Then you would need to do something similar for the write side.

Answer 2

I have downloaded yenc (https://bitbucket.org/dual75/yenc) and tried the following test program but not successful.

#include <stdio.h>
#include "_yenc.h"

int main()
{
    PyObject *in_file, *out_file;
    PyObject *args;
    PyObject *kwds;
    int bytes;
    static char *kwlist[] = { "infile", "outfile", NULL };

    if(!PyArg_ParseTupleAndKeywords(args, kwds, "O!O!|l", kwlist,\
                 &PyBytes_Type, &in_file,\
                 &PyBytes_Type, &out_file,\
                 &bytes)) {
        printf("Yay!\n");
    } else {
        printf("Bad luck :( \n");
    }

    return 0;
}

I compiled the above code using the following:

gcc test.c -I/usr/include/python2.7/

and ran into the following issues:

/tmp/ccRasW2G.o: In function `main':
/home/sangeeth/work/github/yenc/yenc/src/test.c:14: undefined reference to `PyString_Type'
/home/sangeeth/work/github/yenc/yenc/src/test.c:14: undefined reference to `PyString_Type'
/home/sangeeth/work/github/yenc/yenc/src/test.c:14: undefined reference to `PyArg_ParseTupleAndKeywords'
collect2: ld returned 1 exit status

So, could you please share you steps to reproduce this issue?

OTOH, I see

PyArg_ParseTupleAndKeywords (in src/_yenc.c) 

which takes both

&PyFile_Type 

and

&PyString_Type 

and compiles fine (python setup.py build). So what is the problem you're seeing?

Answer 3

This works:

PyObject *decode_file(PyObject *pPySelf, 
              PyObject* pPyArgs, 
              PyObject* pPyKeywords) {
  uLong     totDecodedBytes = 0;
  uInt      decodedBytes;
  uLong     bytesToRead;
  uLong     bytesToDecode = 0;
  Bool      escape = 0;
  Crc32     crc;
  PyObject *pPyInFile = NULL;
  PyObject *pPyOutFile = NULL;
  if (!PyArg_ParseTupleAndKeywords(pPyArgs,
                   pPyKeywords,
                   "OO|l",
                   argnames,
                   &pPyInFile,
                   &pPyOutFile,
                   &bytesToDecode)) {
    return NULL;
  }
  crc_init(&crc, 0xffffffffl);
  while (totDecodedBytes < bytesToDecode) {
    char      *readBuffer;
    Byte       writeBuffer[LONGBUFF];
    Py_ssize_t readBytes;
    PyObject  *pPyReadBuffer;
    bytesToRead = bytesToDecode - totDecodedBytes;
    if (bytesToRead > BLOCK) {
      bytesToRead = BLOCK;
    } else if (bytesToRead == 0) {
      break;
    }
    pPyReadBuffer = PyObject_CallMethod(pPyInFile, "read", "l", bytesToRead);
    if (NULL == pPyReadBuffer) {
      break;
    }
    PyBytes_AsStringAndSize(pPyReadBuffer, &readBuffer, &readBytes);
    decodedBytes = decode_buffer((Byte*)readBuffer,
                 writeBuffer,
                 readBytes,
                 &crc,
                 &escape);
    PyObject_CallMethod(pPyOutFile, "write", "y#", writeBuffer, decodedBytes);
    totDecodedBytes += decodedBytes;
  }
  return Py_BuildValue("(l,L)", totDecodedBytes, (long long)crc.crc);
}