Python 3 C-API IO and File Execution

Question

I am having some serious trouble getting a Python 2 based C++ engine to work in Python3. I know the whole IO stack has changed, but everything I seem to try just ends up in failure. Below is the pre-code (Python2) and post code (Python3). I am hoping someone can help me figure out what I'm doing wrong.I am also using boost::python to control the references.

The program is supposed to load a Python Object into memory via a map and then upon using the run function it then finds the file loaded in memory and runs it. I based my code off an example from the delta3d python manager, where they load in a file and run it immediately. I have not seen anything equivalent in Python3.

---

Python2 Code Begins here:

    // what this does is first calls the Python C-API to load the file, then pass the returned
    // PyObject* into handle, which takes reference and sets it as a boost::python::object.
    // this takes care of all future referencing and dereferencing.
    try{
        bp::object file_object(bp::handle<>(PyFile_FromString(fullPath(filename), "r" )));
        loaded_files_.insert(std::make_pair(std::string(fullPath(filename)), file_object));
    }
    catch(...)
    {
        getExceptionFromPy();
    }

Next I load the file from the std::map and attempt to execute it:

    bp::object loaded_file = getLoadedFile(filename);
    try
    {
        PyRun_SimpleFile( PyFile_AsFile( loaded_file.ptr()), fullPath(filename) );
    }
    catch(...)
    {
        getExceptionFromPy();
    }

---

Python3 Code Begins here: This is what I have so far based off some suggestions here... SO Question Load:

        PyObject *ioMod, *opened_file, *fd_obj;

        ioMod = PyImport_ImportModule("io");
        opened_file = PyObject_CallMethod(ioMod, "open", "ss", fullPath(filename), "r");

        bp::handle<> h_open(opened_file);
        bp::object file_obj(h_open);
        loaded_files_.insert(std::make_pair(std::string(fullPath(filename)), file_obj));

Run:

    bp::object loaded_file = getLoadedFile(filename);
    int fd = PyObject_AsFileDescriptor(loaded_file.ptr());
    PyObject* fileObj = PyFile_FromFd(fd,fullPath(filename),"r",-1,"", "\n","", 0);

    FILE* f_open = _fdopen(fd,"r");

    PyRun_SimpleFile( f_open, fullPath(filename) );

Lastly, the general state of the program at this point is the file gets loaded in as TextIOWrapper and in the Run: section the fd that is returned is always 3 and for some reason _fdopen can never open the FILE which means I can't do something like PyRun_SimpleFile. The error itself is a debug ASSERTION on _fdopen. Is there a better way to do all this I really appreciate any help.

If you want to see the full program of the Python2 version it's on Github

Answer 1

So this question was pretty hard to understand and I'm sorry, but I found out my old code wasn't quite working as I expected. Here's what I wanted the code to do. Load the python file into memory, store it into a map and then at a later date execute that code in memory. I accomplished this a bit differently than I expected, but it makes a lot of sense now.

1. Open the file using ifstream, see the code below
2. Convert the char into a boost::python::str
3. Execute the boost::python::str with boost::python::exec
4. Profit ???

Step 1)

vector<char> input;
ifstream file(fullPath(filename), ios::in);
if (!file.is_open())
{
    // set our error message here
    setCantFindFileError();
    input.push_back('\0');
    return input;
}

file >> std::noskipws;
copy(istream_iterator<char>(file), istream_iterator<char>(), back_inserter(input));
input.push_back('\n');
input.push_back('\0');

Step 2) bp::str file_str(string(&input[0])); loaded_files_.insert(std::make_pair(std::string(fullPath(filename)), file_str)); Step 3)

bp::str loaded_file = getLoadedFile(filename);
// Retrieve the main module
bp::object main = bp::import("__main__");
// Retrieve the main module's namespace
bp::object global(main.attr("__dict__"));
bp::exec(loaded_file, global, global);

Full Code is located on github: