PyString_FromStringAndSize causes segmentation fault

Question

I have a weird segfault with python. Here's the problematic piece of code:

const std::string &fullName = child.getFullName();
const char *fName = fullName.c_str();
const int len = fullName.size();

printf(":: %02d --> %s\n", len, fName);

PyObject *item = PyString_FromStringAndSize(fName, len);
PyList_Append( list, item);

I put the printf there, hoping to get a hint. But the length and the fName values are correct. There's no null values there!

And here's the trace that I get from gdb:

#0  0x0000000000423e00 in PyObject_Malloc ()
#1  0x0000000000513a3f in PyString_FromStringAndSize ()
#2  0x00007ffff554aa5e in recurseObject (list=0x9f5f38, obj=...)
    at file.cpp:59
#3  0x00007ffff554aa74 in recurseObject (list=0x9f5f38, obj=...)
    at file.cpp:62
#4  0x00007ffff554aa74 in recurseObject (list=0x9f5f38, obj=...)
    at file.cpp:62
#5  0x00007ffff554ad27 in listObjects (self=<optimized out>, args=<optimized out>)
    at file.cpp:73

Any idea ? I try to look around, but I couldn't find anything precise about the subject.

EDIT: just to add more fuzziness to this problem. This bug occurs only when I'm using it inside a script. When I calling it from the command line, inside Python, everything works fine!

EDIT: here's what I get from valgrind:

==27681== Invalid read of size 8
==27681==    at 0x423E00: PyObject_Malloc (in /usr/bin/python2.7)
==27681==    by 0x513A3E: PyString_FromStringAndSize (in /usr/bin/python2.7)
==27681==    by 0x7953DB4: recurseObjectChildren(_object*, Alembic::Abc::v4::IObject const&) (iarchive.cpp:55)
==27681==    by 0x7953DCA: recurseObjectChildren(_object*, Alembic::Abc::v4::IObject const&) (iarchive.cpp:58)
==27681==    by 0x7953DCA: recurseObjectChildren(_object*, Alembic::Abc::v4::IObject const&) (iarchive.cpp:58)
==27681==    by 0x7953F26: iArchive_getIdentifiers(_object*, _object*) (iarchive.cpp:69)
==27681==    by 0x498909: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==27681==    by 0x498601: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==27681==    by 0x498601: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==27681==    by 0x49F1BF: PyEval_EvalCodeEx (in /usr/bin/python2.7)
==27681==    by 0x4A9080: PyRun_FileExFlags (in /usr/bin/python2.7)
==27681==    by 0x4A9310: PyRun_SimpleFileExFlags (in /usr/bin/python2.7)
==27681==  Address 0xffffffffffffff00 is not stack'd, malloc'd or (recently) free'd
==27681== 
==27681== 
==27681== Process terminating with default action of signal 11 (SIGSEGV)
==27681==  Access not within mapped region at address 0xFFFFFFFFFFFFFF00
==27681==    at 0x423E00: PyObject_Malloc (in /usr/bin/python2.7)
==27681==    by 0x513A3E: PyString_FromStringAndSize (in /usr/bin/python2.7)
==27681==    by 0x7953DB4: recurseObjectChildren(_object*, Alembic::Abc::v4::IObject const&) (iarchive.cpp:55)
==27681==    by 0x7953DCA: recurseObjectChildren(_object*, Alembic::Abc::v4::IObject const&) (iarchive.cpp:58)
==27681==    by 0x7953DCA: recurseObjectChildren(_object*, Alembic::Abc::v4::IObject const&) (iarchive.cpp:58)
==27681==    by 0x7953F26: iArchive_getIdentifiers(_object*, _object*) (iarchive.cpp:69)
==27681==    by 0x498909: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==27681==    by 0x498601: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==27681==    by 0x498601: PyEval_EvalFrameEx (in /usr/bin/python2.7)
==27681==    by 0x49F1BF: PyEval_EvalCodeEx (in /usr/bin/python2.7)
==27681==    by 0x4A9080: PyRun_FileExFlags (in /usr/bin/python2.7)
==27681==    by 0x4A9310: PyRun_SimpleFileExFlags (in /usr/bin/python2.7)
==27681==  If you believe this happened as a result of a stack
==27681==  overflow in your program's main thread (unlikely but
==27681==  possible), you can try to increase the size of the
==27681==  main thread stack using the --main-stacksize= flag.
==27681==  The main thread stack size used in this run was 8388608.
==27681== 
==27681== HEAP SUMMARY:
==27681==     in use at exit: 5,126,103 bytes in 5,336 blocks
==27681==   total heap usage: 14,574 allocs, 9,238 frees, 12,498,954 bytes allocated
==27681== LEAK SUMMARY:
==27681==    definitely lost: 76 bytes in 3 blocks
==27681==    indirectly lost: 240 bytes in 10 blocks
==27681==      possibly lost: 671,180 bytes in 1,124 blocks
==27681==    still reachable: 4,454,607 bytes in 4,199 blocks
==27681==         suppressed: 0 bytes in 0 blocks
==27681== Reachable blocks (those to which a pointer was found) are not shown.
==27681== To see them, rerun with: --leak-check=full --show-reachable=yes

Not really sure what that means! And I still don't understand why it fails there. I tried different approaches like cloning the string fullName to be sure it's new memory allocated. Even used a new char[] and copy the full name in it. Still the same problem, at the same place.

Any idea anyone ? Is it something common in Python ?

Answer 1

I don't know how to explain this, but this was definitely cause because the heap was corrupted somewhere before. So I managed to track the source of the error.

FTW: something like:

Bob bob;
new (&bob) Bob(someParameter);

might not be that super safe! In some cases, it's perfect, but not everywhere.