Python memory debugging with GDB

Question

We have a Linux application that makes use of OpenSSL's Python bindings and I suspect it is causing random crashes. Occasionally, we see it crash with the message:

Python Fatal Error: GC Object already tracked

which would appear to be either a programming error on the part of the library, or a symptom of memory corruption. Is there any way to know the last line of Python source code it executed, given a core file? Or if it is attached in GDB? I realize it is probably all compiled bytecode, but I'm hoping there someone out there may have dealt with this. Currently it is running with the trace module active and we're hoping it will happen again, but it could be a long while.

Answer 1

Yes, you can do this kind of thing:

(gdb) print PyRun_SimpleString("import traceback; traceback.print_stack()")
  File "<string>", line 1, in <module>
  File "/var/tmp/foo.py", line 2, in <module>
    i**2
  File "<string>", line 1, in <module>
$1 = 0

It should also be possible to use the pystack command defined in the python gdbinit file, but it's not working for me. It's discussed here if you want to look into it.

Also, if you suspect memory issues, it's worth noting that you can use valgrind with python, if you're prepared to recompile it. The procedure is described here.

Answer 2

If you have mac or sun box kicking around you could use dtrace and a version of python compiled with dtrace to figure out what the application was doing at the time. Note: in 10.5 python is pre-compiled with dtrace which is really nice and handy.

If that isn't available to you, then you can import gc and enable debugging which you can then put out to a log file.

To specifically answer your question regarding debugging with GDB you might want to read "Debugging With GDB" on the python wiki.

Answer 3

If you're using CDLL to wrap a C library in python, and this is 64-bit linux, there's a good chance that you're CDLL wrapper is misconfigured. CDLL defaults to int return types on all platforms (should be a long long on 64-bit systems) and just expects you to pass the right arguments in. You may need to verify the CDLL wrapper in this case...

Answer 4

In addition to all above one can quickly implement an adhoc tracer via the trace module.