How to generate a post facto backtrace?

Question

When my KDE programs crash (which is pretty often :( ), I can generate a post-facto backtrace (I imagine it's not really post-facto, just that the errors are caught and saved) and use it to submit a bug report if I have the debug symbols installed. How does that work and how can I add that feature to my own programs?

I now have a problem with a program that I have written (in C++) that crashes intermittently, apparently due to memory mismanagement. It is highly impractical for me to run the program under gdb because it is a large-scale simulation which requires several hours to run and the crash only appears when the size of the system is very high. Being able to automatically dump backtraces into a file would save me many hours.

I assume it involves wrapping everything in some sort of try{}catch(){} routine but how do I get useful backtraces out of that? Is there a better way? All of my programming is on Linux, if that affects the answer.

Answer 1

You can't use try/catch, because those require a correct program, and the crashes you experience are because your program is ill-formed/incorrect/broken. You cannot in general use deterministic programming to work around non-deterministic broken code. Avoid writing broken code in the first place (use tools like Clang's asan/tsan/ubsan, and Valgrind, and write tests), or use a language that doesn't become ill-formed when you make programming errors (like Java or Python).

What normally happens is that when the operating system kills your process because of some illegal operation (e.g. illegal instructions or illegal memory access), it creates a core dump of the process just before it gets removed from the system. The core dump consists of the entire memory content (more or less), and it includes the stack traces of all the running threads.

Some contemporary OSs pipe the coredump to a program that handles it, e.g. by uploading it to a software vendor for analysis. You could do something similar, though you'll have to arrange that with your OS. It's probably enough just to send the stack traces of the running threads rather than the entire memory, though.

You can also create stack traces of a running program, either by using a library (libunwind?) or by attaching a debugger, which interrupts the program, but there's generally not much use for this - the only stack trace that is interesting is the one from where the illegal operation happened, because you want to know what that operation was.

Answer 2

Depending on the kind of crash you encounter, you may also be able to perform some kind of logging operations in your programs terminate handler.

How much debugging information is available at that point depends a lot on your compiler and its settings, and of course there is no guarantee this will work in every situation as you may be, for example, out of memory.

Answer 3

You can integrate google breakpad. Or you can at least take a look the sources. AFAIK it creates dumps by providing various signal handlers.