Debugging: Tracing (and diff-ing) function call tree of two version of the same program

Question

I'm working on the rewriting of some code in a c++ cmd line program.

I changed the low level data structure that it uses and the new version passes all the tests (quite a lot) without any problem and I get the correct output from both the new and the old version... Still, when give certain input, they give different behaviour.

Getting to the point: Being somewhat of a big project I don't have a clue about how to track down when the execution flow diverges, so... is there way to trace the function call tree (possibly excluding std calls) along with, i don't know, line number in the source file and source name? Maybe some gcc or macro kungfu?

I would need a Linux solution since that's where the program runs.

Answer 1

Still, when give certain input, they give different behaviour

I would expand logging in you old and new versions in order to understand better work of your algorithms for certain input. When it become clearer you can for example use gdb if you still need it.

Update

OK, As for me logging is OK, but you do not want to add it.

Another method is tracing. Actually I used it only on Solaris but I see that it exists also on Linux. I have not used it on Linux so it is just an idea that you can test.

You can use SystemTap

User-Space Probing
SystemTap initially focused on kernel-space probing. However, there are many instances where userspace probing can help diagnose a problem. SystemTap 0.6 added support to allow probing userspace processes. SystemTap includes support for probing the entry into and return from a function in user-space processes, probing predefined markers in user-space code, and monitoring user-process events.

I can gurantee that it will work but why don't give it a try?

There is even an example in the doc:

If you want to see how the function xmalloc function is being called by the command ls, you could use the user-space backtrack functions to provide that information.

stap -d /bin/ls --ldd \
-e 'probe process("ls").function("xmalloc") {print_ustack(ubacktrace())}' \
-c "ls /"