Why gdb goes back and forth when debugging MPI program?

Question

By any chance, has anyone experienced the following situation?

For debugging purpose, I used only one mpi process by launching the mpi-based program with "mpi-run -np 1". However, when I debugged the program, repeatetive step-in and step-over happens quite often.

So, let's say, I followed the source code line by line until I've reached a point of interest. Then I tried to step-in, and type "n" expecting to proceed one line. However, the debugger goes back to the first line of the function. Only after I've experience this twice or three times, I can proceed.

Impression is that the debugger is not doing something wrong, since the result is thought to be correct. I am really curious about the reason as to why it's happening.

Thanks in advance!

Answer 1

The observed behaviour is usually a result of compiler optimisations being active. Optimisations might result in the binary code not following completely the structure of the source - it still gives the same result, but the compiler rearranges the operations in such a way as to be more efficiently executed. Also, some functions might get inlined. As a result the correspondence between range of instructions and source lines in the debug information becomes useless.

When doing source level debugging, always make sure that optimisation is turned off using the -O0 flag for most compilers or by not giving any optimisation flags for Sun/Oracle compilers. Note that some compiler options might result in elevated optimisation levels. For example, enabling OpenMP support in Sun/Oracle Studio with -xopenmp automatically raises the optimisation level to -xO3. Instead, -xopenmp=noopt should be used and no optimisation level specified explicitly. Also, some compilers optimise by default, for example Intel's compiler.