Undebuggable non-deterministic heisenbug in single-threaded C++ function call

Question

I'm at the end of my rope here: I have a single-threaded C++ program. Here is some empirical data and background information, I tried to highlight the most important keywords;

• The entire section I'm talking about does not have any syscalls, other than the memory (de-)allocation calls the standard C++ library may perform (std::sets are involved). It's a purely logical algorithm.
• The behaviour of this should be deterministic, depending on the input, which I do not vary.
• If the bug manifests itself, the program simply falls into what looks like an endless loop where it seems to start allocating memory beyond any bound.
• The bug does not manifest itself predictably, I can run the program from the command line and sometimes (perhaps 30%-50%) the bug manifests itself, otherwise, everything runs smoothly and correctly as far as I can tell.
• Once I run the program not directly from the prompt, but in gdb or valgrind, the bug is gone, the program never dies.
• Now comes the best part: I traced the problem to a (templated) non-virtual member function call. Just before the call, I print a message to std::cout, which I can see in the terminal. The first line inside the function also has a debug message, which is never shown.

I don't see any reasonable explanation any more. Maybe you can come up with an idea how to proceed.

---

Edit: The significant lines of code, I changed the line numbers so we can refer to them and omitted irrelevant parts, so not everything seems to make the best sense.

a.cpp

 10     std::set<Array const*>* symbols;
 11     std::set<Array const*> allSymbols;
 12     symbols = &allSymbols;
 //  ... allSymbols are populated with std::inserter
 15     std::cout << "eval; cd = " << &cd << ", cg = " << &cd.cg << std::endl;
 16     senderConstraints = cd.cg.eval(*symbols);

b.cpp

 31     template <typename ArrayContainer>
 32     ConstraintList eval(ArrayContainer const request) {
 33       std::cout << "inside eval ... going to update graph now" << std::endl;

The last line of output is:

eval; cd = 0x2e6ebb0, cg = 0x2e6ebc0

Then it's trapped in the endless loop.

Answer 1

I bet, the second line is printed, when you change

ConstraintList eval(ArrayContainer const request)

to

ConstraintList eval(ArrayContainer const & request)

If so, either the state of allSymbols is corrupted between line 12 and line 15, or your code really looks more like this:

std::set<Array const*>* symbols;
{
    std::set<Array const*> allSymbols;
    symbols = &allSymbols;
    //  ... allSymbols are populated with std::inserter
}
std::cout << "eval; cd = " << &cd << ", cg = " << &cd.cg << std::endl;
senderConstraints = cd.cg.eval(*symbols);

Which is UB, because symbols refers to an already destructed object.