How to diagnose / profile momentary performance hit, C++

Question

Solved: For when simple profiling isn't effective enough, I have written a tool to show me where performance hits occur. Basic information about how the tool works is in the accepted answer below. The source can be found here: http://pastebin.com/ETiW8hE8 (be sure to turn debugging symbols on in the program you're testing)

I've built a game engine in C++ and I have noticed in one particular area of a level that there is a brief performance hit. The game will stop completely for about half a second, and then continue on merrily. I've tried to profile this, but it's difficult isolate the condition since I also have to load the map and perform the in-game task which causes the performance hit. I can make a map load automatically and skip showing menus, etc, and comparing those profile results against a set of similar control data (all the same steps but without actually initiating the performance hit), but it doesn't show anything obvious.

I'm using gmon to profile.

This is a large application with many, many classes and functions. The performance hit only happens once, so there's no way to just trigger the problem many times during one execution to saturate my profiling results in order to make the offending functions more obvious in the profiling results.

What else can I do?

Answer 1

What I would do is try to grab a stack sample in that half second when it's frozen. This would require an alarm clock timer set to go off some small time in the future, like 100ms. Then in some loop, like the frame display loop, that normally takes less than 100ms to repeat, keep resetting the timer. That way, it will act as a watchdog that barks if you don't keep petting it.

Then, stick a breakpoint in the timer interrupt handler. When it gets there, you know you're in the bad slice of time. Then just display the call stack, and it should show you what the problem is.

You might have to repeat the process a few times.

Answer 2

You are not saying anything about whether your application is threaded, but I will assume that it is not.

As per suggestion from mike, get insights by getting a stack trace at and see where it is freezing, you can do that with a bit of luck using pstack, so

   while usleep 100000; do 
      pstack processid
   done >/tmp/stack.log

Should give you some output to go on -- my guess is that you are calling a blocking IO operation, like reading some assets from disk.