vsinstr.exe instrumented large executable performance

Question

I would like to use Visual Studio vsinstr.exe tool for instrumenting an unmanaged c++ executable (legacy app). It is a very large project and this would be a way how to map our huge test automation content to actual code, to identify what test cases are affected when a change is made to code base.

I'm however concerned about performance of such instrumented executable, because we basically need to run whole test automation content to get coverage data (or update it when code is changed) and this would be done each night. To get the picture, the test automation run takes maybe 10 hours (GUI tests, no unit tests because of legacy architecture)

Does anybody have real experience regarding performance of instrumented executables?

Answer 1

I realize this question is getting long in the tooth (getting old) so my answer is intended for other users who stumble across this question.

From my real world experience, instrumented binaries do run significantly slower, often by orders of magnitude. However, I have only instrumented MANAGED binaries and the OP specifically stated unmanaged C++ so "your mileage may vary."

My suggestion would be to run a subset of the tests that take between 2-3 minutes. Run that subset 3 times and average the actual run time results. Then instrument the binaries and run the same tests 3 times and compute the average. Fewer tests and the data might be skewed because of application initialization. More tests and you MAY end up waiting an hour for each instrumented test.