CruiseControl Build times degrade on vmware workstation

Question

I have a very strange problem and my last hope for help I suspect comes from this community.

I have a build system that incorporates compiling three different elements of a wise installer package. The symptoms that I am seeing is that build times for these projects specifically degrade over time on the virtual machines. This is occurring on multiple virtual machines and has been doing so since November of 2013. Fortunately I had a habit of cloning the virtual machines and had a clone of the machine in early December where the symptoms were in the early stages.

For instance a normal build time should finish in 48 to 50 minutes. Times degraded slowly that by the time I noticed the problems times had degraded to 1hr 45min. I am not normally monitoring performance of the system but the results of the build - so I never knew. The cloned machine I had will restore the system so that it builds at 1hr 12 min approx.

Analyzing the build times, all of the time is being used by the wise installer. I have attempted uninstall and reinstalling the application. I have cleaned the temp directory, run chkdsk, and other normal debugging operations.

One of the wise installer projects is a merge module that requires recompile because it updates a database file. This should only take 8 minutes to compile. It takes over 1/2 hour.

Can anyone think of what I could look for to diagnose this problem? What could degrade the system so that in the course of a month the wise installer compile can loose up to 45 minutes on its build time? Build Machine OS: XPsp3 HDD: SSD Other builds do run on the same host machine and can run concurrently but previous to November 2013 this did not have an impact on performance.

Answer 1

The problem is actually with our visual studio build.

For each build we increment the version number of binaries. We have a process that executes and updates the value in all proj files.

When visual studio compiles it adds assembly information to the registry for every .net file. Since the version number changed, the registry entry changes. The registry is never cleaned up. So I have just accumulated a year or more of registry entries.

When wise installer runs it is scanning through the registry, I am not certain why right now. Using ProcMon we can see the process running and reading the reg. Since the registry has bloated so much it slows down the build time.

Now the big question will be how to prevent this problem on a new build machine? How do I clean up all CLSID entries for our build?

On a pristine Win7 build system the entire build completes in under 20 min!

Also, I made one more change to the wise studio settings for the project. Instead of using high compression I switched it to mszip compression. Our output file is 50MB larger but the build time is far quicker - even on a new machine.

SOLUTION:

Add a clean solution task BEFORE changing the version number. This one change has resulted in a very consistent build time over several months.

The problem was that the company had changed the process of changing the assembly version and file version values. Little did we know that a rebuild was leaving assembly information in the registry on every build. So the task list now looks like:

• Clean .NET solutions
• Increment the version for file version and assembly version.
• Build .NET solutions

Hopefully this helps anyone else who encounters the same problem.

Answer 2

If wise installer is causing the issue then you could try running procmon to see what is causing the hang. Windows performance recorder is another useful tool as explained recently by Alois Kraus - http://geekswithblogs.net/akraus1/archive/2014/04/30/156156.aspx