How can I specify a large number of files for an NCOVER merge operation?

Question

I have a large acceptance test suite that runs the source application many thousands of times, creating an NCOVER report for each run. After each test, it merges the generated code coverage report into a large "master" coverage report for the whole application.

My worry here is that I'm running into a Shlemiel the Painter problem, as the merge operation is parsing the large coverage file for every test.

I can pass more than one coverage file to NCover.Reporting.Exe, which actually does the merge, but when I try to pass them all at once I'm running into operating system limits on command line length.

Does NCover.Reporting provide some sort of input coverage report listing that I can save to a file, and have it merge all the reports in one go?

Answer 1

Well, what I settled on was kind of like Stephen Ward's answer. I put all the coverage files into a directory and ran NCover.Reporting.exe Path\To\My\Directory\*.xml //s Merged.nccov.

He's not getting the checkmark though because his answer telling me I could not use a wildcard on the commandline is incorrect; in fact, the wildcard is the only way NCover would recognise the XML files in that directory. I think he's used to the Unix world (considering there was some Unix FUD in his answer), where the shell takes responsibility for handling wild cards. The windows shell doesn't attempt to expand wildcards, so we didn't run into length limits with that.

With this solution though, I was able to make our runs about 20% faster than the old merge-after-each-test method, which is good. (Coverage runs are still four times as long as non coverage runs, but we don't need to run coverage for every test)

When I tried commondream's answer, unfortuately NCover would crash on a regular basis. I don't think it likes that the test system starts .NET 1, 1.1, 2.0, 3.0, 3.5, and 4.0 assemblies , as well as things like silverlight packages and the compact framework (Our product needs to be able to work correctly on all these kinds of assemblies). If you don't need to be insane with running that sort of thing, then the //pn option he describes would probably work fine.

When I tried joe.feser's answer, unfortunately I was unable to figure out the format of NCover's reporting settings file. There's ample documentation on the XML the reporting tool accepts; with the exception of the root XML node it looks for. It refused recognize any of the XML files I generated because the root node was named incorrectly; I assume it was validating against some form of DTD of which I was not aware. If someone would like to post an example file, please go ahead.

Hope this helps anyone with similar issues in the future :)

Answer 2

I would merge 25-50 at a time and at the end, take those and merge them together.

May I ask why you are running each test on it's own. Are you using the include feature to only instrument the code you are running or are all of your coverage files very large?

You could also email support, they may have a better option.

Answer 3

@joe the trouble is you have to generate the config file. in theory he could do that by writing a little helper program that adds all of the files in a DIRECTORY to the config file, but you'd have to use the directory name without a file filter i.e. do

myhelper.exe coveragedirectory

NOT

myhelper.exe coveragedirectory*.xml

command line length limits are a "feature" of DOS 1.0 and hence of the emulator built into windows. Have you tried using powershell?

Answer 4

We just updated NCover to handle multiple coverage runs under NCover.Console much more efficiently. Assuming you are running the same application each time I would make sure that you're updated to NCover 3.4.12 and run NCover.Console with the //coverall option on the script that you're using to spawn your application rather than NCover each time you start your app. That will generate one merged XML file that should handle things for you.

Feel free to drop an email to support@ncover.com and we can help you out with it, if needed. Be sure to mention this url.