Test Reporting

Question

We are migrating our test report data (unit, regression, integration, etc..) from an XML format to a database format for better analysis. Right now the majority of our test analysis is done using the CruiseControl.NET dashboard, but this is limited to primarily the most recent test data. Older test data can be accessed but not easily compared to new test data. We want to pin point problem components and better narrow down bugs. With the onset of tons of information brought on by our newly implemented regression and integration testing I would like to see some better metrics generated (possibly performance and the like). Have you worked with any business intelligence systems that will provide a framework for accurately and easily implementing some sort of analysis and reporting?

I have looked into JasperReports and Pentaho but I'm struggling with implemetation of Pentaho at the moment. Should I continue my fight with the system? Is this what I'm looking for?

Answer 1

You could always just use SQL Server Reporting Services and Report Builder (MS's web based designer) or Report Designer (component of Visual studio). It's pretty easy to get this set up too.

• Report Builder: http://msdn.microsoft.com/en-us/library/ms155933.aspx
• Report Designer: http://msdn.microsoft.com/en-us/library/ms157166.aspx
• Tutorial: http://www.simple-talk.com/sql/learn-sql-server/beginning-sql-server-2005-reporting-services-part-1/
• How to add Reporting Services to an existing SQL Server: http://www.mssqltips.com/tip.asp?tip=1444

There are a few end user reporting solutions around as well that make it easier to dynamically create reports, if you're willing to invest a bit of cash.

My company produce one: http://www.rsinteract.com has a very cheap standard edition with a limited number of reports (30 day free trial). It reports directly off SQL server with Reporting Services installed. It won best of TechEd 2006 - http://windowsitpro.com/article/articleid/53944/best-of-tech-ed-2006-winners.html

We actually use ours to analyse the support requests from clients i.e. which component is failing most, who reports the most bugs etc. Not tried it on test data.

There's also Proclarity, ApexSQL Report, and Tableau all of which are good.

Answer 2

You could try looking at rolling your own (if you know what you're looking for) using Processing written by Ben Fry. It's best accompanied by his book "Visualizing Data".

The tool is free and I guess you can get a free 45 day trial of O'Reilly Books Online to get a head start and see if its right for you. I do know there are chapters on reading and crunching data from all kinds of sources (including XML and databases) and then making meaningful and useful visualisations from them.

I'm currently using it to get my head round the dependency complexities of an inherited code base and its been massively useful.

Answer 3

Which part of Pentaho?

The Kettle project has stuff to convert your Cruise Control info and load it into a relational database. That's probably a good module to get working properly, especially if you're almost done figuring it out. I hope you'll share this stuff. I could use it too.

The Platform will autoschedule stuff once Kettle has it loading.

To make Mondrian really useful you'll need to work out a fact / dimension organization to your test data. That may or may not be worth your trouble at this point.

Once you have your data loaded you'll probably be able to get a lot of benefit out of simple SQL queries like this...

 select * 
  from test 
  where failed='yes' 
  order by testno, date desc

and this...

 select max(date), min(date), testno 
  from test
  where failed='yes'
  group by testno 
  order by testno 

and stuff like that. You might consider creating views in your table server for your favorite queries.

There are myriad ways to convert your sql queries into reports, including the pentaho reporting module, BIRT (an eclipse plugin), Crystal Reports, and all kinds of PHP or JSP stuff you could put together.