How to compose all QtTestLib unit tests' results in a single file while using a single test project?

Question

In our project we are using the QtTestLib for a unit testing. The reasons are that the whole project already uses Qt whenever it's possible and it's a GUI application, so we wanted to have ability for testing GUI interfaces.

Our project is compiled by the MSVC, so we didn't want to have a separated project file for each test 'coz it will clutter the solution. Thus we have created a single project for all tests. All testing should be automated on a CIS (continuous integration), so we tried to plug our tests to the Hudson through an output file in XML format using some XSLT transformations.

But it seems there is a problem with tests' output. If you use a single main() for all tests and merely transmit cmd line arguments to each test:

#include "MyFirstTest.h"
#include "MySecondTest.h"

int main(int argc, char **argv)
{
  int result = 0;
  MyFirstTest test1;
  result |= QTest::qExec(&test1, argc, argv);
  MySecondTest test2;
  result |= QTest::qExec(&test2, argc, argv);
  return result;
}

then you will get a result file rewrited multiple times. So if you want to automate it somewhat using output file (xml for example), you'll get only the last result in it. All other will be overwritten.

We have tried that approach already, it does not give you ability to use some continuous integration systems like Hudson. So my question will be: is there any opportunities to append results in one output file? Of course we can use some workarounds like running each test by QTest::qExec() with a modified parameters to write results in separate files, but it doesn't seem to be the best way. Ideally I want to have a single result file to use it with CIS.

Answer 1

With this trick you can collect the individual test xml reports to temporary buffers/files; all from a single test binary. Lets employ QProcess to collect separate test outputs from within one binary; the test calls itself with modified arguments. First, we introduce a special command-line argument that harnesses the subtests proper - all still within your test executable. For our convenience, we use the overloaded qExec function that accepts a QStringList. Then we can insert/remove our "-subtest" argument more easily.

// Source code of "Test"

int
main( int argc, char** argv )
{
  int result = 0;

  // The trick is to remove that argument before qExec can see it; As qExec could be
  // picky about an unknown argument, we have to filter the helper 
  // argument (below called -subtest) from argc/argc; 

  QStringList args;

  for( int i=0; i < argc; i++ )
  {
     args << argv[i];
  }

  // Only call tests when -subtest argument is given; that will usually
  // only happen through callSubtestAndStoreStdout

  // find and filter our -subtest argument

  size_t pos = args.indexOf( "-subtest" );

  QString subtestName;

  if( (-1 != pos) && (pos + 1 < args.length()) )
  {
    subtestName = args.at( pos+1 );

    // remove our special arg, as qExec likely confuses them with test methods

    args.removeAt( pos );
    args.removeAt( pos );

    if( subtestName == "test1" )
    {
      MyFirstTest test1;
      result |= QTest::qExec(&test1, args);
    }

    if( subtestName == "test2" )
    {
      MySecondTest test2;
      result |= QTest::qExec(&test2, args);
    }

    return result;
}

Then, in your script/commandline call:

./Test -subtest test1 -xml ... >test1.xml
./Test -subtest test2 -xml ... >test2.xml

and here you are - we have the means to separate the tests output. Now we can continue to use QProcess'es ability to collect stdout for you. Just append these lines to your main. The idea is to call our executable again, if no explicit tests are requested, but with our special argument:

bool
callSubtestAndStoreStdout(const String& subtestId, const String& fileNameTestXml, QStringList args)
{
   QProcess proc;

   args.pop_front();

   args.push_front( subtestId );
   args.push_front( "-subtest" );

   proc.setStandardOutputFile( fileNameTestXml );

   proc.start( "./Test", args );

   return proc.waitForFinished( 30000 ); // int msecs
}

int 
main( int argc, char** argv )
{
   .. copy code from main in box above..

   callSubtestAndStoreStdout("test1", "test1.xml", args);
   callSubtestAndStoreStdout("test2", "test2.xml", args);

   // ie. insert your code here to join the xml files to a single report

   return result;
}

Then in your script/commandline call:

./Test -xml           # will generate test1.xml, test2.xml

Indeed, hopefully future QTestLib versions makes this easier to do.

Answer 2

I've used this dirty workaround (works with Jenkins):

int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);
    int result = 0;
    freopen("MyAppTests_Test1.xml", "w", stdout);
    result |= QTest::qExec(new Test1, argc, argv);
    freopen("MyAppTests_Test2.xml",  "w", stdout);
    result |= QTest::qExec(new Test2, argc, argv);
    return result;
}

Then in Jenkins I've added build action "execute shell": ./path_to_MyAppTests -xml

and added Post-build Actions "publish xUnit test result report" (QTestlib). QTestlib Pattern: MyAppTests*.xml

Answer 3

As I cannot comment here yet, I will post it here in addtition to muenalan's answer. There're few fixes that have to be applied to for it to work (at least with Qt5):

1. callSubtestAndStoreStdout has 3 bugs. First, the first arg has to be popped from front (this is the arg 0) before pushing new ones. Second, you have to redirect the output before starting the process. Third, it has to return some value ;)

   QProcess proc;
   args.pop_front();
   args.push_front(subtestId);
   args.push_front("-subtest");
   
   proc.setStandardOutputFile(fileNameTestXml);
   proc.start("sportSystemTest.exe", args);
   return proc.waitForFinished(30000);
   

2. main also has some (obvious) mistakes. The main one is in the if statement:

   if ((-1 != pos) && (pos + 1 < args.length()))
   

as the original one would never fire.

Anyway, thanks for the solution, it solved my big headache :)

Answer 4

In my opinion, trying to build a single executable is a bad idea here: If one of your tests crashes, the others won't execute anymore ...

Another way to run a suite with multiple testcases:

• Create a subdirs project on the toplevel.
• add a subfolder with an own .pro for each testcase, and add this to the subdirs project.
• build the project from the toplevel folder
• run make check on the toplevel makefile. This will call all your testcases. You can also pass parameters, e.g. use nmake -k check TESTARGS="-o result.xml,xml -v2 -maxwarnings 0" with your MSVC environment. The -k switch helps to continue if one test fails.
• The xunit Jenkins plugin as an example allows a pattern like my_build\*\result.xml to search for your xml files, and this way you can parse all generated files without merging to a single file.