boost test unit can not call mpi function

Question

I have looked throughfully around but could not find any reference to this problem.

I wrote a c++ program that I am testing with boost/unit. The serial version works fine and the unit test is working.
Now I have made the program parallel via a function doing embarrassingly parallel work with MPI. If a write down my own test calling the parallel function -- let's call it parafunction -- it is working well, MPI runs all right.
Compilation is done with mpic++ and I use mpixec to run the program.

If I call parafunction in boost test case however, the MPI goes all wrong, the test are launched multiple time and the process crash when several MPI::Init are called. Here is an example of the error I get :

The MPI_comm_size() function was called after MPI_FINALIZE was invoked.

This is disallowed by the MPI standard.

Your MPI job will now abort.

My test case is on a test_unit, automatically handled by a master_test_suite. As I said without the parallelisation it works perfectly well.

Parafunction calls MPI::Init and MPI::Finalize, and no other function of files is supposed to do any MPI related stuff.

Has anyone ever encountered a similar problem before ?

My test run are quite long therefore I could really use the parallel version of my program !

Thanks for your help

Answer 1

A function, which both initialises and then finalises can only be called once, because MPI can be only initialised once during the lifetime of the program and can only be finalised once. To prevent multiple initialisation calls, put the call to MPI_Init() or MPI_Init_thread() in a conditional:

int already_initialised;

MPI_Initialized(&already_initialised);
if (!already_initialised)
   MPI_Init(NULL, NULL);

As for the finalisation, it should be moved outside of your function, probably in an atexit(3) handler if you don't want to pollute the outer scope with MPI calls. For example:

void finalise_mpi(void)
{
   int already_finalised;

   MPI_Finalized(&already_finalised);
   if (!already_finalised)
      MPI_Finalize();
}

...
atexit(finalise_mpi);
...

The atexit() call could be part of the initialisation code, e.g.:

int already_initialised;

MPI_Initialized(&already_initialised);
if (!already_initialised)
{
   MPI_Init(NULL, NULL);
   atexit(finalise_mpi);
}

This would not install the atexit(3) handler if MPI was already initialised. The basic idea is that if MPI was initialised on entry to the function, then it would mean that MPI_Init() was called in the outer scope and one would normally expect that MPI_Finalize() is also called there.

If I were you, I would move MPI initialisation and finalisation out of the parallel processing function. The proper calling sequence would be to initialise MPI, run the tests, then finalise MPI.

I've used the C bindings in the above text as the C++ bindings were deprecated in MPI-2.2 and then deleted in MPI-3.0.