MPI sending messages or giving them as parameter

Question

I have an MPI implementation with one manager and multiple workers. The main function is in a file named run_mpi.cpp. It makes the initialisation of MPI and calls manager.cpp and worker.cpp. I want to have some variables in worker.cpp. I have two ways to do it:

1. I can give variables to worker.cpp as parameter when the main function calls it in run_mpi.cpp.

2. I can send variables in manager.cpp with MPI_Send or MPI_Isend.

In which conditions that I can use them? What are the benefits of them?

Answer 1

You need to treat the two as totally separate programs. MPI provides a communication layer between the instances (ranks) of your program, but there is no shared data between them. Let's say your master reads a file with a list of numbers, sends each of those numbers to a worker, and the worker does something with that number. You'd probably have something like this:

float localx;
float *x;
int rank;
int size;

MPI_Init(&argc,&argv);
MPI_Comm_rank(MPI_COMM_WORLD,rank);
MPI_Comm_size(MPI_COMM_WORLD,size);

x=new float[size];

if (rank==0)
{
  /* Typically, rank 0 is considered the master rank in a master/worker setup
     Open, read, and close file.
     Assume the data is read into array x, which is single precision floating point.
     For ease of illustration, I'm sending to one rank at a time, but this
     is better accomplished using MPI_Scatter. */
  localx=x[0];
  for (int i=1; i<size; i++)
  {
    MPI_Send(&x[i], 1, MPI_FLOAT, i, i, MPI_COMM_WORLD);
  }
  master(localx); // The master subroutine does whatever it needs to do
} else
{
  // If the rank isn't 0, then the rank is a worker
  MPI_Recv(&localx, 1, MPI_FLOAT, 0, rank, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
  worker(localx);
}

delete[] x;

The first three MPI calls will initialize the MPI communication environment (MPI_Init), get the rank number of the current instance (MPI_Comm_rank) and the size of the communicator (MPI_Comm_size). The size of MPI_COMM_WORLD is how many ranks you launched with mpirun. Next, x is allocated to hold one value for each worker.

Typically, if you're using a master/worker model, rank 0 is treated as the master rank. So, if the rank number is 0, we'll go ahead and read the file. As a side note, unless you're using MPI I/O or another parallel I/O, it's safer to only have one rank accessing a file. The master will read the file, which for my example should have one value per worker rank. As I stated in the comment, using MPI_Scatter is a better approach, but if you're new to MPI, we'll stick to the basics for now. First, I set localx to x[0]. Then, I call MPI_Send once for each rank, sending an element of x to each rank. Once the data is sent out, the master rank can call master(localx) and do whatever needs to be done there.

If the rank number is not 0, then the rank is a worker. The workers will not read the file. They'll instead call MPI_Recv to get the data being sent from the master. The worker ranks will simply sit and wait here until the data arrives. Once this is done, then the worker will call worker(localx) and do whatever in there.

One of the big hurdles I've seen people struggle with when learning MPI is that the memory of each rank is separate. Unlike threading, there is absolutely no shared memory (that your application can see, the MPI implementation will likely set something up for itself). Through the point of allocation, none of the ranks have any values set for localx or x. After the master reads it, only the master knows what x contains. Once the data is sent, each worker rank will have a single value from x in the master stored in localx.

I mentioned that using MPI_Scatter is better. Here's how that would be done. Everything before and after the conditional block will stay the same. The conditional block would become:

if (rank==0)
{
  // Read the file, fill x
}
MPI_Scatter(&x[0], size, MPI_FLOAT, &localx, 1, MPI_FLOAT, 0, MPI_COMM_WORLD);
if (rank==0)
{
  master(localx);
} else
{
  worker(localx);
}

The difference is that now, instead of sending and receiving once per worker rank, we scatter an array to variables in each rank all at once. Depending on what you're doing, it may not be possible to use MPI_Scatter. This example is simple enough that you can.