mpi_waitall in mpich2 with null values in array_of_requests

Question

I get the following error with MPICH-2.1.5 and PGI compiler;

Fatal error in PMPI_Waitall: Invalid MPI_Request, error stack:
PMPI_Waitall(311): MPI_Waitall(count=4, req_array=0x2ca0ae0, status_array=0x2c8d220) failed
PMPI_Waitall(288): The supplied request in array element 0 was invalid (kind=0)

in the following example Fortran code for a stencil based algorithm,

Subroutine data_exchange

! data declaration

integer request(2*neighbor),status(MPI_STATUS_SIZE,2*neighbor)
integer n(neighbor),iflag(neighbor)
integer itag(neighbor),neigh(neighbor)

! Data initialization
request = 0; n = 0; iflag = 0;

! Create data buffers to send and recv
! Define values of n,iflag,itag,neigh based on boundary values

! Isend/Irecv look like this
ir=0
do i=1,neighbor
   if(iflag(i).eq.1) then
      ir=ir+1
      call MPI_Isend(buf_send(i),n(i),MPI_REAL,neigh(i),itag(i),MPI_COMM_WORLD,request(ir),ierr)
      ir=ir+1
      call MPI_Irecv(buf_recv(i),nsize,MPI_REAL,neigh(i),MPI_ANY_TAG,MPI_COMM_WORLD,request(ir),ierr)
   endif
enddo

! Calculations

call MPI_Waitall(2*neighbor,request,status,ierr)

end subroutine

The error occurs when the array_of_request in mpi_waitall gets a null value (request(i)=0). The null value in array_of_request comes up when the conditional iflag(i)=1 is not satisfied. The straight forward solution is to comment out the conditional but then that would introduce overheads of sending and receiving messages of 0 sizes which is not feasible for large scale systems (1000s of cores).

As per the MPI-forum link, the array_of_requests list may contain null or inactive handles.

I have tried following,

1. not initializing array_of_requests,
2. resizing array_of_request to match the MPI_isend + MPI_irecv count,
3. assigning dummy values to array_of_request
4. I also tested the very same code with MPICH-1 as wells as OpenMPI 1.4 and the code works without any issue.

Any insights would be really appreciated!

Answer 1

You could just move the first increment of ir into the conditional as well. Then you would have all handles in request(1:ir) at the and of the loop and issue:

call MPI_Waitall(ir,request(1:ir),status(:,1:ir),ierr)

This would make sure all requests are initialized properly.

Another thing: does n(i) in MPI_Isend hold the same value as nsize in the corresponding MPI_Irecv?

EDIT: After consulting the MPI Standard (3.0, Ch. 3.7.3) I think you need to initialize the request array to MPI_REQUEST_NULL if you want give the whole request array to MPI_Waitall.