fatal error in pmpi_gather

Question

So I have a raytracer I'm writing which compiles just fine, but when I get to the MPI_Gather() function I get this error set. If I write to files the whole thing finishes fine, but then I can't run it on a distributed computing system.

Fatal error in PMPI_Gather: Internal MPI error!, error stack:
PMPI_Gather(856)......:
   MPI_Gather(sbuf=0x8e05468, scount=882000, MPI_BYTE, rbuf=0x8df7628, rcount=882000, MPI_BYTE, root=0, MPI_COMM_WORLD) failed
MPIR_Gather_impl(681).: 
MPIR_Gather(641)......: 
MPIR_Gather_intra(152): 
MPIR_Localcopy(378)...:
   memcpy arguments alias each other, dst=0x8df7628 src=0x8e05468 len=882000

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   EXIT CODE: 1
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================

I'm not exactly sure what this error means, so it's hard to get around it.

Here is the source for the main function:

int main(int argc, char **argv) {
  clock_t total_time = clock(), otime;
  init_MPI(argc, argv);   //Initialize OpenMPI
  glutInit(&argc,argv);

  get_params(argc, argv);   //Get parameters from command line
  if (buildScene(scene, cam) == -1) MPI_Abort(MPI_COMM_WORLD,rc); exit(1);
  samples = samples > 0 ? whitted ? 1 : samples : 1;
  if (numprocs == 1) {
    scn = new RGBApixmap(h,w);
    raytrace(h,scn);
    if (smult > 1) *scn = scaleImage(scn,smult);
  } else {
    int rows = h / numprocs;
    subscn = new RGBApixmap(rows,w);
    raytrace(rows, subscn);
    if (smult > 1) *subscn = scaleImage(subscn,smult);

    if (pid == MASTER) scn = new RGBApixmap(h/smult,w/smult);
    MPI_Gather(subscn,rows/smult*w,MPI_BYTE,scn,rows/smult*w,MPI_BYTE,MASTER,MPI_COMM_WORLD);
  }
  if (pid == MASTER) {
    initGlut(argc, argv);
    glutMainLoop();
  }
  MPI_Finalize();
  return 0;
}

EDIT:

I've fixed the problem and posted updated code below:

int main(int argc, char **argv) {
  clock_t total_time = clock(), otime;
  init_MPI(argc, argv);
  glutInit(&argc,argv);    
  bool OK = get_params(argc, argv);
  if (buildScene(scene, cam) == -1) { MPI_Abort(MPI_COMM_WORLD,rc); exit(1); }
  samples = samples > 0 ? whitted ? 1 : samples : 1;
  int rows = h / numprocs;
  subscn = new RGBApixmap(rows,w);
  raytrace(rows, subscn);
  MPI_Barrier(MPI_COMM_WORLD);          /* Synchronize all processes */   
  if (smult > 1) *subscn = scaleImage(subscn,smult); 
  MPI_Barrier(MPI_COMM_WORLD);          /* Synchronize all processes */

  int nElts = subscn->getWidth()*subscn->getHeight();
  RGBA *subscnpix, *scnpix;
  subscnpix = subscn->getPixs();
  scnpix = (RGBA*)malloc(sizeof(RGBA)*((w/smult)*(h/smult)));

  MPI_Datatype pixel;
  MPI_Type_contiguous(4,MPI_UNSIGNED_CHAR,&pixel);
  MPI_Type_commit(&pixel);

  MPI_Gather(subscnpix,nElts,pixel,scnpix,nElts,pixel,MASTER,MPI_COMM_WORLD);

  scn = new RGBApixmap(h/smult,w/smult,scnpix);

  MPI_Type_free(&pixel);

  MPI_Barrier(MPI_COMM_WORLD);          /* Synchonize all processes */
  if (pid == MASTER) {
    initGlut(argc, argv);
    glutMainLoop();
  }
  MPI_Finalize();
  return 0;
}

Answer 1

Your send and receive buffers in the call to MPI_Gather in rank MASTER overlap which is against the restriction, imposed by the MPI standard. subscn is at 0x8e05468 while scn is at 0x8df7628. The memory span from 0x8df7628 to 0x8e05468 is only 56896 bytes and you are trying to write to it 882000 times numprocs bytes, which won't work.

This is due to the fact that you divide the number of rows by the scaling factor but still forget to divide the image width w by the scaling factor.

MPI_Gather(subscn, rows/smult*w/smult, MPI_BYTE,
           scn, rows/smult*w/smult, MPI_BYTE,
           MASTER, MPI_COMM_WORLD);

Note that if the pixmap is not greyscale, you'd also have to multiply the number of data elements by the number of colour components (possibly counting an alpha channel too) or create a contiguous derived datatype instead.