MPI_Comm_rank always writes 0

Question

How can I get the expected output

rank 0
size 2
rank 1
size 2

or some permutation of those lines?

ranktest.c

#include <mpi.h>
#include <stdio.h>
int main(int argc, char *argv[]){
    MPI_Init(NULL, NULL);
    int world_rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
    int world_size;
    MPI_Comm_size(MPI_COMM_WORLD, &world_size);
    printf("rank %d\n", world_rank);
    printf("size %d\n", world_size);
    MPI_Finalize();
    return 0;
}

compilation and running

tsbertalan@hustlenbustle:~$ mpicc ranktest.c
tsbertalan@hustlenbustle:~$ mpirun -np 2 ./a.out 
rank 0
size 1
rank 0
size 1

On a different host:

tsbertalan@stamp:~$ mpicc ranktest.c
tsbertalan@stamp:~$ mpirun -np 2 ./a.out 
rank 0
size 2
rank 1
size 2

I've tried

tsbertalan@hustlenbustle:~$ sudo aptitude reinstall openmpi-bin libopenmpi-dev

but nothing changed. Both /etc/openmpi/openmpi-default-hostfile and /etc/openmpi/openmpi-mca-params.conf contain nothing but comments on both hosts. What could possibly be different here?

Changing to MPI_Init(&argc, &argv), or alternately, int main() also does nothing.

The real problem, thanks to user3469194:

linuxmint@linuxmint ~ $ sudo aptitude remove libopenmpi-dev mpich2
linuxmint@linuxmint ~ $ sudo aptitude install libmpich2-dev openmpi-bin
linuxmint@linuxmint ~ $ mpicc ranktest.c
linuxmint@linuxmint ~ $ mpirun -np 2 ./a.out
rank 0
size 1
rank 0
size 1
linuxmint@linuxmint ~ $ sudo aptitude remove libmpich2-dev openmpi-bin
linuxmint@linuxmint ~ $ sudo aptitude install libopenmpi-dev mpich2
linuxmint@linuxmint ~ $ mpicc ranktest.c
linuxmint@linuxmint ~ $ mpirun -np 2 ./a.out
[linuxmint:16539] [[INVALID],INVALID] ORTE_ERROR_LOG: A system-required executable either could not be found or was not executable by this user in file ../../../../../../orte/mca/ess/singleton/ess_singleton_module.c at line 357

(plus much more)

Responses to some suggestions: (See this github repo, commits for 1 December, 2012.)

Try moving the definitions of world_rank and world_size before MPI_Init(), does it change anything?

Naturally, this doesn't work so well:

tsbertalan@perrin:~/svn/524hw4$ git checkout 7b5e229 ranktest.c
(reverse-i-search)`clean': tsbertalan@hustlenbustle:~/Documents/School/12fall2012/524apc524/hw/hw4$ make ^Cean && make ranktest && mpirun -np 2 ranktest
tsbertalan@perrin:~/svn/524hw4$ make clean && make ranktest && mpirun -np 2 ranktest
rm -f heat_serial heat_omp heat_mpi heat_serial_O* ranktest
mpicc ranktest.c -o ranktest
*** An error occurred in MPI_Comm_rank
*** before MPI was initialized
*** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
[perrin:15206] Abort before MPI_INIT completed successfully; not able to guarantee that all other processes were killed!
*** An error occurred in MPI_Comm_rank
*** before MPI was initialized
*** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
[perrin:15207] Abort before MPI_INIT completed successfully; not able to guarantee that all other processes were killed!
tsbertalan@perrin:~/svn/524hw4$ git checkout HEAD ranktest.c

Or, on my home computer:

tsbertalan@hustlenbustle:~/Documents/School/12fall2012/524apc524/hw/hw4$ git checkout 7b5e229 ranktest.c
tsbertalan@hustlenbustle:~/Documents/School/12fall2012/524apc524/hw/hw4$ vim ranktest.c 
tsbertalan@hustlenbustle:~/Documents/School/12fall2012/524apc524/hw/hw4$ make clean && make ranktest && mpirun -np 2 ranktest
rm -f heat_serial heat_omp heat_mpi heat_serial_O* ranktest
mpicc ranktest.c -o ranktest
Attempting to use an MPI routine before initializing MPICH
Attempting to use an MPI routine before initializing MPICH
tsbertalan@hustlenbustle:~/Documents/School/12fall2012/524apc524/hw/hw4$ git checkout HEAD ranktest.c

This is almost always an issue of running a program compiled with one MPI with another's mpirun. Does the first machine (hustlenbustle) also have mpich2 installed? Where do things appear on the path? In particular, what is the result of which mpicc and which mpirun?

I'm recompiling before each attempt, on each computer. I went ahead and made a make target for this. However, as per request:

tsbertalan@hustlenbustle:~$ which mpicc
/usr/bin/mpicc
tsbertalan@hustlenbustle:~$ which mpirun
/usr/bin/mpirun

tsbertalan@perrin:~/svn/524hw4$ which mpicc
/usr/bin/mpicc
tsbertalan@perrin:~/svn/524hw4$ which mpirun
/usr/bin/mpirun

And, for shiggles, here's the output of some aptitude searches for hnb and perrin. Let me know if I should search for something else.

Under Open MPI the following command should print out the version: mpirun -V. If it doesn't print mpiexec (OpenRTE) 1.x.x. then you might have a mismatch of the run-times.

tsbertalan@hustlenbustle:~$ mpirun -V
mpirun (Open MPI) 1.4.3

tsbertalan@perrin:~/svn/524hw4$ mpirun -V
mpirun (Open MPI) 1.4.1

However, I am recompiling for each test.

Perhaps sudo aptitude reinstall SOMETHING might help?

Answer 1

Found this code in a version of mpic.c I had on my computer (that was causing me trouble when I installed a new package that brought mpi with it). It seems your computer has something like this on it, and the other host has the correct version.

int MPI_Comm_rank( MPI_Comm comm, int *rank)
{
  *rank=0;
  return 0;
}

As you can see, rank is always set to 0 (and the similar size function is probably setting the variable to 1).

Answer 2

I had this problem too. The problem was that mpicc was OpenMPI (to see this, just run mpicc -v), whilst mpirun was MPICH2 (mpirun -V). I simply solved it by uninstalling MPICH2.