Spawn issue with mpi4py in the Anaconda Python distribution

Question

mpi4py.MPI.COMM_SELF.Spawn seems to throw errors when using Anaconda's Python distribution. My question is whether it is possible that mpi4py is built against the wrong version of MPI (mpich v openmpi) in my installation of Anaconda in some way.

The issue is that the following code

# spawn.py
import mpi4py.MPI as MPI
import sys
print 'mpi4py:',MPI.__file__
comm = MPI.COMM_SELF.Spawn(sys.executable,
                           args=['hello.py'],
                           maxprocs=1)

gives the following error

$ /home/wd15/anaconda/bin/python spawn.py
mpi4py: /home/wd15/anaconda/lib/python2.7/site-packages/mpi4py/MPI.so
[mpiexec@loki] match_arg (./utils/args/args.c:122): unrecognized argument pmi_args
[mpiexec@loki] HYDU_parse_array (./utils/args/args.c:140): argument matching returned error
[mpiexec@loki] parse_args (./ui/mpich/utils.c:1387): error parsing input array
[mpiexec@loki] HYD_uii_mpx_get_parameters (./ui/mpich/utils.c:1438): unable to parse user arguments
...

I believe that Anaconda's mpi4py is using mpich. If I use the system python it works.

$ /usr/bin/python spawn.py
mpi4py: /usr/lib/python2.7/dist-packages/mpi4py/MPI.so
hello

The system mpi4py uses openmpi. Could there be some issue with mpi4py being built with conflicting MPI versions in Anaconda?

Also, the mpi4py package seems to function correctly in Anaconda when running under an mpirun call. For example

# test_mpi4py.py
import mpi4py.MPI as MPI
print MPI.COMM_WORLD.Get_rank()

will give

$ mpirun -n 2 /home/wd15/anaconda/bin/python test_mpi4py.py
0
1

Answer 1

I ran into the same problem and one solution was compiling mpi4py with openmpi instead of mpich (see 'Compute Pi' example in the mpi4py documentation).

See this unresolved issue.

Tested on: Ubuntu 16.04 Anaconda 4.0.0 python 3.5.0 mpich 3.2.0 openmpi 1.10.2 mpi4py 2.0.0

Answer 2

Anaconda is bundled with MPICH. Rather than using anaconda's mpi4py, you will need to build your own installation from source, explicitly linking against the system MPI installation.