OpenMPI node & network topology

Question

i am currently building a small utility library as part of a larger project. OpenMPI has a well documented API library but i am a little puzzled when it comes to the lower level communication between nodes. I know that when writing your algorithm, you distribute it across all nodes that are expected to do the computation each communicating with the rest, executing portion of the algorithm based on their "global" MPI rank (as defined in the algorithm), and all the nodes synchronize back and forth. However, the reason i placed global in quotes is, does openMPI communicate on the ip level, hence say that i already have a long executing algorithm, but there is a node that is idling, not running any MPI processes, if i execute my MPI algorithm on it, will it join the MPI_COMM_WORLD and become part of the overall network topology or is there some "voodoo" i need to do to make that node part of the MPI_COMM_WORLD. also, if nodes can become part of MPI_COMM_WORLD for that particular algorithm, how can i register/identify this new node?

any reading references are also helpful.

many thanks!

tl;dr are MPI nodes hot-swappable from the MPI_COMM_WORLD?

Answer 1

You cannot join nodes to a communicator once it has been created. This is also true for MPI_COMM_WORLD which is just a precreated communicator. Only the processes that are launched as part of the initial SPMD launch become part of MPI_COMM_WORLD. But you can spawn additional processes using the MPI-2 process management facilities, described in Chapter 10 of the current MPI Standard version 2.2.

The newly spawned processes have their own MPI_COMM_WORLD and the processes of the initial group can only communicate with them through a special intercommunicator (MPI_COMM_WORLD is intracommunicator) that is returned by the spawn operation. Since process spawning is a collective operation, all processes in MPI_COMM_WORLD will get a handle to that intercommuniator.

MPI 3.0 (when it's out sooner or later) will bring fall-through fault tolerance that will allow to exclude/remove faulty processes from MPI_COMM_WORLD (or from any other communicator) and MPI 3.1 will eventually bring something that will allow faulty processes to be replaced.