Configure ipython / ipyparallel MPI distributed cluster

Question

I am trying to understand how to configure a distributed MPI cluster using ipython / ipyparallel. I do not have a strong MPI background.

I followed the following instructions in ipyparallel docs (using ipcluster in mpiexec / mpirun mode) and this is great for distributing computations on a single node machine. Therefore, we create a profile mpiby configuring it in accordance with the above instructions and starting the cluster

$ ipython profile create --parallel --profile=mpi
$ vim ~/.ipython/profile_mpi/ipcluster_config.py

Then on host A, I start the controller and 4 MPI mechanisms:

$ ipcontroller --ip='*' --profile=mpi    
$ ipcluster engines --n=4 --profile=mpi

Running the following snippet:

from ipyparallel import Client
from mpi4py import MPI

c = Client(profile='mpi')
view = c[:]

print("Client MPI.COMM_WORLD.Get_size()=%s" % MPI.COMM_WORLD.Get_size())
print("Client engine ids %s" % c.ids)

def _get_rank():
    from mpi4py import MPI
    return MPI.COMM_WORLD.Get_rank()

def _get_size():
    from mpi4py import MPI
    return MPI.COMM_WORLD.Get_size()

print("Remote COMM_WORLD ranks %s" % view.apply_sync(_get_rank))
print("Remote COMM_WORLD size %s" % view.apply_sync(_get_size))

gives

Client MPI.COMM_WORLD.Get_size()=1
Client engine ids [0, 1, 2, 3]
Remote COMM_WORLD ranks [1, 0, 2, 3]
Remote COMM_WORLD size [4, 4, 4, 4]

Then, on host B, I launch 4 MPI engines. I run the fragment again, which gives

Client MPI.COMM_WORLD.Get_size()=1
Client engine ids [0, 1, 2, 3, 4, 5, 6, 7]
Remote COMM_WORLD ranks [1, 0, 2, 3, 2, 3, 0, 1]
Remote COMM_WORLD size [4, 4, 4, 4, 4, 4, 4, 4]

, ipcluster 4, , . MPI.

:

• ipython/ipyparallel, , MPI . ipyparallel MPI, , , MPI-, IPython MPI Machine? , , ipyparallel , , , .
• - , MPI ipyparallel? googled , .
• , ipython/ipyparallel MPI, ?

• , , , MPI . , :

  • OpenMPI .

  • MPI - / node , MPI_Comm_spawn. , MPI_Comm_spawn

    MPI_Comm_spawn maxprocs MPI, , . . MPI_COMM_WORLD, .

    grep ipyparallel code , .

• , , MPI , .

  , , , Torque/SLURM .. , , , , mpiexec, , .

• : , ipyparallel MPI, ipyparall MPI.

Answer 1

MPI IPython :

mpiexec [-n N] ipengine

MPI. , MPI, , . , , - , , , mpiexec, IPython .

IPython , MPI . :

# ~/mpi_hosts
machine1 slots=4
machine2 slots=4

script:

# test_mpi.py
import os
import socket
from mpi4py import MPI

MPI = MPI.COMM_WORLD

print("{host}[{pid}]: {rank}/{size}".format(
    host=socket.gethostname(),
    pid=os.getpid(),
    rank=MPI.rank,
    size=MPI.size,
))

:

$ mpiexec -machinefile ~/mpi_hosts -n 8 python test_mpi.py 
machine1[32292]: 0/8
machine1[32293]: 1/8
machine1[32294]: 2/8
machine1[32295]: 3/8
machine2[32296]: 4/8
machine2[32297]: 5/8
machine2[32298]: 6/8
machine2[32299]: 7/8

,

c.MPILauncher.mpi_args = ["-machinefile", "~/mpi_hosts"]

~/.ipython/profile_default/ipcluster_config.py

ipcluster start -n 8