Mpi4py freezes when trying to send big data

Question

Recently, I encountered the problem of trying to exchange big data between several processors using the "Send" command from the mpi4py library. Even a 1000x3 floating point array is too large to send. Any ideas how to overcome this problem?

thanks in advance.

Answer 1

I found a simple solution. Divide the data into small enough pieces ...

Answer 2

Isend ( Send). , , , . , comm.barrier() .

Answer 3

Sending / output of big data: point-to-point:

#!/usr/bin/env python
from mpi4py import MPI
import numpy

comm = MPI.COMM_WORLD
rank = comm.Get_rank()

if rank == 0:
    data = numpy.arange(300*100000, dtype='f')
    comm.Send([data, MPI.FLOAT], dest=1, tag=77)
elif rank == 1:
    data = numpy.empty(300*100000, dtype='f')
    comm.Recv([data, MPI.FLOAT], source=0, tag=77)
    print data

Running this with two processors:

% ~/work/soft/mpich/bin/mpiexec -np 2 ./send-numpy.py
[  0.00000000e+00   1.00000000e+00   2.00000000e+00 ...,   2.99999960e+07
   2.99999980e+07   3.00000000e+07]