MPI Sending and Receiving Questions

Question

I have questions about the steps to send and receive MPI.

Suppose we have 2 MPI streams that are trying to send a message to each other. Below are three code snippets:

First (block "send" and "accept"):

... int data = ...; ... MPI_Send( &data, sizeof( int ), MPI_INT, (my_id == 0)?1:0, 0, MPI_COMM_WORLD ); MPI_Status status; MPI_Recv( &data, sizeof( int ), MPI_INT, (my_id == 0)?1:0, 0, MPI_COMM_WORLD, &status ); ... 

The second (non-blocking 'send', but blocking 'receive'):

 ... int data = ...; ... MPI_Request request; MPI_Isend( &data, sizeof( int ), MPI_INT, (my_id == 0)?1:0, 0, MPI_COMM_WORLD, &request); MPI_Status status; MPI_Recv( &data, sizeof( int ), MPI_INT, (my_id == 0)?1:0, 0, MPI_COMM_WORLD, &status ); // Synchronize sender & receiver MPI_Wait( &request, &status); ... 

Third (non-blocking “accept” with blocking “send”):

 ... int data = ...; ... MPI_Request request; MPI_Irecv( &data, sizeof( int ), MPI_INT, (my_id == 0)?1:0, 0, MPI_COMM_WORLD, &request ); MPI_Send( &data, sizeof( int ), MPI_INT, (my_id == 0)?1:0, 0, MPI_COMM_WORLD); MPI_Status status; // Synchronize sender & receiver MPI_Wait( &request, &status); ... 

I think there are potential problems with the three codes, but I want your opinion. So, I have the following questions:

• What are the (potential) problems (if any) with the 3 codes above?

• Which of the three codes above is valid / correct, given the MPI standard, so that it can work with all MPI implementations?

• What is the best way (if not one of 3, please write) to do this?

• In the third code, what if we change the order of the MPI_Irecv and MPI_Send calls?

PS: By the way, I tried to execute them using Scali MPI, and they all worked!

Answer 1

Your first implementation is likely to cause a dead end, especially if comminication is executed in synchronized mode (it may have worked in your tests because the connection was buffered, which is unlikely to be the case for big data).

The other two implementations should work without deadlock. I believe that it is better to start practicing receive operations before sending, so I would personally support the third implementation. From the MPI standard, section 3.7 :

Recommendations to users

[...]

The messaging model assumes that the message is initiated by the sender. Typically, a message will have lower overhead if reception has already been sent when the sender initiates the exchange of data (data can be moved directly to the receive buffer, and there is no need to queue a pending send request). However, the receive operation can only be completed after the corresponding dispatch has occurred. Using non-blocking receivers allows you to achieve lower communication costs without blocking the receiver while it is waiting to be sent.

A third implementation with the order MPI_Send / MPI_Irecv may be blocked in the MPI_Send call for the same reasons as the first implementation.