MPI combines several intercoms into one internal group

Question

I am trying to configure a bunch of spawned processes into one internal group. I need to deploy individual processes to unique working directories, as these subprocesses will write out a bunch of files. After all the processes are generated, they want to combine them into one internal communicator. To try this, I installed a simple test program.

int main(int argc, const char * argv[]) { int rank, size; const int num_children = 5; int error_codes; MPI_Init(&argc, (char ***)&argv); MPI_Comm parentcomm; MPI_Comm childcomm; MPI_Comm intracomm; MPI_Comm_get_parent(&parentcomm); if (parentcomm == MPI_COMM_NULL) { printf("Current Command %s\n", argv[0]); for (size_t i = 0; i < num_children; i++) { MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, 1, MPI_INFO_NULL, 0, MPI_COMM_WORLD, &childcomm, &error_codes); MPI_Intercomm_merge(childcomm, 0, &intracomm); MPI_Barrier(childcomm); } } else { MPI_Intercomm_merge(parentcomm, 1, &intracomm); MPI_Barrier(parentcomm); } printf("Test\n"); MPI_Barrier(intracomm); printf("Test2\n"); MPI_Comm_rank(intracomm, &rank); MPI_Comm_size(intracomm, &size); printf("Rank %d of %d\n", rank + 1, size); MPI_Barrier(intracomm); MPI_Finalize(); return 0; } 

When I run this, I get all 6 processes, but my inner intelligence speaks only between the parent and the last child spawned. Result obtained

 Test Test Test Test Test Test Test2 Rank 1 of 2 Test2 Rank 2 of 2 

Is there a way to combine several communicators into one communicator? Also note that I create these files at a time, since I need each subprocess to run in a unique working directory.

Answer 1

If you intend to do this by typing MPI_COMM_SPAWN several times, you will need to do this more carefully. After you call SPAWN for the first time, the spawned process must also take part in the next SPAWN call, otherwise it will be left outside the communicator that you are merging. it looks like this:

The problem is that only two processes are involved in each MPI_INTERCOMM_MERGE , and you cannot combine the three communicators so that in the end you will never end up with one big communicator.

If instead each process participates in a merge, it ends with one large communicator at the end:

Of course, you can just run all your additional processes at once, but it looks like you might have other reasons not to.

Answer 2

I understand that I have been out of date with this answer for a year, but I thought that maybe other people would want to see the implementation of this. As the initial respondent said, there is no way to combine three (or more) communicators. You must create a new internal unit at a time. Here is the code I'm using. This version removes the original intercom; you may or may not want to do this depending on your specific application:

 #include <mpi.h> // The Borg routine: given // (1) a (quiesced) intra-communicator with one or more members, and // (2) a (quiesced) inter-communicator with exactly two members, one // of which is rank zero of the intra-communicator, and // the other of which is an unrelated spawned rank, // return a new intra-communicator which is the union of both inputs. // // This is a collective operation. All ranks of the intra- // communicator, and the remote rank of the inter-communicator, must // call this routine. Ranks that are members of the intra-comm must // supply the proper value for the "intra" argument, and MPI_COMM_NULL // for the "inter" argument. The remote inter-comm rank must // supply MPI_COMM_NULL for the "intra" argument, and the proper value // for the "inter" argument. Rank zero (only) of the intra-comm must // supply proper values for both arguments. // // NB It would make a certain amount of sense to split this into // separate routines for the intra-communicator processes and the // remote inter-communicator process. The reason we don't do that is // that, despite the relatively few lines of code, what going on here // is really pretty complicated, and requires close coordination of the // participating processes. Putting all the code for all the processes // into this one routine makes it easier to be sure everything "lines up" // properly. MPI_Comm assimilateComm(MPI_Comm intra, MPI_Comm inter) { MPI_Comm peer = MPI_COMM_NULL; MPI_Comm newInterComm = MPI_COMM_NULL; MPI_Comm newIntraComm = MPI_COMM_NULL; // The spawned rank will be the "high" rank in the new intra-comm int high = (MPI_COMM_NULL == intra) ? 1 : 0; // If this is one of the (two) ranks in the inter-comm, // create a new intra-comm from the inter-comm if (MPI_COMM_NULL != inter) { MPI_Intercomm_merge(inter, high, &peer); } else { peer = MPI_COMM_NULL; } // Create a new inter-comm between the pre-existing intra-comm // (all of it, not only rank zero), and the remote (spawned) rank, // using the just-created intra-comm as the peer communicator. int tag = 12345; if (MPI_COMM_NULL != intra) { // This task is a member of the pre-existing intra-comm MPI_Intercomm_create(intra, 0, peer, 1, tag, &newInterComm); } else { // This is the remote (spawned) task MPI_Intercomm_create(MPI_COMM_SELF, 0, peer, 0, tag, &newInterComm); } // Now convert this inter-comm into an intra-comm MPI_Intercomm_merge(newInterComm, high, &newIntraComm); // Clean up the intermediaries if (MPI_COMM_NULL != peer) MPI_Comm_free(&peer); MPI_Comm_free(&newInterComm); // Delete the original intra-comm if (MPI_COMM_NULL != intra) MPI_Comm_free(&intra); // Return the new intra-comm return newIntraComm; }