So, this is a little more subtle and requires some understanding of how the Gather team puts complex types.
If you look at the MPI_Gather examples , they consist of a 1st array, and it's pretty easy to figure out what is going to happen; you get (say) 10 ints from each process, and Gather is smart enough to put 10 ints from rank 0 at the beginning, 10 from rank 1 at position 10-19 in the array, etc.
More complex layouts like this one are a bit more complicated. First, the location of the data from the point of view of the sender is different from the location of the data from the receivers. At the sender point, you start with the element of the array [1][2] , go to [1][5] (in an array of size 7x7), then go to the elements of the array [2][3] - [2][5] and so on. .d. There are CHUNK_ROWS data blocks, each of which is divided into two lines.
Now consider how the receiver should receive them. Let's say he gets rank 0 data. He gets it into the elements of the array [0][0]-[0][4] - so far so good; but then it will receive the next block of data in [1][0]-[1][4] in a 10x10 array. This is a jump through 5 elements. The layout in the memory is different. Thus, the recipient will have to access another type of Subarray , after which the senders send it because the memory layout is different.
So, although you can send something that looks like this:
sizes[0] = CHUNK_ROWS+2; sizes[1] = CHUNK_COLUMNS+2; subsizes[0] = CHUNK_ROWS; subsizes[1] = CHUNK_COLUMNS; starts[0] = 1; starts[1] = 1; MPI_Type_create_subarray(ndims,sizes,subsizes,starts,MPI_ORDER_C,MPI_INT,&sendsubarray); MPI_Type_commit(&sendsubarray);
you will get what looks like this:
sizes[0] = ROWS; sizes[1] = COLUMNS; subsizes[0] = CHUNK_ROWS; subsizes[1] = CHUNK_COLUMNS; starts[0] = 0; starts[1] = 0; MPI_Type_create_subarray(ndims,sizes,subsizes,starts,MPI_ORDER_C,MPI_INT,&recvsubarray); MPI_Type_commit(&recvsubarray);
It is relatively important to notice the difference in the sizes array.
Now we are getting closer. Note that your MPI_Gather line changes something like this:
MPI_Gather(DEBUG_CH[0],1,sendsubarray,recvptr,1,recvsubarray,0,MPI_COMM_WORLD);
There were several things that did not work on the previous version, MPI_Gather(DEBUG_CH[0],1,subarray,ddd[0],CHUNK_ROWS*CHUNK_COLUMNS,MPI_INT,0,MPI_COMM_WORLD); - firstly, note that you are referencing ddd[0] , but for each rank except rank 0, ddd=NULL , and therefore this will fail. So, create a new variable say recvptr , and in rank zero set it to ddd[0] . (It doesn’t matter where other processes consider this, because they don’t receive.) In addition, I think you do not want to receive CHUNK_ROWS*CHUNK_COLUMS MPI_INTs , because this will shift them in memory, and I understand that you want them to be laid in the same way as in subordinate tasks, but in a larger array.
So, now we get somewhere, but the above still fails for an interesting reason. For examples from the 1st array, it is enough to simply find out where the data of the nth rank go. The way it is calculated is to find the amount of data received and start the next element immediately afterwards. But that will not work. "Only after" the end of the zero data rank is not where the data of rank one should begin ( [0][5] ), but instead [4][5] is the element after the last element in the basement of rank 0. Here is the data that you get from different ranks, overlap! Therefore, we will have to bother with extents of data types and manually specify where the ranking data begins. The second is the easy part; you use the MPI_Gatherv function when you need to manually specify the amount of data from each processor or where it goes. The first is the more difficult part.
MPI allows you to specify the lower and upper boundaries of a given data type - where, given a piece of memory, the first bit of data for this type will go, and where it "ends", which here means only that the next can begin. (The data may spread beyond the upper bound of the type, which, in my opinion, makes these names misleading, but this is a way of things.) You can indicate that this is all that you like, which makes it convenient for you; since we are dealing with elements in an int array, let's say the size of our MPI_INT is of the same type.
MPI_Type_create_resized(recvsubarray, 0, 1*sizeof(int), &resizedrevsubarray); MPI_Type_commit(&resizedrecvsubarray);
(Note that we should only do this for the type received, from the send type, since we send only one of them, it does not matter).
Now we will use gatherv to indicate where each element begins - in units of the "size" of this new modified type, which is only 1 integer. Therefore, if we want something to fall into a large array in [0][5] , the offset from the beginning of the large array is 5; if we want him to be there in position [5][5] , the offset is 55.
Finally, note that collective and dispersing groups suggest that even a “master” is involved. It’s easiest to get this work to work, even if the wizard has its own part of the global array.
So the following works for me:
#include <mpi.h> #include <iostream> #include <cstdlib> using namespace std; #define ROWS 10 #define COLUMNS 10 #define CHUNK_ROWS 5 #define CHUNK_COLUMNS 5 #define TAG 0 int** alloca_matrice(int righe, int colonne) { int** matrice=NULL; int i; matrice = (int **)malloc(righe * sizeof(int*)); if(matrice != NULL){ matrice[0] = (int *)malloc(righe*colonne*sizeof(int)); if(matrice[0]!=NULL) for(i=1; i<righe; i++) matrice[i] = matrice[0]+i*colonne; else{ free(matrice); matrice = NULL; } } else{ matrice = NULL; } return matrice; } int main(int argc, char* argv[]) { int my_id, numprocs,length,i,j; int ndims, sizes[2],subsizes[2],starts[2]; int** DEBUG_CH=NULL; int** ddd=NULL; int *recvptr=NULL; char name[BUFSIZ]; MPI_Datatype sendsubarray; MPI_Datatype recvsubarray; MPI_Datatype resizedrecvsubarray; //MPI_Status status; MPI_Init(&argc, &argv) ; MPI_Comm_rank(MPI_COMM_WORLD, &my_id) ; MPI_Comm_size(MPI_COMM_WORLD, &numprocs) ; // Ottiene quanti processi sono attivi if (numprocs != 4) { MPI_Abort(MPI_COMM_WORLD,1); } MPI_Get_processor_name(name, &length); //creo una sottomatrice ripulita dalle ghost cells ndims=2; sizes[0] = CHUNK_ROWS+2; sizes[1] = CHUNK_COLUMNS+2; subsizes[0] = CHUNK_ROWS; subsizes[1] = CHUNK_COLUMNS; starts[0] = 1; starts[1] = 1; MPI_Type_create_subarray(ndims,sizes,subsizes,starts,MPI_ORDER_C,MPI_INT,&sendsubarray); MPI_Type_commit(&sendsubarray); DEBUG_CH = alloca_matrice(CHUNK_ROWS+2,CHUNK_COLUMNS+2); for(i=0; i<CHUNK_ROWS+2; i++){ for(j=0; j<CHUNK_COLUMNS+2; j++){ if(i==0 || i==CHUNK_ROWS+1 || j==0 || j==CHUNK_COLUMNS+1) DEBUG_CH[i][j] = 5; else DEBUG_CH[i][j] = my_id; } } recvptr=DEBUG_CH[0]; if(my_id==0){ ddd = alloca_matrice(ROWS,COLUMNS); sizes[0] = ROWS; sizes[1] = COLUMNS; subsizes[0] = CHUNK_ROWS; subsizes[1] = CHUNK_COLUMNS; starts[0] = 0; starts[1] = 0; MPI_Type_create_subarray(2,sizes,subsizes,starts,MPI_ORDER_C,MPI_INT,&recvsubarray); MPI_Type_commit(&recvsubarray); MPI_Type_create_resized(recvsubarray, 0, 1*sizeof(int), &resizedrecvsubarray); MPI_Type_commit(&resizedrecvsubarray); recvptr = ddd[0]; } int counts[5]={1,1,1,1}; int disps[5] ={0,5,50,55}; MPI_Gatherv(DEBUG_CH[0],1,sendsubarray,recvptr,counts,disps,resizedrecvsubarray,0,MPI_COMM_WORLD); if(!my_id){ for(i=0; i<ROWS; i++){ for(j=0; j<COLUMNS; j++){ printf("%d ",ddd[i][j]); } printf("\n"); } } if(my_id == 0) { MPI_Type_free(&resizedrecvsubarray); MPI_Type_free(&recvsubarray); free(ddd[0]); free(ddd); } else { MPI_Type_free(&sendsubarray); free(DEBUG_CH[0]); free(DEBUG_CH); } MPI_Finalize(); // Chiusura di MPI. return 0; }