EDIT: Inspired by the hint from @Dan's comment, it can be simplified further to
R = reshape(rot90(M), size(M));
Original answer:
It should be an easy way to do it.
F = rot90(M); R = F(reshape(1:numel(M), size(M)))
which returns
R = 0 0 0 0 0 0 0 2 0 0 0 0 2 3 1 0 0 0 2 6 2 1 0 0 3 8 3 2 0 0
The idea is that when you rotate the matrix, you get
>> F = rot90(M) F = 0 2 1 1 0 0 3 2 2 0 2 6 3 0 0 2 8 0 0 0 3 0 0 0 0 0 0 0 0 0
which is a 6 by 5 matrix. If you are considering linear indexing over F, then the corresponding indices
>> reshape(1:30, size(F)) 1 7 13 19 25 2 8 14 20 26 3 9 15 21 27 4 10 16 22 28 5 11 17 23 29 6 12 18 24 30
where the elements 6, 11, 12, 16, 17, 18 and ... are now null if you change this to a 5 by 6 matrix, which you get
>> reshape(1:30, size(M)) 1 6 11 16 21 26 2 7 12 17 22 27 3 8 13 18 23 28 4 9 14 19 24 29 5 10 15 20 25 30
Now those elements that correspond to zero values ββare on top, exactly what we wanted. Therefore, passing this index array to F , we get the desired R