Another approach is to use bwdistgeodesic to find the order of the angles by their distance along your edge. This should work for any polygon where you can detect a continuous edge.
We start by loading an image from Qaru and converting it to a black and white image to make it easier to find the edge.
A = imread('http://i.stack.imgur.com/dpbpP.jpg'); A_bw = im2bw(A,100/255); %binary image A_bw1 = imcomplement(A_bw); %inverted binary image
The bwmorph function provides many possibilities for processing black and white images. We will use the remove option to find the edge of our polygon, but you can also use a different edge detector if you want.
%Find the edges A_edges = bwmorph(A_bw, 'remove'); [edge_x, edge_y] = find(A_edges');
Let us visualize the edges we discovered.
figure; imshow(A_edges);

Ok, we have a nice clear continuous rib. Now find the corners. I use corner , but you can replace your favorite angle detector
A_corners = corner(A_bw1, 'QualityLevel',.3);
Let us visualize our initial angular order.
figure; imshow(A_bw1); hold on plot(A_corners(:,1), A_corners(:,2), 'r.', 'MarkerSize', 18) text(A_corners(:,1), A_corners(:,2), strsplit(num2str(1:length(A_corners))), 'Color', 'g', 'FontSize', 24); hold off

Another thing that you might not notice is that they are not directly at the edges. To begin with, I will find the closest point along the edge to each corner point, and then visualize the corners in red and the closest border points in green.
[~, ind] = min(pdist2(A_corners, [edge_x, edge_y]), [], 2); A_edge_corners = [edge_x(ind), edge_y(ind)]; figure; imshow(A_edges); hold on; plot(A_corners(:,1), A_corners(:,2), 'r.', 'MarkerSize', 18) plot(A_edge_corners(:,1), A_edge_corners(:,2),'g.', 'MarkerSize', 18) hold off;

To calculate the distance around the edge for each corner, we will use the corner point approximation, A_edge_corners (green dot) on the edge, and not the A_corners corner dot (red dot) itself.
Now we have all the parts that we need to use bwdistgeodesic . This function finds the distance to the starting point for each non-zero pixel in a black and white image. We are interested in the distance from the starting angle to each point on the edge. Give it a try.
% Calculate distance from seed corner first_corner = A_edge_corners(1,:); D = bwdistgeodesic(A_edges, first_corner(1), first_corner(2)); figure; imagesc(D);

We start from the right-most corner, and pixels moving away from the corner increase. But this is not exactly what we want. If we order angles using these values, we get ordering at a distance from the starting point.
[~, corner_order] = sort(D(sub2ind(size(D), A_edge_corners(:,2), A_edge_corners(:,1)))); A_corners_reorder1 = A_corners(corner_order, :); figure; imshow(A_bw1); hold on plot(A_corners_reorder1(:,1), A_corners_reorder1(:,2),'r.', 'MarkerSize', 18) text(A_corners_reorder1(:,1), A_corners_reorder1(:,2), strsplit(num2str(1:length(A_corners))), 'Color', 'g', 'FontSize', 24); hold off

To solve this problem, we just need to break the edge so that the ordering takes place only in one direction from the starting point. If you are interested in ordering clockwise or counterclockwise, you will need to split the edge according to a set of rules depending on the orientation of the edge. If the direction does not matter, you can just find the neighboring pixel in the starting corner and break it there.
%Break the edge into one path by removing a pixel adjacent to first corner %If the corner is near the edge of the image, you would need to check for %edge conditions window = A_edges(first_corner(2)-1:first_corner(2)+1, first_corner(1)-1:first_corner(1)+1); window(2,2) = 0; %Exclude the corner itself [x, y] = find(window, 1); A_edges(first_corner(2)+x-2, first_corner(1)+y-2) = 0; figure; imshow(A_edges); hold on; plot(first_corner(1), first_corner(2), 'r.', 'MarkerSize', 18) hold off;

Now the distance from the starting point along the edge can follow only one path
%Find order the pixels along edge D = bwdistgeodesic(A_edges, first_corner(1), first_corner(2)); figure; imagesc(D);

This gives us the desired edge order.
[~, corner_order] = sort(D(sub2ind(size(D), A_edge_corners(:,2), A_edge_corners(:,1)))); A_corners = A_corners(corner_order, :); figure; imshow(A_bw1); hold on plot(A_corners(:,1), A_corners(:,2),'r.', 'MarkerSize', 18) text(A_corners(:,1), A_corners(:,2), strsplit(num2str(1:length(A_corners))), 'Color', 'g', 'FontSize', 24); hold off

This method also works with polygons that are not balanced with respect to the centroid, for example, in the second demo image.

For fun, I present the third image with a vertex on the opposite side of the centroid, as a clearer example of an unbalanced polygon. My method also correctly analyzes this image.

