OpenCV: Is it possible to define a rectangle from corners?

Question

I have a photo where a person holds a sheet of paper. I would like to define the rectangle of this sheet of paper.

I tried using different guides from OpenCV and different SO answers and sample code to detect squares / rectangles, but the problem is that they all rely on some kind of outline.

If I follow the example squares.cpp, I get the following results from the outlines:

As you can see, the fingers are part of the outline, so the algorithm does not find the square.

I also tried using the approach HoughLines(), but I get similar results above:

I can detect corners reliably though:

There are other angles in the image, but I limit all the angles found for <50 and the corners for a sheet of paper always .

Is there any algorithm for finding a rectangle of several angles in an image? I can not find an existing approach.

Answer 1

You can apply a morphological filter to close the gaps in your edge image. Then, if you find the contours, you can find the inner closed loop, as shown below. Then find the convex hull of this contour to get a rectangle.

Closed Edges:

Contour:

ConvexHull:

, . .

Mat im = imread("Sh1Vp.png", 0); // the edge image
Mat kernel = getStructuringElement(MORPH_ELLIPSE, Size(11, 11));
Mat morph;
morphologyEx(im, morph, CV_MOP_CLOSE, kernel);

int rectIdx = 0;
vector<vector<Point>> contours;
vector<Vec4i> hierarchy;
findContours(morph, contours, hierarchy, CV_RETR_CCOMP, CV_CHAIN_APPROX_SIMPLE, Point(0, 0));
for (size_t idx = 0; idx < contours.size(); idx++)
{
    RotatedRect rect = minAreaRect(contours[idx]);
    double areaRatio = abs(contourArea(contours[idx])) / (rect.size.width * rect.size.height);
    if (areaRatio > .95)
    {
        rectIdx = idx;
        break;
    }
}
// get the convexhull of the contour
vector<Point> hull;
convexHull(contours[rectIdx], hull, false, true);

// visualization
Mat rgb;
cvtColor(im, rgb, CV_GRAY2BGR);
drawContours(rgb, contours, rectIdx, Scalar(0, 0, 255), 2);
for(size_t i = 0; i < hull.size(); i++)
{
    line(rgb, hull[i], hull[(i + 1)%hull.size()], Scalar(0, 255, 0), 2);
}