Combining overlapping groups in an image

Question

I use opencv_contrib to detect text areas in the image.

This is the original image.

This image is after the text areas are found:

As you can see, there are overlapping groups in the image. For example, it seems that there are two groups around Hello World and two around Some more sample text

Question In scenarios like this, I can save the maximum possible box by combining the two fields. For these examples, which will start with H and end with d so that it covers Hello World . My reason is that I would like to crop part of this image and send it to tesseract.

Here is the corresponding code that draws the fields.

 void groups_draw(Mat &src, vector<Rect> &groups) { for (int i=(int)groups.size()-1; i>=0; i--) { if (src.type() == CV_8UC3) rectangle(src,groups.at(i).tl(),groups.at(i).br(),Scalar( 0, 255, 255 ), 2, 8 ); } } 

Here is what I have tried. My ideas in the comments.

 void groups_draw(Mat &src, vector<Rect> &groups) { int previous_tl_x = 0; int previous_tl_y = 0; int prevoius_br_x = 0; int previous_br_y = 0; //sort the groups from lowest to largest. for (int i=(int)groups.size()-1; i>=0; i--) { //if previous_tl_x is smaller than current_tl_x then keep the current one. //if previous_br_x is smaller than current_br_x then keep the current one. if (src.type() == CV_8UC3) { //crop the image Mat cropedImage = src(Rect(Point(groups.at(i).tl().x, groups.at(i).tl().y),Point(groups.at(i).br().x, groups.at(i).br().y))); imshow("cropped",cropedImage); waitKey(-1); } } } 

Update

I am trying to use [groupRectangles][4] to accomplish this:

 void groups_draw(Mat &src, vector<Rect> &groups) { vector<Rect> rects; for (int i=(int)groups.size()-1; i>=0; i--) { rects.push_back(groups.at(i)); } groupRectangles(rects, 1, 0.2); } 

However, this gives me an error:

 textdetection.cpp:106:5: error: use of undeclared identifier 'groupRectangles' groupRectangles(rects, 1, 0.2); ^ 1 error generated. 

Answer 1

Firstly, the reason you get overlapping bounding boxes is because the text detector module works with inverted channels (for example, gray and inverted gray), and because of this, the inner areas of some characters like o and g grouped as characters. Therefore, if you want to detect only one text mode (white text on a dark background), just transmit the inverted channels. Replace:

  for (int c = 0; c < cn-1; c++) channels.push_back(255-channels[c]); 

WITH

 for (int c = 0; c < cn-1; c++) channels[c] = (255-channels[c]); 

Now for your question, the rectangles have defined the intersection and union operations:

 rect = rect1 & rect2 (rectangle intersection) rect = rect1 | rect2 (minimum area rectangle containing rect2 and rect3 ) rect &= rect1, rect |= rect1 (and the corresponding augmenting operations) 

You can use these operators, iterating over rectangles, to detect intersecting rectangles and combine them as follows:

 if ((rect1 & rect2).area() != 0) rect1 |= rect2; 

Edit:

First sort the groups of rectangles by area from largest to smallest:

 std::sort(groups.begin(), groups.end(), [](const cv::Rect &rect1, const cv::Rect &rect2) -> bool {return rect1.area() > rect2.area();}); 

Then, iterating over the rectangles, when the two rectangles intersect, add the smaller to the larger, and then delete it:

 for (int i = 0; i < groups.size(); i++) { for (int j = i + 1; j < groups.size(); j++) { if ((groups[i] & groups[j]).area() != 0) { groups[i] |= groups[j]; groups.erase(groups.begin() + j--); } } } 

Answer 2

One approach is to compare each rectangle with any other rectangle to see if they intersect or intersect. If they do enough, you can combine them into one larger rectangle.