Computer Vision - filter convex hulls and convex defects using OpenCV

Question

I have a problem processing digital signals. I am trying to detect fingertips similar to the solution presented here: Detecting hands and fingers using JavaCV .

However, I am not using JavaCV, but OpenCV for Android, which is slightly different. I managed to do all the steps presented in the textbook, but filtering convex hulls and convex defects. Here's what my image looks like:

Here is the image in a different resolution:

As you can clearly see, there are many yellow dots (convex hulls), as well as many red dots (convexity). Sometimes there is no red dot between two yellow points, which is rather strange (how are convex hulls calculated?)

I need to create a simillar filtering function, as in the previous link, but using OpenCV data structures.

A convex hull is a type of MatOfInt ... Convex defects is a type of MatOfInt4 ...

I also created some additional data structures because stupid OpenCV uses different types of data containing the same data in different ways ...

convexHullMatOfInt = new MatOfInt(); convexHullPointArrayList = new ArrayList<Point>(); convexHullMatOfPoint = new MatOfPoint(); convexHullMatOfPointArrayList = new ArrayList<MatOfPoint>(); 

Here is what I have done so far, but it does not work well. Probably the problem is data conversion incorrectly:

Creation of convex hulls and convex defects:

 public void calculateConvexHulls() { convexHullMatOfInt = new MatOfInt(); convexHullPointArrayList = new ArrayList<Point>(); convexHullMatOfPoint = new MatOfPoint(); convexHullMatOfPointArrayList = new ArrayList<MatOfPoint>(); try { //Calculate convex hulls if(aproximatedContours.size() > 0) { Imgproc.convexHull( aproximatedContours.get(0), convexHullMatOfInt, false); for(int j=0; j < convexHullMatOfInt.toList().size(); j++) convexHullPointArrayList.add(aproximatedContours.get(0).toList().get(convexHullMatOfInt.toList().get(j))); convexHullMatOfPoint.fromList(convexHullPointArrayList); convexHullMatOfPointArrayList.add(convexHullMatOfPoint); } } catch (Exception e) { // TODO Auto-generated catch block Log.e("Calculate convex hulls failed.", "Details below"); e.printStackTrace(); } } public void calculateConvexityDefects() { mConvexityDefectsMatOfInt4 = new MatOfInt4(); try { Imgproc.convexityDefects(aproximatedContours.get(0), convexHullMatOfInt, mConvexityDefectsMatOfInt4); if(!mConvexityDefectsMatOfInt4.empty()) { mConvexityDefectsIntArrayList = new int[mConvexityDefectsMatOfInt4.toArray().length]; mConvexityDefectsIntArrayList = mConvexityDefectsMatOfInt4.toArray(); } } catch (Exception e) { Log.e("Calculate convex hulls failed.", "Details below"); e.printStackTrace(); } } 

Filtration:

 public void filterCalculatedPoints() { ArrayList<Point> tipPts = new ArrayList<Point>(); ArrayList<Point> foldPts = new ArrayList<Point>(); ArrayList<Integer> depths = new ArrayList<Integer>(); fingerTips = new ArrayList<Point>(); for (int i = 0; i < mConvexityDefectsIntArrayList.length/4; i++) { tipPts.add(contours.get(0).toList().get(mConvexityDefectsIntArrayList[4*i])); tipPts.add(contours.get(0).toList().get(mConvexityDefectsIntArrayList[4*i+1])); foldPts.add(contours.get(0).toList().get(mConvexityDefectsIntArrayList[4*i+2])); depths.add(mConvexityDefectsIntArrayList[4*i+3]); } int numPoints = foldPts.size(); for (int i=0; i < numPoints; i++) { if ((depths.get(i).intValue()) < MIN_FINGER_DEPTH) continue; // look at fold points on either side of a tip int pdx = (i == 0) ? (numPoints-1) : (i - 1); int sdx = (i == numPoints-1) ? 0 : (i + 1); int angle = angleBetween(tipPts.get(i), foldPts.get(pdx), foldPts.get(sdx)); if (angle >= MAX_FINGER_ANGLE) // angle between finger and folds too wide continue; // this point is probably a fingertip, so add to list fingerTips.add(tipPts.get(i)); } } 

Results (white dots - fingertips after filtering):

Could you help me write the correct filtering function?

UPDATE 08/14/2013

I use the standard openCV function to approximate the contours. I have to change the approximation value with the resolution of the resolution change and the distance from the hand to the camera, which is quite difficult to do. If the resolution is less, then the finger consists of a smaller pixel, so the approximation value should be a lover. Same thing with distance. Keeping this maximum will result in a complete loss with your finger. Therefore, I believe that approximation is not a good approach to solving the problem, however, a small value may be useful to speed up the calculations:

 Imgproc.approxPolyDP(frame, frame, 2 , true); 

If I use high values, the result will be similar to the image below, which would be good only if the distance and resolution have not changed. In addition, I am completely surprised that the default methods for points and points of defects in the case do not have useful arguments for passing (minimum angle, distance, etc.) ...

Below is the effect that I would always like to achieve regardless of resolution or the distance from the hand to the camera. Also I do not want to see yellow dots when I close my palm ...

To summarize, I would like to know:

• how to filter points
• How can I make resolution and distance-independent approximation that will always work.
• If someone knows or has some material (graphical representation, explanation) about these data structures used in OpenCV, I would be happy to read it. (Mat, MatOfInt, MatOfPoint, MatOfPoint2, MatOfPoint4, etc.).

Answer 1

The convex hull with low resolution can be used to determine the position of the hand as a whole, it is not useful for the fingers, but provides an area of ​​interest and an appropriate scale.

Then a higher resolution analysis will be applied to your approximate contour, it is easy to skip any points that do not pass the criteria “length and angle” from the last two, although you may want to “average” instead of “skip completely”.

An example of your code is a single pass to calculate the bulge defects and then remove them .. this is a logical mistake .. you need to delete the points when you go. (a) it’s faster and easier to do everything in one -pass (b) he avoids removing points from the first pass and must add them later, because any deletion changes the previous calculations.

This basic technique is very simple and therefore works for the main open hand. However, he does not understand the essence of the hand or gestures, so setting the parameters for scale, angle and length will only ever make you "still."

Method references: filter length and convex defect angle Simen Andresen blog http://simena86.imtqy.com/blog/2013/08/12/hand-tracking-and-recognition-with-opencv/

C # library based on the Kinect SDK with the addition of fingers http://candescentnui.codeplex.com/ http://blog.candescent.ch/2011/11/improving-finger-detection.html

"Self-soluble and organized neural gas" (SGONG) Prof. Nikos Papamarkos http://www.papamarkos.gr/uploaded-files/Hand%20gesture%20recognition%20using%20a%20neural%20network%20shape%20fitting%20technique.pdf

Commercial product David Holtz and Michael Backwald founders of Leap Motion http://www.engadget.com/2013/03/11/leap-motion-michael-buckwald-interview/

Answer 2

I think you missed this point:

Corpus creation and defect analysis are accelerated by using a low poly approximation of the contour rather than the original.