OpenCV - Java - Matching Two Opposite Images Using a Matcher Descriptor

Question

I am trying to match 2 opposing images using OpenCV DescriptorMatcher with no luck. Images: http://i61.tinypic.com/28whu0g.jpg (from left to right) and http://i61.tinypic.com/x35vte.jpg (from right to left).

My code is very similar to the many examples I've seen on StackOverflow and the Internet, but still I always don't get a match.

String firstImageSourcePath = "RTL_IMAGE_PATH"; String secondImageSourcePath = "LTR_IMAGE_PATH"; Mat firstImageSrcImgMat = Highgui.imread(firstImageSourcePath); Mat secondImageSrcImgMat = Highgui.imread(firstImageSourcePath); if (firstImageSrcImgMat.empty() || secondImageSrcImgMat.empty()) { System.out.println("Failed to load images"); return; } System.out.println("Loaded image at " + firstImageSourcePath + " and " + secondImageSourcePath); FeatureDetector featureDetector = FeatureDetector.create(FeatureDetector.BRISK); MatOfKeyPoint firstImgMatOfKeyPoints = new MatOfKeyPoint(); MatOfKeyPoint secondImgMatOfKeyPoints = new MatOfKeyPoint(); featureDetector.detect(firstImageSrcImgMat, firstImgMatOfKeyPoints); featureDetector.detect(secondImageSrcImgMat, secondImgMatOfKeyPoints); System.out.println("Detected " + firstImgMatOfKeyPoints.size() + " and " + secondImgMatOfKeyPoints + " blobs in the images"); List<KeyPoint> firstImgKeyPoints = firstImgMatOfKeyPoints.toList(); List<KeyPoint> secondImgKeyPoints = secondImgMatOfKeyPoints.toList(); System.out.println("First Image key points: " + firstImgKeyPoints); System.out.println("Second Image key points: " + secondImgKeyPoints); Mat firstImgDescriptors = new Mat(); Mat secondImgDescriptors = new Mat(); DescriptorExtractor extractor = DescriptorExtractor.create(DescriptorExtractor.BRISK); extractor.compute(firstImageSrcImgMat, firstImgMatOfKeyPoints, firstImgDescriptors); extractor.compute(secondImageSrcImgMat, secondImgMatOfKeyPoints, secondImgDescriptors); System.out.println("descriptorsA.size() : " + firstImgDescriptors.size()); System.out.println("descriptorsB.size() : " + secondImgDescriptors.size()); MatOfDMatch matches = new MatOfDMatch(); DescriptorMatcher matcher = DescriptorMatcher.create(DescriptorMatcher.BRUTEFORCE_HAMMINGLUT); // BRUTEFORCE_HAMMINGLUT matcher.match(firstImgDescriptors, secondImgDescriptors, matches); System.out.println("matches.size() : " + matches.size()); System.out.println("matches : " + matches); MatOfDMatch matchesFiltered = new MatOfDMatch(); List<DMatch> matchesList = matches.toList(); List<DMatch> bestMatches = new ArrayList<DMatch>(); Double max_dist = 0.0; Double min_dist = 100.0; for (int i = 0; i < matchesList.size(); i++) { Double dist = (double) matchesList.get(i).distance; if (dist > 0) System.out.println("dist : " + dist); if (dist < min_dist && dist != 0) { min_dist = dist; } if (dist > max_dist) { max_dist = dist; } } System.out.println("max_dist : " + max_dist); System.out.println("min_dist : " + min_dist); if (min_dist > 50) { System.out.println("No match found, min_dist under minimum value"); return; } double threshold = 3 * min_dist; double threshold2 = 2 * min_dist; if (threshold > 75) { threshold = 75; } else if (threshold2 >= max_dist) { threshold = min_dist * 1.1; } else if (threshold >= max_dist) { threshold = threshold2 * 1.4; } System.out.println("Threshold : " + threshold); for (int i = 0; i < matchesList.size(); i++) { Double dist = (double) matchesList.get(i).distance; if (dist < threshold) { bestMatches.add(matches.toList().get(i)); System.out.println(String.format(i + " best match added : %s", dist)); } } matchesFiltered.fromList(bestMatches); System.out.println("matchesFiltered.size() : " + matchesFiltered.size()); if (matchesFiltered.rows() >= 1) { System.out.println("match found"); } else { System.out.println("match not found"); } 

any hint what am i doing wrong?

Answer 1

As Iwillnotexist-Idonotexist said, the first problem is the threshold that you apply. Try using a threshold that is independent of the distance between descriptors that do not work well, as some descriptors are much more discriminatory than others. I think this will give you better results. I advise you to use the correlation test proposed by D. Low in SIFT paper. Please see section 7.1: http://cs.ubc.ca/~lowe/papers/ijcv04.pdf

The second problem is that you use BRISK to detect features in your images. There are errors in this OpenCV implementation (you can check here: http://code.opencv.org/issues/3976 ), so try using a different FeatureDetector such as FAST, ORB, etc. (The handle is fine so you can continue to use it)

I finished testing on your pictures, and I managed to get some results with various detectors / descriptors: (key points without matching → yellow)

BRISK detector and descriptor:

• key points of the left image: 74
• key points of the right image: 86
• matches: 3 (Even with a broken detector, I got matches)

ORB detector with BRISK as a handle:

• left key image points: 499
• keywords for the right image: 500
• matches: 26

detector and descriptor ORB

• left key image points: 841
• key points of the right image: 907
• matches: 43

All results were obtained using a relationship test to remove false matches. Hope this helps!

EDIT:

 BruteForceMatcher<Hamming> matcher; vector< vector<DMatch> > matches; vector <DMatch> goodMatches; matcher.knnMatch(imgDescriptors1, imgDescriptors2, matches, 2); // Ratio Test for (unsigned int matchIdx = 0; matchIdx < matches.size(); ++matchIdx) { const float ratio = 0.8; // As in Lowe paper (can be tuned) if (matches[matchIdx][0].distance < ratio * matches[matchIdx][1].distance) { goodMatches.push_back(matches[matchIdx][0]); } } 

Answer 2

Java version:

 DescriptorMatcher descriptorMatcher; descriptorMatcher=DescriptorMatcher.create(DescriptorMatcher.BRUTEFORCE_HAMMING ); MatOfDMatch good_matches; good_matches = new MatOfDMatch(); LinkedList<MatOfDMatch> dmatchesListOfMat = new LinkedList<>(); descriptorMatcher.knnMatch(imgDescriptors1, imgDescriptors2, dmatchesListOfMat, 2); LinkedList<DMatch> good_matchesList = new LinkedList<>(); for (int matchIndx = 0; matchIndx < dmatchesListOfMat.size() ; matchIndx++) { double ratio = 0.8; if (dmatchesListOfMat.get(matchIndx).toArray()[0].distance < ratio * dmatchesListOfMat.get(matchIndx).toArray()[1].distance) { good_matchesList.addLast(dmatchesListOfMat.get(matchIndx).toArray()[0]); } } good_matches.fromList(good_matchesList);