OpenCV findFundamentalMat is very unstable and sensitive

Question

I am working on a project for my university where we want the Quadrcopter to stabilize with its camera. Unfortunately, the Fundamental Matrix reacts very intelligently to small changes inside featurpoints, I will give you examples later.

I think my match already works very well thanks to ocv. I use SURF Features and map them to the knn-Method:

SurfFeatureDetector surf_detect; surf_detect = SurfFeatureDetector(400); //detect keypoints surf_detect.detect(fr_one.img, fr_one.kp); surf_detect.detect(fr_two.img, fr_two.kp); //extract keypoints SurfDescriptorExtractor surf_extract; surf_extract.compute(fr_one.img, fr_one.kp, fr_one.descriptors); surf_extract.compute(fr_two.img, fr_two.kp, fr_two.descriptors); //match keypoints vector<vector<DMatch> > matches1,matches2; vector<DMatch> symMatches,goodMatches; FlannBasedMatcher flann_match; flann_match.knnMatch(fr_one.descriptors, fr_two.descriptors, matches1,2); flann_match.knnMatch(fr_two.descriptors, fr_one.descriptors, matches2,2); //test matches in both ways symmetryTest(matches1,matches2,symMatches); std::vector<cv::Point2f> points1, points2; for (std::vector<cv::DMatch>::const_iterator it= symMatches.begin(); it!= symMatches.end(); ++it) { //left keypoints float x= fr_one.kp[it->queryIdx].pt.x; float y= fr_one.kp[it->queryIdx].pt.y; points1.push_back(cv::Point2f(x,y)); //right keypoints x = fr_two.kp[it->trainIdx].pt.x; y = fr_two.kp[it->trainIdx].pt.y; points2.push_back(cv::Point2f(x,y)); } //kill outliers with ransac vector<uchar> inliers(points1.size(),0); findFundamentalMat(Mat(points1),Mat(points2), inliers,CV_FM_RANSAC,3.f,0.99f); std::vector<uchar>::const_iterator itIn= inliers.begin(); std::vector<cv::DMatch>::const_iterator itM= symMatches.begin(); for ( ;itIn!= inliers.end(); ++itIn, ++itM) { if (*itIn) { goodMatches.push_back(*itM); } } 

Now I want to calculate the main matrix with these matches. I use the 8POINT method for this example - I already tried it with LMEDS and RANSAC - there it only worsens because there are more matches that change.

  vector<int> pointIndexes1; vector<int> pointIndexes2; for (vector<DMatch>::const_iterator it= goodMatches.begin(); it!= goodMatches.end(); ++it) { pointIndexes1.push_back(it->queryIdx); pointIndexes2.push_back(it->trainIdx); } vector<Point2f> selPoints1, selPoints2; KeyPoint::convert(fr_one.kp,selPoints1,pointIndexes1); KeyPoint::convert(fr_two.kp,selPoints2,pointIndexes2); Mat F = findFundamentalMat(Mat(selPoints1),Mat(selPoints2),CV_FM_8POINT); 

When I call these calculations in a cycle on the same pair of images, the result F changes very much - it is not possible to extract motion from such calculations.

I generated an example in which I filtered out some matches so you can see the effect that I mentioned for myself.

http://abload.de/img/div_c_01ascel.png

http://abload.de/img/div_c_02zpflj.png

Is there something wrong with my code, or should I think about other reasons, such as image quality and so on?

Thanks in advance for your help! derfreak

Answer 1

To summarize what others have already stated and detailed,

• As currently implemented in OpenCV, the 8-point algorithm does not deviate from the outlier. This is a least squares algorithm and cannot be used with RANSAC or LMEDS because these flags override the 8-point flag . It is recommended that the input points be normalized to improve the matrix condition number in the linear equation, as indicated in “In Defense of the 8-Point Algorithm” . However, the OpenCV implementation automatically normalizes the input points , so there is no need to manually normalize them.

• 5-point and 7-point algorithms deviate from emissions using RANSAC or LMEDS. If you use RANSAC, you may need to adjust the threshold to get good results. The OpenCV documentation shows that the default threshold for RANSAC is 1.0, which, in my opinion, is a bit large. I would recommend using something around 0.1 pixels. On the other hand, if you use LMEDS, you will not need to worry about the threshold, because LMEDS minimizes the median error, rather than calculating the values. LMEDS and RANSAC have similar accuracy if the correct threshold is used, and both have comparable calculation times.

• The 5-point algorithm is more reliable than the 7-point algorithm, since it has only 5 degrees of freedom (3 rotation and 2 for converting a unit vector) instead of 7 (additional 2 parameters for the camera reference points). This minimal parameterization allows rotation and translation to be easily extracted from the matrix using SVD and avoids the problem of degeneration of a flat structure .

• However, to obtain accurate results using a 5-point algorithm, the focal length must be known. The document suggests that the focal length should be known within 10%, otherwise the 5-point algorithm is no better than other un-calibrated algorithms. If you have not calibrated the camera yet, see the OpenCV camera calibration guide . In addition, if you use ROS, there is a good camera calibration package .

• When using the OpenCV findEssentialMat function, I recommend first passing the pixel points to undistortPoints . This not only cancels the lens distortion effect, but also converts the coordinates into normalized image coordinates. The normalized coordinates of the image (not to be confused with the normalization performed in the 8-point algorithm) are the agnostic coordinates of the camera, which are independent of any internal parameters of the camera. They represent the angle of the bearing vector to the point in the real world. For example, the normalized coordinate of the image (1, 0) will correspond to the carrier angle of 45 degrees from the optical axis of the camera in the x direction and 0 degrees in the y direction.

• After using RANSAC to obtain a good hypothesis, the best score can be improved by using iterative reliable non-linear least squares. This is mentioned in the article and is described in more detail in "Bundle Adjustment - Modern Synthesis" . Unfortunately, it seems that the OpenCV implementation in the 5-point algorithm does not use iterative refinement methods.

Answer 2

Even if your algorithm is correct, the 8-matrix calculation of the matrix F is very error prone due to image noise. The fewer matches you use, the better. The best you can do is do a 5-point calculation of the Essential (E) matrix, but this will require you to pre-calibrate the camera and convert the found pixel points of the image after SIFT / SURF to normalized pixels (metric pixel locations). Then apply the Nister 5-point algorithm from either the freely available Matlab implementation or the Bundler (C ++ Noah Snawell implementation). In my experience with SfM, a 5-point E-matrix is ​​much better / more stable than a 7 or 8-point F-matrix. And of course, RANSAC after 5 points to get more reliable grades. Hope this helps.

Answer 3

The 8-point algorithm is the easiest way to calculate the fundamental matrix, but if you take action, you can perform it well. The key to obtaining good results is the correct thorough normalization of the input data before constructing the solvable equations. Many algorithms can do this.

The pixel point coordinate must be changed to the camera coordinates . I do not see that you are doing this. As far as I understand, your

vector<int> pointIndexes1; expressed in pixel coordinates. You must know the parameters of the built-in camera if you want to get more stable results. You can find them in many ways: the openCV tutorial . Then you have two options for normalizing it. You can apply for your core matrix,

Mat E = Kt() * F * K; where K are the parameters of the internal camera. [cm. on the wiki]

However, this assumption is incorrect. If the camera calibration matrix K is known, then you can apply the inverse to the point x to obtain the point expressed in the normalized coordinates of the camera.

pointNormalize1= K.inv()*pointIndexes1 where pointIndexes1(2) , z is 1.

In the case of 8PA, simple point conversion is improved and therefore in the stability of the results. The proposed normalization is the translation and scaling of each image, so that the centroid of the control points is at the origin, and the rms distance of the points from the beginning is equal! [Sqrt {2}]. Please note that before denormalization, it is recommended to fulfill the singularity condition.

Link: check if: you are still interested