Problems with opencv stereoRectifyUncalibrated

Question

I am trying to fix and build a mismatch displayed for a pair of images using OpenCV stereoRectifyUncalibrated, but I am not getting very good results. My code is:

template<class T> T convertNumber(string& number) { istringstream ss(number); T t; ss >> t; return t; } void readPoints(vector<Point2f>& points, string filename) { fstream filest(filename.c_str(), ios::in); string line; assert(filest != NULL); getline(filest, line); do{ int posEsp = line.find_first_of(' '); string posX = line.substr(0, posEsp); string posY = line.substr(posEsp+1, line.size() - posEsp); float X = convertNumber<float>(posX); float Y = convertNumber<float>(posY); Point2f pnt = Point2f(X, Y); points.push_back(pnt); getline(filest, line); }while(!filest.eof()); filest.close(); } void drawKeypointSequence(Mat lFrame, Mat rFrame, vector<KeyPoint>& lKeyp, vector<KeyPoint>& rKeyp) { namedWindow("prevFrame", WINDOW_AUTOSIZE); namedWindow("currFrame", WINDOW_AUTOSIZE); moveWindow("prevFrame", 0, 300); moveWindow("currFrame", 650, 300); Mat rFrameAux; rFrame.copyTo(rFrameAux); Mat lFrameAux; lFrame.copyTo(lFrameAux); int size = rKeyp.size(); for(int i=0; i<size; i++) { vector<KeyPoint> drawRightKeyp; vector<KeyPoint> drawleftKeyp; drawRightKeyp.push_back(rKeyp[i]); drawleftKeyp.push_back(lKeyp[i]); cout << rKeyp[i].pt << " <<<>>> " << lKeyp[i].pt << endl; drawKeypoints(rFrameAux, drawRightKeyp, rFrameAux, Scalar::all(255), DrawMatchesFlags::DRAW_OVER_OUTIMG); drawKeypoints(lFrameAux, drawleftKeyp, lFrameAux, Scalar::all(255), DrawMatchesFlags::DRAW_OVER_OUTIMG); imshow("currFrame", rFrameAux); imshow("prevFrame", lFrameAux); waitKey(0); } imwrite("RightKeypFrame.jpg", rFrameAux); imwrite("LeftKeypFrame.jpg", lFrameAux); } int main(int argc, char* argv[]) { StereoBM stereo(StereoBM::BASIC_PRESET, 16*5, 21); double ndisp = 16*4; assert(argc == 5); string rightImgFilename(argv[1]); // Right image (current frame) string leftImgFilename(argv[2]); // Left image (previous frame) string rightPointsFilename(argv[3]); // Right image points file string leftPointsFilename(argv[4]); // Left image points file Mat rightFrame = imread(rightImgFilename.c_str(), 0); Mat leftFrame = imread(leftImgFilename.c_str(), 0); vector<Point2f> rightPoints; vector<Point2f> leftPoints; vector<KeyPoint> rightKeyp; vector<KeyPoint> leftKeyp; readPoints(rightPoints, rightPointsFilename); readPoints(leftPoints, leftPointsFilename); assert(rightPoints.size() == leftPoints.size()); KeyPoint::convert(rightPoints, rightKeyp); KeyPoint::convert(leftPoints, leftKeyp); // Desenha os keypoints sequencialmente, de forma a testar a consistência do matching drawKeypointSequence(leftFrame, rightFrame, leftKeyp, rightKeyp); Mat fundMatrix = findFundamentalMat(leftPoints, rightPoints, CV_FM_8POINT); Mat homRight; Mat homLeft; Mat disp16 = Mat(rightFrame.rows, leftFrame.cols, CV_16S); Mat disp8 = Mat(rightFrame.rows, leftFrame.cols, CV_8UC1); stereoRectifyUncalibrated(leftPoints, rightPoints, fundMatrix, rightFrame.size(), homLeft, homRight); warpPerspective(rightFrame, rightFrame, homRight, rightFrame.size()); warpPerspective(leftFrame, leftFrame, homLeft, leftFrame.size()); namedWindow("currFrame", WINDOW_AUTOSIZE); namedWindow("prevFrame", WINDOW_AUTOSIZE); moveWindow("currFrame", 650, 300); moveWindow("prevFrame", 0, 300); imshow("currFrame", rightFrame); imshow("prevFrame", leftFrame); imwrite("RectfRight.jpg", rightFrame); imwrite("RectfLeft.jpg", leftFrame); waitKey(0); stereo(rightFrame, leftFrame, disp16, CV_16S); disp16.convertTo(disp8, CV_8UC1, 255/ndisp); FileStorage file("disp_map.xml", FileStorage::WRITE); file << "disparity" << disp8; file.release(); imshow("disparity", disp8); imwrite("disparity.jpg", disp8); moveWindow("disparity", 0, 0); waitKey(0); } 

The drawKeyPoint sequence is how I visually check the consistency of the points that I have for both images. By choosing each of my key points in sequence, I can be sure that the key point i in image A is the key point i in image B.

I also tried playing with the ndisp parameter, but that didn't help much.

I tried it for the following two images:

Leftimage

RightImage

got the following fixed pair:

Rectifiedleft

Rectifiedright

and finally the following mismatch display

DisparityMap

Which, as you can see, is pretty bad. I also tried the same pair of images with the following stereogram pattern example: "http://programmingexamples.net/wiki/OpenCV/WishList/StereoRectifyUncalibrated and SBM_Sample.cpp from the sample opencv instructional code for building the mismatch map, and got a very similar result .

I am using opencv 2.4

Thanks in advance!

Answer 1

In addition to possible calibration problems, your images clearly lack texture for stereo block compatibility. This algorithm will see many ambiguities and too large differences on the flat (non-texed) parts.

Note that the key points seem to match well, so even if the straightening output seems strange, this is probably correct.

You can check your code on standard images from the middle page of Middlebury to check the performance.

Answer 2

I would suggest stereoscaling with a checkerboard or take a few pictures with a checkerboard and use stereocalibrate.cpp on your computer. I say this because you are using stereorectifyuncalibrated . Although the algorithm does not need to know the internal parameters of the cameras, it is heavily dependent on epipolar geometry. Therefore, if the camera lenses have significant distortion, it would be better to fix it before calculating the main matrix and calling this function. For example, distortion coefficients can be estimated for each stereo camera head separately using calibrateCamera() . Then the images can be corrected using undistort() , or only the coordinates of the point can be adjusted using undistortPoints() .