OpenCV - Inclined camera and triangulation for stereo vision

Question

I use a stereo system, so I'm trying to get the world coordinates of some points by triangulation.

My cameras have an angle, the Z axis direction (depth direction) is not normal for my surface. Therefore, when I observe a flat surface, I do not get a constant depth, but a “linear” variation, right? And I want depth from the baseline ... How can I redesign?

Part of my code with my projective arrays and triangulation function:

#C1 and C2 are the cameras matrix (left and rig) #R_0 and T_0 are the transformation between cameras #Coord1 and Coord2 are the correspondant coordinates of left and right respectively P1 = np.dot(C1,np.hstack((np.identity(3),np.zeros((3,1))))) P2 =np.dot(C2,np.hstack(((R_0),T_0))) for i in range(Coord1.shape[0]) z = cv2.triangulatePoints(P1, P2, Coord1[i,],Coord2[i,]) 

-------- EDIT LATER -----------

Thanks scribbleink, so I tried to apply your suggestion. But I think I have a mistake, because it does not work well, as you can see below. Point clouds seem to be distorted and curved towards the edges of the image.

 U, S, Vt = linalg.svd(F) V = Vt.T #Right epipol U[:,2]/U[2,2] # The expected X-direction with C1 camera matri and C1[0,0] the focal length vecteurX = np.array([(U[:,2]/U[2,2])[0],(U[:,2]/U[2,2])[1],C1[0,0]]) vecteurX_unit = vecteurX/np.sqrt(vecteurX[0]**2 + vecteurX[1]**2 + vecteurX[2]**2) # The expected Y axis : height = 2048 vecteurY = np.array([0, height -1, 0]) vecteurY_unit = vecteurY/np.sqrt(vecteurY[0]**2 + vecteurY[1]**2 + vecteurY[2]**2) # The expected Z direction : vecteurZ = np.cross(vecteurX,vecteurY) vecteurZ_unit = vecteurZ/np.sqrt(vecteurZ[0]**2 + vecteurZ[1]**2 + vecteurZ[2]**2) #Normal of the Z optical (the current Z direction) Zopitcal = np.array([0,0,1]) cos_theta = np.arccos(np.dot(vecteurZ_unit, Zopitcal)/np.sqrt(vecteurZ_unit[0]**2 + vecteurZ_unit[1]**2 + vecteurZ_unit[2]**2)*np.sqrt(Zopitcal[0]**2 + Zopitcal[1]**2 + Zopitcal[2]**2)) sin_theta = (np.cross(vecteurZ_unit, Zopitcal))[1] #Definition of the Rodrigues vector and use of cv2.Rodrigues to get rotation matrix v1 = Zopitcal v2 = vecteurZ_unit v_rodrigues = v1*cos_theta + (np.cross(v2,v1))*sin_theta + v2*(np.cross(v2,v1))*(1. - cos_theta) R = cv2.Rodrigues(v_rodrigues)[0] 

Answer 1

Your expected z direction is arbitrary to the reconstruction method. In general, you have a rotation matrix that rotates the left camera in the desired direction. You can easily build this matrix R. Then you just need to multiply your reconstructed points by transposing R.

Answer 2

To add an answer to fireant , here is one candidate’s solution, assuming that the expected X-direction is the same as the line connecting the projection centers of the two cameras.

• Calculate focal lengths f_1 and f_2 (using hole model calibration).
• Decide where camera 2 epipole is located in camera 1 frame. To do this, you can use either the main matrix (F) or the main matrix (E) of a pair of stereo cameras. In particular, the left and right epipoles lie in the zero space F, so you can use the singular decomposition of values . For a solid theoretical background, see Hartley and Zisserman, Second edition, Table 9.1 “Summary of Key Matrix Properties” on page 246 ( freely available PDF file of the chapter ).
• The projection center of camera 1, i.e. (0, 0, 0) and the location of the right epipole, i.e. (e_x, e_y, f_1) together define a beam that aligns with the line connecting the centers of the camera. This can be used as the expected X-direction. Call this vector v_x.
• Assuming that the expected Y axis is pointing down in the image plane, i.e., from (0, 0, f_1) to (0, height-1, f_1), where f is the focal length. Call this vector as v_y.
• The expected direction Z is now a cross product of the vectors v_x and v_y.
• Using the expected Z direction along with the optical axis (Z axis) of camera 1, you can then calculate the rotation matrix from two 3D vectors using, say, the method specified in fooobar.com/questions/1191268 / ....

Practical Note: The expectation that a flat object exactly aligns with a stereo base level is unlikely to be without much effort in my practical experience. A certain amount of planar and additional rotation is required.

One-time efforts: It depends on whether you need to do this once, for example. for a one-time calibration, in which case, just follow this real-time evaluation process, then rotate a pair of stereo cameras until the dispersion of the depth map is minimized. Then lock the camera and pray that someone will not hit it later.

Repeatability: If you need to maintain alignment of depth estimation maps with truly arbitrary Z-axes that change for each new captured frame, then you should consider investing in a plane estimation method and make it more reliable.