Accurate Matrix Definition in OpenCv StereoRectify

Question

Typically, the definition of the projection matrix P is a 3x4 matrix that projects a point from world coordinates into image / pixel coordinates. The projection matrix can be divided into:

• K : 3x4 K camera matrix with internal parameters
• T : 4x4 transformation matrix with external parameters

Then the projection matrix P = K * T.

What are the clear definitions of the following input to OpenCV stereoRectify:

• cameraMatrix1 - The first camera matrix (I believe this is the desired part K of the projection matrix, right?) .
• R is the rotation matrix between the coordinate systems of the first and second cameras. (what does “between” mean? Is it rotation from cam1 to cam2 or from cam2 to cam1?)
• T - vector translation between camera coordinate systems. (Same thing above. Is the translation from cam1 → cam2 or cam2-> cam1)
• R1 is the output 3x3 rectification conversion (rotation matrix) for the first camera. (This is a rotation after correction, so the new outer part of the projection matrix becomes T1new = R1 * T1old?)
• P1 is the output 3x4 projection matrix in the new (fixed) coordinate systems for the first camera. (What is meant by the “projection matrix in the new coordinate system”? It seems that this projection matrix depends on the rotation matrix R1 from the projection from the world coordinates to the image / pixel coordinates, therefore from the above definition neither the “projection matrix” nor the “camera matrix” ", and some mixture of two)

Answer 1

• CAMERAMATRIX1 is the K internal matrix computed by the stereo calibration function () in opencv. you got it right!
• R is the rotation matrix of the frame cam2 frame.rt cam1. Similarly, T is the translation vector of the beginning cam2 wrt cam1 initial.
• If you look in O'Riley’s book “LEARNING OPENCV” p. 434, you will see that R1 (/ Rl ) and R2 (/ Rr ).
  
  Rl = [ dy correct ] [ gh ]; Rr = [ Rect ] [ p ];
  
  let the image planes of the camera be plane 1 and plane 2. When stereo rectification has not been performed, plane 1 and plane 2 will not be parallel at all. In addition, epilines will not be parallel to the baseline of the stereo camera. So what Rl does is that it converts the plane of the left image parallel to the plane of the right image (which is converted via Rr), and the epilines in both images are now parallel.
• P1 and P2 are new projection matrices after stereo rectification. Remember that the camera matrix (K) converts a point in 3d space to a 2d image plane. But P1 and P2 transform a point in 3d space into straightened 2d image planes.
  if you calibrated a previously installed stereo camera and watched the values ​​of P1 and K1, you will find that they are quite similar to each other if your stereo setup is almost configured (obviously within the human range).

---