How to find a camera matrix for augmented reality?

Question

I want to enlarge a virtual object using x, y, z meters across the camera. OpenCV has camera calibration functions, but I don’t understand how exactly I can give the coordinates in meters

I tried to simulate a camera in Unity, but did not get the expected result.

I set the projection matrix as follows and create a unit cube at z = 2.415 + 0.5. Where 2.415 is the distance between the eye and the projection plane (Pinhole camera model) Since the cubic face is on the front clipping plane, and its size should not cover the entire viewport?

Matrix4x4 m = new Matrix4x4(); m[0, 0] = 1; m[0, 1] = 0; m[0, 2] = 0; m[0, 3] = 0; m[1, 0] = 0; m[1, 1] = 1; m[1, 2] = 0; m[1, 3] = 0; m[2, 0] = 0; m[2, 1] = 0; m[2, 2] = -0.01f; m[2, 3] = 0; m[3, 0] = 0; m[3, 1] = 0; m[3, 2] = -2.415f; m[3, 3] = 0; 

Answer 1

I finished measuring the field of view manually. When you know FOV, you can easily create a projection matrix. No need to worry about units, because at the end the projection looks like (X * d / Z, Y * d / Z). Whatever the units of X, Y, Z, the X / Z ratio remains the same.

Answer 2

The global scale of your calibration (i.e., units of three-dimensional spatial coordinates) is determined by the geometry of the calibration object that you are using. For example, when you calibrate in OpenCV using images of a flat checkerboard, the inputs to the calibration procedure are the corresponding pairs (P, p) of three-dimensional points P and their images p, (X, Y, Z) the coordinates of the 3D points are expressed in mm, cm , inches, miles, regardless of what is required by the size of the used target (and the optics that displays it), and the 2D coordinates of the images are in pixels. The result of the calibration procedure is a set of parameters (components of the projection matrix P and nonlinear distortion parameters k) that "convert" the 3D coordinates expressed in these metric units into pixels.

If you do not know (or do not want to use) the actual dimensions of the calibration target, you can simply lure them out but leave their relationship unchanged (for example, a square remains a square, although the true length of its side may not be known). In this case, your calibration will be determined to an unknown global scale. This is actually a common case: in most virtual reality applications you don't care what the global scale is, as long as the results look right on the image.

For example, if you want to add an even more fluffy pair of three-dimensional lips to Angelina Jolie’s video and combine them with the original video so that the new fake lips remain attached and look “natural” on her face, you just need to rescale the 3D model of the fake lips so that she correctly coincided with the image of the lips. Regardless of whether the model is 1 yard or one mile from the CG camera in which you are creating the composition, it does not matter at all.

Answer 3

To find an additional object, you need to find the camera pose and orientation. This is the same as finding the exterior of the camera. You must also first calculate the internal properties of the camera (which is called calibraiton).

OpenCV allows you to do all this, but is not trivial, it requires work on your own. I give you the key, first you need to find out something on the stage that you know how it looks, so you can calculate the camera view by analyzing this object, call it a marker. You can start with typical fiducials, they are easy to spot.

Look at this thread .