How to process missing data in the structure from motion optimization / package settings

Question

I work on the structure from the motion application, and I track the number of markers placed on the object to determine the rigid structure of the object.

The application essentially uses the standard Levenberg-Marquardt optimization to view multiple cameras and minimizes the differences between the expected marker points and marker points obtained in 2D from each view.

For each marker point and each view, the following function is minimized:

double diff = calculatedXY[index] - observedXY[index] 

Where the calculated value of XY depends on a number of unknown parameters that need to be found using optimization, and XY is observed - this is the position of the marker in 2D. In general, I have (marker points * views) a number of functions similar to the one I'm trying to minimize.

I encoded the camera simulation, seeing all the points of the marker, but I was wondering how to deal with situations where during the launch the points are not visible due to lighting, occlusion or simply are not in the camera viewing mode. When the application launches, I will use a webcam to view the object, so it’s likely that not all markers will be visible right away and depending on how reliable my computer vision algorithm is, I may not be able to detect the marker all the time.

I thought that the diff value should be 0 (sigma-squared difference = 0) in the case where the marker point cannot be detected, can this distort the results?

Another thing that I noticed is that the algorithm is not so good when it is presented with too many views. It is more likely to evaluate a bad decision when too many views are submitted. Is this a common problem with setting up the bundle due to the increased probability of getting to a local minimum when presenting too many views?

Answer 1

It is common practice to simply exclude terms that correspond to missing markers. I.e. do not try to minimize calculateXY-observedXY if there is no member observedXY . There is no need to set anything to zero, you should not even consider this term in the first place - just skip it (or, I think, in your code this is equivalent to setting the error to zero).

Correction of the ligament can greatly fail if you simply throw a large number of observations on it. Build your solution gradually by allowing multiple views first and then keep adding.

You might want to try some sort of “reliable” approach. Instead of using the least squares, use the loss function 1 . This allows your optimization to survive, even if there are a few incorrect comments. You can still do this within Levenberg-Marquardt, you just need to include the derivative of your Jacobian loss function.