Lucas Canada Optical Stream, Direction Vector

Question

I am working on optical flow and based on the lectures here and some samples on the Internet, I wrote this Python code .

All code and sample images are also there. At small offsets of about 4-5 pixels, the direction of the vector, calculated, seems to be fine, but the magnitude of the vector is too small (so I had to multiply u, v by 3 before their graphics).

Is it due to algorithm limitations or code errors? The note to the lecture above also says that the movement should be small, “u, v is less than 1 pixel,” which may be why. What is the reason for this limitation?

Answer 1

@belisarius says: "LK uses a first-order approximation, and therefore (u, v) should be ideally <1, if not, higher-order members dominate behavior, and you are toasts."

Answer 2

The standard conclusion from the optical flow restriction equation (OFCE, slide 5 of your reference) is that "your movement should be less than a pixel, the less they kill you." Although technically true, you can overcome this in practice by using large, medium-sized windows. This requires you to perform normal statistics, i.e. Not purely the smallest square value as suggested in the slides. Similarly, faster calculations and better results can be achieved by regularizing Tikhonov. This requires setting the setting value (Tikhonov constant). This can be done as a global constant, or allowed to be adjusted for local information in the image (for example, Shi-Tomashi trust, the so-called determinant of the structure tensor).

Note that this does not replace the need for multiscale approaches to deal with large movements. It can slightly expand the range that any single scale can handle.

Implementations, visualizations, and code are available in the tutorial format here, although in Matlab there is no Python.