Tracking multiple moving objects with KalmanFilter in OpenCV / C ++ - How to assign tracks to individually detected objects

Question

I'm working on real-time motion detection and object tracking in OpenCV / C ++, and I'm currently stuck in the tracking part.

Matlab is an example of what I want to do: http://www.mathworks.de/de/help/vision/examples/motion-based-multiple-object-tracking.html (I am concerned about the tracking part and how to transfer it to C ++ / OpenCV)

My part of the movement works with OpenCVs BackgroundSubtractor MOG2, which allows me to find outlines and filter out smaller outlines.

For tracking, I'm currently trying to use KalmanFilter (with a similar implementation of this ), which is now called in every frame if a moving object was found and draws a line along the way. My detection and tracking part looks something like this:

BackgroundSubtractorMOG2 bg; bg.operator()(frame, threshold); bg.getBackgroundImage(background); ... //morphological operations to remove noise etc. findContours(threshold, ...); ... //filtering to reject contours which are too smalle/too big for(int i = 0; i < contours.size(); i++){ approxPolyDP(...); boundRect = boundingRect(...); x = boundRect.x + boundRect.width/2; y = boundRect.y + boundRect.height/2;} kalmanFilter.track(x,y); kalmanFilter.draw(frame); 

Current problem: I have a scene with 0 moving objects, then 1 object is moved, detected through the contours and tracked. Then the 2nd object moves into the field of view, it is detected and makes the tracker jump on it instead of following the first or marking as individually (which I need).

The current tracker takes the x and y coordinates of the found object. Similarly, when another object is detected, the tracker still assumes that it is the same object , but with different coordinates than expected.

As you can see, there is no function for assigning “tracks” to a specific object , which is probably the biggest problem. I read about the Hungarian algorithm, but am not quite sure how to implement it in my functions.

What would be a good way to get work to track for multiple objects?

My idea was that if I had each object uniquely identified , I could check if the ID remains the same, and if not , let the tracker know that this is a new object to track separately from another. Not sure if this is necessary or even useful, although if so, how to do it.

Answer 1

try something like this:

 for each contour: if its already tracked with ID 'A': kalmanFilter_with_id_A.track(x,y); else createNewKalmanFilterWithID A 

you need some kind of mechanism to decide if it is already being tracked. In simple tracking, you simply decide by measuring the distances to the contours in the last frame, and if it's close enough, then this is an old object. This is pretty wrong, so you might want to take a look at the best tracking methods, for example. probabilistic tracking.

such a simple mode:

 for each contour 'currentFrameC': for each contour 'lastFrameC' if distance(currentFrameC, lastFrameC) is smallest and < threshold currentFrameC is the same object as lastFrameC so give it the same ID if no shortest contour with dist < thres was found, create a new object with a new ID and create a new KalmanFilter with this same ID for that object call kalmanFilter with ID for each found contour ID 

Answer 2

As Mikka suggested, you have your own algorithm to distinguish tracked_object from new_detection. The simplest solution is to compare the distance between the centroids.

Other applications include

• Correlation of overlapping cells with codes
• Hungarian algorithm

(I provided links to code and articles at the end.)

---

BoundingBoxOverlapRatio Method:

Steps

• Frame_1: Define blobs -> Draw boundingRectangles for each blob bbox_frame1_obj1 , bbox_frame1_obj2 ...

• Frame_2: Identify blobs -> Draw boundingRectangles bbox_frame2_obj1

  -> compare the overlap of bbox_frame2_obj1 with bbox_frame1_obj1 , bbox_frame1_obj2 , etc.

The number of overlaps indicates the likelihood of being the same object, depending on which positions of the object can be updated.

But this method is suitable for scenarios that include slow moving objects, such as pedestrian detection, or possibly not overlapping at all.

---

Hungarian algorithm

This algorithm from 5 to 6 steps serves as a simple but reliable solution for this kind of assignment tasks.

It works as shown below.

Create a matrix with rows as prev_position objects and columns as new_detections

If we have 3 objects in memory and the new frame has 4 drops, this leads to a 3x4 matrix with each element being the distance between the objects and the detections.

Element (1,1) → the distance between the centroid obj_1 and the detected detection_1

Element (1,2) → the distance between the centroid obj_1 and the detected detection_2 ...

Example matrix (units in pixels):

 5 15 30 20 20 30 3 40 12 4 15 50 

The situation here is simple and straightforward, which leads to the conclusion

 obj_1 -> detection_1 (5px between obj_1 and detection_1) obj_2 -> detection_3 (3px between obj_2 and detection_3) obj_3 -> detection_2 (4px between obj_3 and detection_2) 

Thus, detect_4 must be a new object, and a new track and identifier must be assigned.

After that, obj_4 should be tracked together, and the next matrix has 4 rows ...

But with more objects, the task gets more lumps. And the Hungarian algorithm helps solve more complex and huge matrices for optimal assignment of detections to objects.

---

Below are some useful resources for tracking multiple objects and the Hungarian algorithm.

• Matlab Documentation

  • Pedestrian Detector - Provides a neat, step-by-step approach for handling tracking of multiple objects.

• C ++ implementation of MATLABs Multi-Object tracker (includes Hungarian algorithm)

  • Youtube

  • Github

• BoundingBoxOverlapRatio Method:

  • Matlab documentation . C ++ implementation should be direct.

• Hungarian algorithm:

  • Explanation and Code
  • Simple explanation and steps to solve