C ++ / OpenCV: how to use BOWImgDescriptorExtractor to determine which clusters are associated with which images in a dictionary?

Question

My goal is to take the image as a query and find its best match in the image library. I use the SURF functions in openCV 3.0.0 and the Bag of Words approach to find a match. I need a way to find out if the request image has a match in the library. If so, I want to know the index of the image, which is the closest match.

Here is my code for reading in all images (300 in total in the image library) and extracting and clustering functions:

Mat training_descriptors(1, extractor->descriptorSize(), extractor->descriptorType()); //read in all images and set to binary char filepath[1000]; for (int i = 1; i < trainingSetSize; i++){ cout << "in for loop, iteration: " << i << endl; _snprintf_s(filepath, 100, "C:/Users/Randal/Desktop/TestCase1Training/%d.bmp", i); Mat temp = imread(filepath, CV_LOAD_IMAGE_GRAYSCALE); Mat tempBW; adaptiveThreshold(temp, tempBW, 255, ADAPTIVE_THRESH_GAUSSIAN_C, THRESH_BINARY, 11, 2); detector->detect(tempBW, keypoints1); extractor->compute(tempBW, keypoints1, descriptors1); training_descriptors.push_back(descriptors1); cout << "descriptors added" << endl; } cout << "Total descriptors: " << training_descriptors.rows << endl; trainer.add(training_descriptors); Ptr<DescriptorMatcher> matcher = DescriptorMatcher::create("FlannBased"); BOWImgDescriptorExtractor BOW(extractor, matcher); Mat library = trainer.cluster(); BOW.setVocabulary(library); 

I wrote the following code to find a match. The problem is that BOW.compute returns only the indexes of the clusters (words) that exist both in the image and in the image library. imgQ is a request image.

 Mat output; Mat imgQBW; adaptiveThreshold(imgQ, imgQBW, 255, ADAPTIVE_THRESH_GAUSSIAN_C, THRESH_BINARY, 11, 2); imshow("query image", imgQBW); detector->detect(imgQBW, keypoints2); extractor->compute(imgQBW, keypoints2, descriptors2); BOW.compute(imgQBW, keypoints1, output); cout << output.row(0) << endl; 

I need to know which clusters in BoW correspond to the images. My output right now - output.row (0) is just an array with all the cluster indexes found in the library. Do I really not understand this conclusion? Is there a way to determine which image has the most suitable clusters?

Answer 1

I also did something similar based on this code:

https://github.com/royshil/FoodcamClassifier/blob/master/training_common.cpp

But the above part after the completion of clustering. You need to train using your ML (I used SVM) and your cluster centers, a visual package of words that you have. Moreover, you need to find all the “closest” points to your clustered points and train them using histograms. Then you will have a frequency histogram (bag with key points) that you must train.

 Ptr<ifstream> ifs(new ifstream("training.txt")); int total_samples_in_file = 0; vector<string> classes_names; vector<string> lines; //read from the file - ifs and put into a vector for(int i=0;i<lines.size();i++) { vector<KeyPoint> keypoints; Mat response_hist; Mat img; string filepath; string line(lines[i]); istringstream iss(line); iss >> filepath; string class_to_train; iss >> class_to_train; class_ml = "class_" + class_to_train; if(class_ml.size() == 0) continue; img = imread(filepath); detector->detect(img,keypoints); bowide.compute(img, keypoints, response_hist); cout << "."; cout.flush(); //here create the logic for the class to train(class_0, eg) and the data you need to train. } 

You can find more details in this git project:
https://github.com/royshil/FoodcamClassifier
The documentation is here:
http://www.morethantechnical.com/2011/08/25/a-simple-object-classifier-with-bag-of-words-using-opencv-2-3-w-code/