Using clustered indexes from a point cloud in an rgb image

Question

I work with a depth map obtained from two images (I took it from opencv StereoBM), and now I need to find clusters in them. I decided to use segment segmentation pcl http://www.pointclouds.org/documentation/tutorials/region_growing_segmentation.php . I converted cv :: Mat to a point cloud after reading this article http://blog.martinperis.com/2012/01/3d-reconstruction-with-opencv-and-point.html and now I have cluster indexes. These are functions here https://gist.github.com/Daiver/5586252 Now I want to use these indices to display clusters on a depth map with StereoBM (cv :: Mat)

I am trying to do this, but I am not satisfied with the results

pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloud; //cloud from depth map and rgb image std::vector <pcl::PointIndices> clusters;// clusters inices, extracted before for(int j = 0; j < clusters.size(); j++) { cv::Mat to_show = cv::Mat::zeros(288, 384, cv::DataType<uchar>::type);//image has size that equal size of rgb image and depth map for(int i = 0; i < clusters[j].indices.size(); i++) { to_show.data[clusters[j].indices[i]] = 200;// regions in this Mat must be equal with regions from depth map } cv::imshow("", to_show); cv::waitKey(); } 

Result Some Clusters Another cluster

Visualized cloud

How can I design clusters in cv :: Mat? PS sorry for my written mistakes. English in my native language

UPD I tried to “restore” the depth map using loops similar to loops in the mat_to_cloud function.

 int counter = 0; cv::Mat to_show = cv::Mat::zeros(288, 384, cv::DataType<uchar>::type); for(int i = 0; i < cloud->height; i++) { for(int j = 0; j < cloud->width; j++) { to_show.at<uchar>(i, j) = cloud->at(counter).z; counter++; } } 

And one more order of cycles int counter = 0; cv :: Mat to_show = cv :: Mat :: zeros (288, 384, cv :: DataType :: type); for (int j = 0; j <cloud-> width; j ++) {for (int i = 0; i <cloud-> height; i ++) {to_show.at (i, j) = cloud-> at (counter) .z; Counter ++; }}

I do not know why this image looks like

Answer 1

I solved my problem, but my solution is a bit dirty and stupid. I made my own simple reprogramming function

 void straight_reproject_cloud(cv::Mat& img_rgb, cv::Mat& img_disparity, pcl::PointCloud<pcl::PointXYZRGB>::Ptr& point_cloud_ptr) { uchar pr, pg, pb; for (int i = 0; i < img_rgb.rows; i++) { uchar* rgb_ptr = img_rgb.ptr<uchar>(i); uchar* disp_ptr = img_disparity.ptr<uchar>(i); for (int j = 0; j < img_rgb.cols; j++) { uchar d = disp_ptr[j]; if ( d == 0 ) continue; //Discard bad pixels pb = rgb_ptr[3*j]; pg = rgb_ptr[3*j+1]; pr = rgb_ptr[3*j+2]; //Insert info into point cloud structure pcl::PointXYZRGB point; point.x = j; point.y = i; point.z = d; uint32_t rgb = (static_cast<uint32_t>(pr) << 16 | static_cast<uint32_t>(pg) << 8 | static_cast<uint32_t>(pb)); point.rgb = *reinterpret_cast<float*>(&rgb); point_cloud_ptr->push_back (point); } } } 

This function adds points to the cloud directly from images without changes. The new cloud does not match the old reprogramming, but I can work with it.

And now the coordinate points in the cloud are consistent with the coordinates in the images. I can show all the clusters from the cloud in the image:

 for(int k = 0; k < clusters.size(); k++) { cv::Mat res = cv::Mat::zeros(img_rgb.rows, img_rgb.cols, CV_8U); //for(int i =0 ; i < point_cloud_ptr->points.size(); i++) for(int j =0 ; j < clusters[k].indices.size(); j++) { int i = clusters[k].indices[j]; int x = point_cloud_ptr->at(i).x; int y = point_cloud_ptr->at(i).y; res.at<uchar>(y, x) = (int)(point_cloud_ptr->at(i).z); } cv::imshow("rec2", res); cv::waitKey(); } 

New cloud

Answer 2

I have not worked with PCL before, but it looks like this line might be wrong:

 // regions in this Mat must be equal with regions from depth map to_show.data[clusters[j].indices[i]] = 200; 

to_show is an opencv matrix, but you are using indexes from a point cloud. First you need to convert the indexes to pixel coordinates.