Convertion Between cv::Mat and XnDepthPixel*

Question

i am working with OpenNI 1.5.4.0 and OpenCV 2.4.5, plus Qt for visualization purpose (only RGB images). Basically, i am retrieving the depth and rgb frames from a Kinect and store them on the hard drive, using the conversion:

/* Depth conversion */

cv::Mat depth = cv::Mat(2, sizes, CV_16UC1, (void*) pDepthMap); //XnDepthPixel *pDepthMap 

/* RGB conversion */

///image is a QImage* , pImageMap is a XnUInt8*

for(int i = 0; i < height; ++i)
{
    for (unsigned y = 0; y < height; ++y)
    {
       uchar *imagePtr = (*image).scanLine(y);
       for (unsigned x=0; x < width; ++x)
       {
          imagePtr[0] = pImageMap[2];
          imagePtr[1] = pImageMap[1];
          imagePtr[2] = pImageMap[0];
          imagePtr[3] = 0xff;
          imagePtr+=4;
          pImageMap+=3;
       }
    }
}

Now, i want to load those images from hard drive, in order to compute the 3D pointclouds as a post-processing computation. I am loading the Depth maps as:

depth_image = cv::imread(m_rgb_files.at(w).toStdString(), CV_LOAD_IMAGE_ANYDEPTH | CV_LOAD_IMAGE_ANYCOLOR );

but applying the formulas:

depth = depth_image.at<int>(j,i);  //depth_image is stored as 16bit
p.z = (float)depth * 0.001f;    //converting from millimeters to meters
p.x = (float)(i - m_params.center_x) * depth * m_params.focal_length; //focal_length read using OpenNI function
p.y = (float)(j - m_params.center_y) * depth * m_params.focal_length;

the pointcloud obtained is a mess.

If i do the "online" processing, using directly the native XnDepthPixel* data, the result is perfect, using the formulas written before. Can anybody give me a "hint" about my fault?

thanks in advance

EDIT: i was following also this resource, but it doesn't work for me, because XnDepthPixel contains the real world data in millimeters

Answer 1

I think there is a possible problem here:

depth = depth_image.at<int>(j,i);  //depth_image is stored as 16bit

If the depth image really is 16-bit, should it really be:

depth = depth_image.at<short>(j,i);  //depth_image is stored as 16bit

Because int is 32-bits, not 16.

Answer 2

Just as an addition, if you've built OpenCV with OpenNI support, you can retrieve the depth image as a cv::Mat which you can easily save with cv::imwrite Here's a minimal example:

#include "opencv2/core/core.hpp"
#include "opencv2/highgui/highgui.hpp"

#include <iostream>

using namespace cv;
using namespace std;

int main(){
    VideoCapture capture;
    capture.open(CV_CAP_OPENNI);

    if( !capture.isOpened() ){
        cout << "Can not open a capture object." << endl;
        return -1;
    }
    cout << "ready" << endl;

    for(;;){
        Mat depthMap,depthShow;
        if( !capture.grab() ){
            cout << "Can not grab images." << endl;
            return -1;
        }else{
            if( capture.retrieve( depthMap, CV_CAP_OPENNI_DEPTH_MAP ) ){
                const float scaleFactor = 0.05f;
                depthMap.convertTo( depthShow, CV_8UC1, scaleFactor );
                imshow("depth",depthShow);
                if( waitKey( 30 ) == 115)    {//s to save
                    imwrite("your_depth_naming_convention_here.png",depthShow);
                }
            }
        }
        if( waitKey( 30 ) == 27 )    break;//esc to exit
    }

}