Using opencv to create a panoramic image from 3d points

Question

I have a PointGrey Ladybug3 Camera. It's a panoramic (multi)camera (5 camera to do a 360º and 1 camera looking up). I've done all the calibration and rectification so what I end up is from all pixels of the 6 images I know it's 3d position wrt a global frame. What I would do now is convert this 3d points to a panoramic image. The most common is a radial (Equirectangular) projection like the following one:

For all the 3D points (X,Y,Z) it's possible to find theta and phi coordinate like:

My question is, Is it possible to do this automatically with opencv? Or if I do this manually what is the best way to convert that bunch of pixels in theta,phi coordinates to an image?

The official ladybug SDK uses OpenGL for all this operations, but I was wondering if it's possible to do this in opencv.

Thanks,

Josep

Answer 1

The approach I used to solve this problem was the following:

1. Create an empty image with the desired output size.
2. For every pixel in the output image find the theta and phi coordinates. (Linearly) Theta goes from -Pi to Pi and phi from 0 to Pi
3. Set a projection radius R and find 3D coordinate from theta, phi and R.
4. Find for how many cameras is the 3D point visible and the correspondent pixel position.
5. Copy the pixel of the image where the pixel is closer to the principal point. Or any other valid criteria...

My code looks like:

cv::Mat panoramic;
panoramic=cv::Mat::zeros(PANO_HEIGHT,PANO_WIDTH,CV_8UC3);
double theta, phi;
double R=calibration.getSphereRadius();
int result;

double dRow=0;
double dCol=0;

for(int y = 0; y!= PANO_HEIGHT; y++){
    for(int x = 0; x !=PANO_WIDTH ; x++) {
            //Rescale to [-pi, pi]
        theta=-(2*PI*x/(PANO_WIDTH-1)-PI); //Sign change needed.
            phi=PI*y/(PANO_HEIGHT-1);

        //From theta and phi find the 3D coordinates.
        double globalZ=R*cos(phi);
            double globalX=R*sin(phi)*cos(theta);
        double globalY=R*sin(phi)*sin(theta);


        float minDistanceCenter=5000; // Doesn't depend on the image.

        float distanceCenter;

        //From the 3D coordinates, find in how many camera falls the point!
        for(int cam = 0; cam!= 6; cam++){
            result=calibration.ladybugXYZtoRC(globalX, globalY, globalZ, cam, dRow, dCol);
            if (result==0){ //The 3d point is visible from this camera
                cv::Vec3b intensity = image[cam].at<cv::Vec3b>(dRow,dCol);
                distanceCenter=sqrt(pow(dRow-imageHeight/2,2)+pow(dCol-imageWidth/2,2));
                if (distanceCenter<minDistanceCenter) {
                    panoramic.ptr<unsigned char>(y,x)[0]=intensity.val[0];
                    panoramic.ptr<unsigned char>(y,x)[1]=intensity.val[1];
                    panoramic.ptr<unsigned char>(y,x)[2]=intensity.val[2];

                    minDistanceCenter=distanceCenter;
                }
            }
        }

    }
}