OpenCV Rectification with too much black areas on images C++

Question

I'm using OpenCV to calibrate and rectify a stereo system. I have a stereo camera with convergent eyes, actually I'm running these functions on this order:

for(int j=0; j < ChessBoard.numSquares; j++)
    obj.push_back(Point3f((j/ChessBoard.numCornersHor)*ChessBoard.squareDim, (j%ChessBoard.numCornersHor)*ChessBoard.squareDim, 0.0f));
[...]

Then I loop this for the numbers of the images that I would like to acquire

found_L = findChessboardCorners(image_L, ChessBoard.board_sz, corners_L, CALIB_CB_ADAPTIVE_THRESH + CALIB_CB_NORMALIZE_IMAGE + CV_CALIB_CB_FILTER_QUADS + CALIB_CB_FAST_CHECK);
found_R= findChessboardCorners(image_R, ChessBoard.board_sz, corners_R, CALIB_CB_ADAPTIVE_THRESH + CALIB_CB_NORMALIZE_IMAGE + CV_CALIB_CB_FILTER_QUADS + CALIB_CB_FAST_CHECK);
found = found_L && found_R;
if(found)
  { 
    cornerSubPix(image_L, corners_L, Size(11, 11), Size(-1, -1), TermCriteria(CV_TERMCRIT_EPS + CV_TERMCRIT_ITER, 30, 0.1));
    cornerSubPix(image_R, corners_R, Size(11, 11), Size(-1, -1), TermCriteria(CV_TERMCRIT_EPS + CV_TERMCRIT_ITER, 30, 0.1));
    drawChessboardCorners(image_L, ChessBoard.board_sz, corners_L, found);
    drawChessboardCorners(image_R, ChessBoard.board_sz, corners_R, found);

    image_points[0].push_back(corners_L);
    image_points[1].push_back(corners_R);
    object_points.push_back(obj);
    printf("Right: coordinates stored\n");
    printf("Left: coordinates stored\n");
  }

After this block I call this two:

cameraMatrix[0] = Mat::eye(3, 3, CV_64F);
cameraMatrix[1] = Mat::eye(3, 3, CV_64F);

calibrateCamera(object_points, image_points[0], imageSize, cameraMatrix[0], distCoeffs[0], rvecs_L, tvecs_L);

calibrateCamera(object_points, image_points[1], imageSize, cameraMatrix[1], distCoeffs[1], rvecs_R, tvecs_R);

and then:

rms = stereoCalibrate(object_points, image_points[0], image_points[1],
                    cameraMatrix[0], distCoeffs[0],
                    cameraMatrix[1], distCoeffs[1],
                    imageSize, R, T, E, F,
                    TermCriteria(CV_TERMCRIT_ITER+CV_TERMCRIT_EPS, 100, 1e-5),
                    CV_CALIB_FIX_ASPECT_RATIO+CV_CALIB_FIX_INTRINSIC);

finally:

stereoRectify(cameraMatrix[0], distCoeffs[0],
                  cameraMatrix[1], distCoeffs[1],
                  imageSize, R, T, R1, R2, P1, P2, Q,
                  CALIB_ZERO_DISPARITY, -1, imageSize, &roi1, &roi2 );

initUndistortRectifyMap(cameraMatrix[0], distCoeffs[0], R1, P1, imageSize, CV_16SC2, map11, map12);
initUndistortRectifyMap(cameraMatrix[1], distCoeffs[1], R2, P2, imageSize, CV_16SC2, map21, map22);
remap(imgL, imgL, map11, map12, INTER_LINEAR,BORDER_CONSTANT, Scalar());
remap(imgR, imgR, map21, map22, INTER_LINEAR,BORDER_CONSTANT, Scalar());

This is basically what I'm doing, but the result is really bad, because the images have a really big black area. Here is an example:

and this is the rectify image that I have to obtain, that is taken rectified directly from the camera:

As you can see it seems that the image is translated on the right side and cut, with the right one is the same but is translated on the left side and the result is nearly the same.

So how can I achieve a better result similar to the last one? Where is the problem? As an additional data I have noticed that rms is not so good, about 0.4, and the re-projection error is about 0.2, I know that they must me a bit lower, but I have try many many times with different pattern, illumination and so on, in calibration but I always take the same result or even worst.

Answer 1

Try to call stereoRectify like this:

stereoRectify(cameraMatrix[0], distCoeffs[0],
              cameraMatrix[1], distCoeffs[1],
              imageSize, R, T, R1, R2, P1, P2, Q,
              0, -1, imageSize, &roi1, &roi2 );

i.e. use 0instead of the flag CALIB_ZERO_DISPARITY.

Also, in order to improve the RMS obtained by stereoCalibrate, try with flag CV_CALIB_USE_INTRINSIC_GUESS (see this related answer):

rms = stereoCalibrate(object_points, image_points[0], image_points[1],
                cameraMatrix[0], distCoeffs[0],
                cameraMatrix[1], distCoeffs[1],
                imageSize, R, T, E, F,
                TermCriteria(CV_TERMCRIT_ITER+CV_TERMCRIT_EPS, 100, 1e-5),
                CV_CALIB_USE_INTRINSIC_GUESS+
                    CV_CALIB_FIX_ASPECT_RATIO+CV_CALIB_FIX_INTRINSIC);