opencv stereo calibration -> warp camera image A into camera image B

Question

I successfully calibrated intrinsics and extrinsics using opencv for 2 cameras (a low res cam on the left and a high res cam on the right). I stereo calibrated from 2 sets of images (chessboard pattern- approx 20 images from each camera). I now have a left.yml, right.yml, and a translationRotation.yml. Now I want to warp/distort the right image so it appears the same (perspective/distortion/etc) as the left image. I know I have all the data from the calibration, it's just knowing how to use with the right opencv function/s to do the operation- eg. warpPerspective, remap, or something else?

Background info- I'm basically replacing a low resolution rgb camera feed (left), with a high resolution rgb camera feed (right). Using Openframeworks/c++ with ofxCv.

Thanks.

Answer 1

If i understand right, you have different distortions on each camera, and you want to undistort one of them and apply the other camera's distortion. For the undistortion part here's what I've done in the past.

Using OF, on setup you can create the distortion/undistortion maps, using cv::initUndistortRectifyMap, that will give you two maps in the form of cv::Mat objects. This function receives the camera matrix and distortion coefficients from the calibration.

Then you can apply the mapping using cv::remap. Which will store the result in its second parameter.

After undistorting you'd want to apply the inverse mapping, for applying the other camera's distortion. It seems the warpPerspective function you mentioned would work well for that, but a mapping could probably be computed from the .yml files you got. It's just a matter of applying the opposite mapping.

Use ofxCV's wrapping to switch from ofImage to Mat whenever it's necessary to make sure you don't make unnecessary copies.