two different results from one object by OpenCV's camera calibration

Question

I am using a 8x8 chessboard to calibrate my camera with OpenCV; but I have some curious result:

I try to calibrate the camera with two photos of the chessboard, for one of them, undistorted photo is correct, but the corners of the other one are stretched:

Can anyone tell me why I'm getting these curious results? And how I can solve this problem?

Answer 1

Try using more images, with the board in different orientations. I would use at least 10. Also, be sure to move the board around to get points all over the field of view.

Answer 2

The first image: it is almost parallel to the image plane. In this case the vanishing points in the image are poorly observed and the constraints on the camera intrinsics are vague as a consequence.

Camera calibration from vanishing points in images of architectural scenes, BMVC (1999), by R Cipolla, T Drummond, D Robertson

Also, the distortion correction is wrong only in areas where there is no observed data (along the boundaries of the image). This tells us that the calibrated model is bad at extrapolation but good at interpolation. This is a classic example of over-fitting.

My guess is that in the second image, the intrinsics are better constrained because of better depth variance. The constraints on the intrinsics allow better estimation of the distortion using the available data.