For a telecentric lens, the focal length approaches infinity. What you are seeing is numeric instability.
Halcon has dedicated calibration procedures for telecentric lenses that solve for magnification rather than focal length. There may be similar code base for Matlab / OpenCV.
It's also useful to consider that for a telecentric lens tilting the calibration grid, such that the grid is not perpendicular to optical axis, doesn't produce a perspective distortion. Therefore the tilt direction has two possible solutions that appear identical to the camera. The system of equations used must deal with that phenomena also.