Is it possible to obtain fronto-parallel view of image or camera position by 2d-3d points relation?

Question

Is it possible to obtain front-parallel view of image or camera position by 2d-3d points relation using OpenCV?

For this I have intrinsic and extrinsic parameters. I have also 3d coordinates of set of control points (which lies in one plane) on image (relation 2d-3d).

In fact I need location and orientation of camera, but it is not difficult to find it if I can convert image to fronto-parallel view.

If it is not possible to do with OpenCV, are the other libraries which can solve this task?

Answer 1

Solution is based on the formulas in the OpenCV documentation Camera Calibration and 3D Reconstruction

Let's consider numerical form without distortion coefficient (in contrast with matrix form).

We have u and v.
It is easy to calculate x' and y'.
But x and y can not be calculate because we can choose any non-zero z.
Line in 3d corresponds to one point in 2d image.

To solve this we take two points for z=1 and z=2. Then we find 2 points in 3d space which specify line (x1,y1,z1) and (x2,y2,z2).

Then we can apply R^-1 to (x1,y1,z1) and (x2,y2,z2) which results in line determined by two points (X1, Y1, Z1) and (X1, Y1, Z1).

Since our control points lie in one plane (let plane is Z=0 for simplicity) we can find corresponding X and Y point which is a point in 3d.

After applying normalization from mm to pixels we obtain fronto-parallel image.

(If we have input image distorted we should undistort it first)