Getting x,y in 3d coordinate system from 2d image pixel

Question

Using camera with fixed position I have to track an object in motion and to localize it in some pre chosen coordinate system. Idea is to put a specific color marker on the top of the object with known height, so the Z coordinate will be known. After color tracking is done, I need to find out the X and Y from u and v image coordinates where the color is found. I have already done camera calibration and I'm planning to use cv::solvePnP to get rotation and translation vector. Also, I have learned about color tracking here and wrote a C++ api from it.

I have already found a similar question here and learned a lot from it, but the difference is that I only need to evaluate X and Y, since the Z coordinate is known (measured before). How can I get those X and Y coordinates with pre specified Z? Thanks for help!

Answer 1

Since you have calibrated (i presume on z=0) you can go from pixel to rw coordinates using the camera, translation and rotation matrices.

1. You find a point you want to know the rw coordinates of. (the color marker)
2. You lookup the x and y coordinate of this point (assuming z=0)
3. You calculate the angles in x and y (in relation to the principle point)
4. You use these angles to calculate what x and y would be at z=objectHeight

Answer 2

After you calibrated and obtained extrinsic and intrinsic parameters, you can use the informations about

• u,v coordinates
• z coordinate

to find X e Y in world coordinates:

• Pc = R * Pw + T

Where Pc cam coordinate, Pw world coordinate, R rotation matrix and T translation vector.