3D to 2D matching

Question

I have point clouds (each point has a colour) of objects and images that show these objects. I want to find interest points in 2D/3D and match those so I know which parts of my image (at least those that had interest points) are found in the point cloud.

So I would need to find interest points first, get their descriptors and match them. If possible, this should work with current fast and memory conserving algorithms like BRISK or ORB (no patented algorithms!) from OpenCV. But I don't know how to implement them for 3D. Is this even possible? I found a paper (Hough Transform and 3D SURF for robust three dimensional classification) that talked about a 3D extension to SURF which would be a start but I can't find any info about that 3D extension. Even then, the question would be how feasible such an extension would be for BRISK or others current algorithms.

So please, give me advice on how to proceed.

Answer 1

It's called epipolar geometry and stereo matching.

1) You would need two images(2D) that you generated 3D point cloud from. 2) From those two images, you can create fundamental matrix and then generate epipolar points. Quite easy to do if you do it in MATLAB, not sure about OpenCV. 3) Those epipolar points from two separate images will draw lines to 3D world.

I suggest you to read about epipolar geometry and stereo matching for 2D -> 3D