How to evaluate Feature descriptors with a Matching Algorithm

Question

I'm trying to evaluate Feature Detectors and Descriptors with the FLANN algorithm based on this tutorial

I want to build a ROC curve for the evaluation part therefore I have to get the TP, FN, FP and TN. The thing is, I don't know how to get these values! I have read a lot of papers but none of them explain, for instance how they get the false positives. In the given tutorial you can set a certain threshold such that you can count the good and the bad matches, but thats not a justification that everthing was matched correctly. Should I count it for every image pairs by hand or is their a common technique do solve it automatically.

Thanks in advance for any help!

Answer 1

You have to have so called "ground truth" - manually checked correspondences or transformation matrix (fundamental or homography) between two images. Correspondences which are consistent with this matrix are correct.

Check approach used in classical papers by Mykolajczyk et al. "A comparison of affine region detectors", "A PERFORMANCE EVALUATION OF LOCAL DESCRIPTORS" and Moreels and Perona "Evaluation of Features Detectors and Descriptors based on 3D Objects"