SIFT matching gives very poor results

Question

I'm working on a project where I will use homography as features in a classifier. My problem is in automatically calculating homographies, i'm using SIFT descriptors to find the points between the two images on which to calculate homography but SIFT are giving me very poor results, hence i can't use them in my work.

I'm using OpenCV 2.4.3.

At first I was using SURF, but I had similar results and I decided to use SIFT which are slower but more precise. My first guess was that the image resolution in my dataset was too low but i ran my algorithm on a state-of-the-art dataset (Pointing 04) and I obtained pretty much the same results, so the problem lies in what I do and not in my dataset.

The match between the SIFT keypoints found in each image is done with the FlannBased matcher, i tried the BruteForce one but the results were again pretty much the same.

This is an example of the match I found (image from Pointing 04 dataset)

The above image shows how poor is the match found with my program. Only 1 point is a correct match. I need (at least) 4 correct matches for what I have to do.

Here is the code i use:

This is the function that extracts SIFT descriptors from each image

void extract_sift(const Mat &img, vector<KeyPoint> &keypoints, Mat &descriptors, Rect* face_rec) {

        // Create masks for ROI on the original image
    Mat mask1 = Mat::zeros(img.size(), CV_8U);  // type of mask is CV_8U

    Mat roi1(mask1, *face_rec);
    roi1 = Scalar(255, 255, 255);

    // Extracts keypoints in ROIs only
    Ptr<DescriptorExtractor> featExtractor;
    Ptr<FeatureDetector> featDetector;
    Ptr<DescriptorMatcher> featMatcher;

    featExtractor = new SIFT(); 
    featDetector = FeatureDetector::create("SIFT"); 

    featDetector->detect(img,keypoints,mask1); 
    featExtractor->compute(img,keypoints,descriptors); 

}

This is the function that matches two images' descriptors

void match_sift(const Mat &img1, const Mat &img2, const vector<KeyPoint> &kp1,
            const vector<KeyPoint> &kp2, const Mat &descriptors1, const Mat &descriptors2,
            vector<Point2f> &p_im1, vector<Point2f> &p_im2) {

//  Matching descriptor vectors using FLANN matcher
Ptr<DescriptorMatcher> matcher = DescriptorMatcher::create("FlannBased");
std::vector< DMatch > matches;
matcher->match( descriptors1, descriptors2, matches );

double max_dist = 0; double min_dist = 100;

// Quick calculation of max and min distances between keypoints
for( int i = 0; i < descriptors1.rows; ++i ){
    double dist = matches[i].distance;
    if( dist < min_dist ) min_dist = dist;
    if( dist > max_dist ) max_dist = dist;
 }

 // Draw only the 4 best matches
 std::vector< DMatch > good_matches;

 // XXX: DMatch has no sort method, maybe a more efficent min extraction algorithm can be used here?
 double min=matches[0].distance;
 int min_i = 0;
 for( int i = 0; i < (matches.size()>4?4:matches.size()); ++i ) {
     for(int j=0;j<matches.size();++j)
         if(matches[j].distance < min) {
            min = matches[j].distance;
            min_i = j;
         }
    good_matches.push_back( matches[min_i]);
    matches.erase(matches.begin() + min_i);
    min=matches[0].distance;
    min_i = 0;
 }

  Mat img_matches;
  drawMatches( img1, kp1, img2, kp2,
               good_matches, img_matches, Scalar::all(-1), Scalar::all(-1),
               vector<char>(), DrawMatchesFlags::NOT_DRAW_SINGLE_POINTS );
  imwrite("imgMatch.jpeg",img_matches);
  imshow("",img_matches);
  waitKey();

  for( int i = 0; i < good_matches.size(); i++ )
  {
    // Get the points from the best matches
    p_im1.push_back( kp1[ good_matches[i].queryIdx ].pt );
    p_im2.push_back( kp2[ good_matches[i].trainIdx ].pt );
  }
}

And these functions are called here:

extract_sift(dataset[i].img,dataset[i].keypoints,dataset[i].descriptors,face_rec);

[...]

//  Extract keypoints from i+1 image and calculate homography
    extract_sift(dataset[i+1].img,dataset[i+1].keypoints,dataset[i+1].descriptors,face_rec);
    dataset[front].points_r.clear(); // XXX: dunno if clearing the points every time is the best way to do it..
    match_sift(dataset[front].img,dataset[i+1].img,dataset[front].keypoints,dataset[i+1].keypoints,
        dataset[front].descriptors,dataset[i+1].descriptors,dataset[front].points_r,dataset[i+1].points_r);

    dataset[i+1].H = findHomography(dataset[front].points_r,dataset[i+1].points_r, RANSAC);

Any help on how to improve the matching performance would be really appreciated, thanks.

Answer 1

You apparently use the "best four points" in your code w.r.t. the distance of the matches. In other words, you consider that a match is valid if both descriptors are really similar. I believe this is wrong. Did you try to draw all of the matches? Many of them should be wrong, but many should be good as well.

The distance of a match just tells how similar both points are. This doesn't tell if the match is coherent geometrically. Selecting the best matches should definitely consider the geometry.

Here is how I would do:

1. Detect the corners (you already do this)
2. Find the matches (you already do this)
3. Try to find a homography transform between both images by using the matches (don't filter them before!) using findHomography(...)
4. findHomography(...) will tell you which are the inliers. Those are your good_matches.