When doing matching there are always some refinement steps for getting rid out of outliers.
What I usually do is discarding matches that have a distance over a threshold, for example:
for (int i = 0; i < matches.size(); i++ )
{
if(matches[i].distance > 200)
{
matches.erase(matches.begin()+i-1);
}
}
Then, I use RANSAC to see which matches fit the homography model. OpenCV has a function for this:
for( int i = 0; i < matches.size(); i++ )
{
trainMatches.push_back( cv::Point2f(keypointsB[ matches[i].trainIdx ].pt.x/500.0f, keypointsB[ matches[i].trainIdx ].pt.y/500.0f) );
queryMatches.push_back( cv::Point2f(keypointsA[ matches[i].queryIdx ].pt.x/500.0f, keypointsA[ matches[i].queryIdx ].pt.y/500.0f) );
}
Mat h = cv::findHomography(trainMatches,queryMatches,CV_RANSAC,0.005, status);
And I just draw the inliers:
for(size_t i = 0; i < queryMatches.size(); i++)
{
if(status.at<char>(i) != 0)
{
inliers.push_back(matches[i]);
}
}
Mat imgMatch;
drawMatches(imgA, keypointsA, imgB, keypointsB, inliers, imgMatch);
Just try different thresholds and distances until you get the desired resutls.