OpenCV (C++) using SIFT descriptors increases the number of detected features?

Question

I have a bit confusing situation while using the SIFT descriptors implementation from OpenCV.

I'm trying to test various feature detector + descriptor calculation methods, so I am using a combination of cv::FeatureDetector and cv::DescriptorExtractor interfaces which allow me to simply change between different detector methods and descriptors.

When calling cv::DescriptorExtractor::compute(...) (the variant for a single image), the documentation says that it is possible for the number of key points given to the algorithm to decrease if it is impossible to calculate their descriptors, and I understand how and why that is done.

But, what happens to me is that the number of key points after the descriptor computations actually increases. It is clearly so, and I'm not trying to stop it from happening, I am just hoping for an explanation as to why (just an intuitive description would be cool, altough I appreciate anything more that that).

I've got layers upon layers of wrappers around the actual OpenCV that don't have any code (just setting up some local non-OpenCV flags), so here's the OpenCV code that's being called at the bottom of it all:

cv::Ptr<cv::FeatureDetector> dect = cv::FeatureDetector::create("MSER");
cv::Mat input = cv::imread("someImg.ppm", 0);
std::vector<cv::KeyPoint> keypoints;
dect->detect(input, keypoints);

cv::Ptr<cv::DescriptorExtractor>deEx=cv::DescriptorCalculator::create("SIFT");

std::cout << "before computing, feats size " << keypoints.size() << std::endl;
// code to print out 10 features

cv::Mat desc;
deEx->compute(input, keypoints, desc);

std::cout << "after computing, feats size " << keypoints.size() << std::endl;
// code to print out 10 features

I've printed out the first 10 key points just before and after the descriptor calculations, so here are some concrete numbers as an example:

before computing, feats size 379
feat[0]: 10.7584 39.9262 176.526 0 12.5396
feat[1]: 48.2209 207.904 275.091 0 11.1319
feat[2]: 160.894 313.781 170.278 0 9.63786
feat[3]: 166.061 239.115 158.33 0 19.5027
feat[4]: 150.043 233.088 171.887 0 11.9569
feat[5]: 262.323 322.173 188.103 0 8.65429
feat[6]: 189.501 183.462 177.396 0 12.3069
feat[7]: 218.135 253.027 171.763 0 123.069
feat[8]: 234.508 353.236 173.281 0 11.8375
feat[9]: 234.404 394.079 176.23 0 8.99652
after computing, feats size 463
feat[0]: 10.7584 39.9262 13.1313 0 12.5396
feat[1]: 48.2209 207.904 69.0472 0 11.1319
feat[2]: 48.2209 207.904 107.438 0 11.1319
feat[3]: 160.894 313.781 9.57937 0 9.63786
feat[4]: 166.061 239.115 166.144 0 19.5027
feat[5]: 150.043 233.088 78.8696 0 11.9569
feat[6]: 262.323 322.173 167.259 0 8.65429
feat[7]: 189.501 183.462 -1.49394 0 12.3069
feat[8]: 218.135 253.027 -117.067 3 123.069
feat[9]: 218.135 253.027 7.44055 3 123.069

I can see from this example that the original feat[1] and feat[7] have spanned in to two new key points each, but I do not see any logical explanation for the compute method to do that :(

The printout I have given here is from using MSER for detection of keypoints, and then trying to calculate SIFT descriptors, but the same increase in size also happens with STAR, SURF, and SIFT (i.e. DoG) keypoints detected. I didn't try to change the SIFT descriptor in to something else, but if someone thinks it's relevant to the question, I'll try it and edit it in my question.

Answer 1

Looks like it is due to OpenCV using Rob Hess's SIFT implementation, which sometimes duplicates the keypoints with more than one dominant orientation.

Looking around the OpenCV reported bugs did the trick, the issue was reported here.

It is not a bug, the behavior was not corrected in the newer versions but instead just documented. Since I am obliged to the OpenCV version I am using right now (v2.1), it did not occur to me to look at newer documentation for additional behavior since the behavior described in the old one made sense to me.

Answer 2

First of all, as you can see in the documentation cv::DescriptorExtractor::compute take a std::vector<cv::Keypoints> in argument which in non const. It means that this vector can be modified by cv::DescriptorExtractor::compute. In practice, KeyPointsFilter::runByImageBorder and KeyPointsFilter::runByKeypointSize (two non-const functions) will be applied to the vector and will remove the keypoint for which a descriptor cannot be computed. No re-extraction of keypoints will be done. You should post the few lines of code you are using for further diagnostic.

--

Well, I finally found where the problem occurs: cv::SiftDescriptorExtractor::compute method calls SIFT::operator() which (re)calculate orientation of features and also duplicates the points with several dominant orientations. Solution could be to change descriptorParams.recalculateAngles to false.

Answer 3

This is not a bug but by design:

SIFT returns multiple interest points at the same location with different orientations if there is not clearly a single dominant orientation. Usually, up to three (depending on the actual image patch) orientations are estimated.