Machine learning How to compare different feature sets

Question

So let's say I have two different sets of features A and B. I'm trying to determine which set of features is the best. I'm using leave-one-out cross validation as the final metric as my data set is small. I'm trying to figure out my experimental setup and I'm deciding between a few ways:

1) Give feature set A to my classifer (and optionally run feature selection) on it and give feature set B to the same classifier (also optionally run feature selection) and then compare the LOOCV error between these 2 classifiers?

2) Give feature set A AND B to the classifier and then definitely run feature selection it and then draw a higher level conclusion based on the features selected. (e.g. if more from A are selected state that feature set A appears to hold more prognostic value)

3) Some other way which I dunno

Answer 1

The standard procedure for what you're describing would be very similar to method 1:

1. Train two instances of the same classifier on feature set A and B respectively.
2. Evaluate each using some form of cross validation, say 10 fold cross validation, or leave one out like you have been using.

That said if you're not strictly restricted to feature set A xor B then you may achieve better results deriving a new set C using a similar method to what you described in 2.

It is difficult to trust the results using leave one out cross validation, it would probably be better to use 10-fold. This may be one of those situations where more data would help greatly if you can get it and if not you may not be able to perform your analysis.

Answer 2

In your method 1, I was wondering why you are using different classifier for different features? I am more opt for your method 2. By keep all the features in your training process, you may better select the appropriate features, since the weights for unnecessary features will decay out. And it works well when you have a lot of features, each of which contributes a bit. In this approach, the regularization factor may also be added.