Does the phenomenon of over-fitting of data varies with training algorithms?

Question

Suppose I have a dataset which I want to train using Neural network and SVM. Is it possible that with my dataset after training, the Neural network is overfit while the SVM is not? Like can a dataset be overfit for one training algorithms and not be overfit for another training algorithm? Is it even possible? Or if it is not overfit for one training algorithm, can be assume that it will not be overfit for other training algorithms too?

Answer 1

A model is over-fitting if it makes good predictions on a test set but bad predictions on new data. This is generally a good indication that the used model is too complex.

The complexity of the model is often quantified as the number of free parameters. These are the parameters that need to be set in order to fit the data. More parameters allow more flexibility in what can be expressed but also increase the change of over-fitting.

The type of model also restricts what functions can be learned. For example, linear models can only learn linear functions (really).

The simple answer to your question is yes. For example, a SVM with little parameters might fit data well while a ANN with many parameters might over-fit on the same data.

A fair comparison however compares models with the same number of parameters. But the answer remains yes. One model might be better suited than another to fit the intrinsic structure of the data. For example, on the same data, a decision tree with 100 nodes might over-fit, a linear model might under-fit and a ANN might work perfectly. It all depends on the underlying structure that you want to model.

Answer 2

e.g. linear SVM is less prone to overfitting than non-linear SVM, especially when you have a large number of features (comparing to your training samples)

But yeah in general the more complex a model is the more prone it is to overfitting, e.g. http://arxiv.org/abs/1412.1897