unigrams & bigrams (tf-idf) less accurate than just unigrams (ff-idf)?

Question

This is a question about linear regression with ngrams, using Tf-IDF (term frequency - inverse document frequency). To do this, I am using numpy sparse matrices and sklearn for linear regression.

I have 53 cases and over 6000 features when using unigrams. The predictions are based on cross validation using LeaveOneOut.

When I create a tf-idf sparse matrix of only unigram scores, I get slightly better predictions than when I create a tf-idf sparse matrix of unigram+bigram scores. The more columns I add to the matrix (columns for trigram, quadgram, quintgrams, etc.), the less accurate the regression prediction.

Is this common? How is this possible? I would have thought that the more features, the better.

Answer 1

It's not common for bigrams to perform worse than unigrams, but there are situations where it may happen. In particular, adding extra features may lead to overfitting. Tf-idf is unlikely to alleviate this, as longer n-grams will be rarer, leading to higher idf values.

I'm not sure what kind of variable you're trying to predict, and I've never done regression on text, but here's some comparable results from literature to get you thinking:

• In random text generation with small (but non-trivial) training sets, 7-grams tend to reconstruct the input text almost verbatim, i.e. cause complete overfit, while trigrams are more likely to generate "new" but still somewhat grammatical/recognizable text (see Jurafsky & Martin; can't remember which chapter and I don't have my copy handy).
• In classification-style NLP tasks performed with kernel machines, quadratic kernels tend to fare better than cubic ones because the latter often overfit on the training set. Note that unigram+bigram features can be thought of as a subset of the quadratic kernel's feature space, and {1,2,3}-grams of that of the cubic kernel.

Exactly what is happening depends on your training set; it might simply be too small.

Answer 2

As larsmans said, adding more variables / features makes it easier for the model to overfit hence lose in test accuracy. In the master branch of scikit-learn there is now a min_df parameter to cut-off any feature with less than that number of occurrences. Hence min_df==2 to min_df==5 might help you get rid of spurious bi-grams.

Alternatively you can use L1 or L1 + L2 penalized linear regression (or classification) using either the following classes:

• sklearn.linear_model.Lasso (regression)
• sklearn.linear_model.ElasticNet (regression)
• sklearn.linear_model.SGDRegressor (regression) with penalty == 'elastic_net' or 'l1'
• sklearn.linear_model.SGDClassifier (classification) with penalty == 'elastic_net' or 'l1'

This will make it possible to ignore spurious features and lead to a sparse model with many zero weights for noisy features. Grid Searching the regularization parameters will be very important though.

You can also try univariate feature selection such as done the text classification example of scikit-learn (check the SelectKBest and chi2 utilities.