Passing Parameters to a score_func in scikit during cross validation

Question

I want to compute the micro average precision in scikit during cross validation.

The docs here:

http://scikit-learn.org/0.10/modules/cross_validation.html

Say you can pass in a custom scoring function from the metrics module such as metrics.precision_score:

http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score

But by default it computes the scores for each class. If I try and pass in average="micro" like so:

cross_validation.cross_val_score(clf, x, label, cv=5,
score_func=metrics.precision_score(average="micro"))

I receive the following error:

TypeError: precision_score() takes at least 2 arguments (1 given)

I can't pass in all of the other arguments it wants (y_true, y_pred) as I don't know what y_pred is.

Is there a way to get the micro average precision from cross validation in scikit ?

Answer 1

The problem here is that you are calling (with argument average) metrics.precision_score instead of passing the function itself. An ad hoc method to remedy this is to create a function

def micro_average_precision_score(y_true, y_pred):
    metrics.precision_score(y_true, y_pred, average="micro")

and then use it as your score_func, ie score_func=micro_average_precision_score.

On an important side note: score_func is deprecated (since 0.13 if I am not mistaken). You are referring to scikit learn docs of version 0.10. Is that the version you use?

The new way of passing scorers is by using scorer objects. The associated keyword is scoring= and not score_func=. You can make a scorer object out of any scoring function, for example the one defined above, by using make_scorer

from sklearn.metrics.score import make_scorer
scorer = make_scorer(micro_average_precision_score, greater_is_better=True)

or, equivalently:

scorer = make_scorer(metrics.precision_score, 
                     greater_is_better=True, average="micro")