What is the motivation behind Pearson's coeficient in Apache Mahout

Question

This question is with respect to the Recommendation part of Apache Mahout using Pearson's coefficient for measuring similarity between users. According to my understanding, here is how Pearson's coefficient measures similarity between users.

Let's say 2 users rate 5 items a, b, c, d and e. Let's say their ratings are a1, b1, c1, d1, e1 and a2, b2, c2, d2, e2. Now, Pearson's coefficient gives an estimate of the least square straight line fit considering the points (a1, a2), (b1, b2), (c1, c2), (d1, d2) and (e1, e2). I understand that the main purpose behind doing this (based on my reading elsewhere) is that you want to represent the accuracy of being able to predict the other user's rating for any object, given the current user's rating. Now if the points lie on a straight line, it means you can predict the other person's rating given the current user's rating. So, the coefficient would be 1. On the other hand, if the points don't lie on a straight line, the least square fit is represented on a scale of -1 to 1, 0 being that they are totally offset and there is no relationship at all, and 1/-1 being a perfect fit.

Now, my question is that why only straight line? Why cannot we decide if they lie on a parabola for example and then calculate a similar coefficient accordingly (as to how well a parabola fits)? Why do we check only for a straight line fit?

Thanks Abhishek S

Answer 1

Well that's just how the Pearson correlation coefficient is defined.

It is one possible similarity metric and it is indeed measuring the degree to which two sets of preferences move together proportionally. If you believe that this assumption is appropriate for your data set, it will be a reasonable metric. If not, you'd use another metric.

You could implement a similarity metric that defines similarity some other way, sure.

But, in practice, why would another relationship make sense here? Similarity ought to be symmetric, and a two sets of preferences can't each be proportional to the square of the other. I think the common linear relationship is the reasonable one to look for.

Of course there are plenty of other similarity metrics, arguably better, that have nothing to do with evaluating a relationship like this, like the log-likelihood ratio.