Concerning a recommendation engine

Question

What's a fast "if user A and user B like product C, they might be interested to follow each other" algorithm. I don't think that calculating their similarity at runtime is smart enough, because it will slow down the response. On the other hand, computing an overnight index will require making an (N*N-1) different runs, where N is the number of users ... not very clever, too. Plus, every time a user likes a new product, or a new user registers, indexes have to be recomputed.

What's the smartest thing which could be applied here? Some sort of ultrafast hashing, to which then only the new items are added?

Answer 1

Well among the algorithms that I studied in a course in Uni, there was one dealing with things like this. Their recommended approach was to compute a "similarity" index for each pair of users (which I guess is your N*N method mentioned) and then based on this determine to which users a particular user is closest to.

Of course, you are not required to immediately recalculate the similarity indexes for every change, just once in a while, somewhat like a search engine crawler works. In fact, once you have computed the initial index, you can use various heuristic methods to recompute more often for users who change their preferences fast, and much slower for those who only change them rarely.

Answer 2

Have you thought about an RDF database? Like OWLIM http://www.ontotext.com/owlim