How do you think Reddit handles reindexing their posts to keep the accurately in order?

Question

I can't imagine it indexing per vote. It would strain the server innapropriately.

I mention this because I'm trying to do something similar on a project of mine, and can't figure out what the best way to index objects after they have been voted on.

I am using Sunspot-Solr.

Answer 1

Well, I can't say for sure, but they are an open source project. http://code.reddit.com/

Answer 2

I know I'm late, but today reddit fixed their search in a way that brings a new light to your question. The answer is that search engines like Solr can't deal with frequent updates like reddit votes. They're using IndexTank now, and it definitely works! (yes, I am a developer an IndexTank)

Answer 3

I think that if you are want to influence search results via vote counts you can, but you need to do it in an offline/delayed manner. I think the less "real time" your voting impact on search results is, then the better Solr would work with it, so Reddit is a bad use case. Just update the voting every 6 or 12 or 24 hours, and only do it for the subset of documents that have been voted upon. At some point you'll be able to update just a portion of a Solr document, that will help!

Answer 4

Actually they are indexing once per vote.

We designed the new search (using IndexTank) to update upvotes/downvotes/user karma every time it changes.

The good thing about indextank is that it allows you to update document variables (numeric) affecting the scoring system withot having to reindex the textual data. This allows the index to remain fast even with a huge number of (document variable) updates.