How does MyISAM scale compared to Solr for Django searching?

Question

Imagine you have a web application written in Django and Python 2.65, and MySQL 5.1 is your database of choice.

Now, imagine you will need to scale your app to handle searching 100's of thousands of document and potentially 100's of thousands of users will be using it.

Reality: Haystack 1.0 with PySolr and Solr 1.4.0 is proving slow in the above scenarios. Is MyISAM a more workable alternative or should I spend more time working with my current configuration using Solr in a "smarter" way?

Suggestions? Tips?

Thank you for any help! Michaux

Answer 1

I assume you're talking about comparing Solr vs MySQL full-text search, otherwise it would be comparing apples to oranges.

I don't know about Haystack or PySolr, but Solr itself should have no problems handling documents in the order of 100000 with lots of users. Those two parameters alone are not enough to spec the problem, though. For example, frequency of updates, real frequency of requests, document size, page size, sorting, faceting, etc.

Solr is easily scalable, both vertically and horizontally, and is Apache-licensed, while the horizontal scaling solution for MySQL is GPL+commercially licensed.

I disagree with Badger's answer about Tomcat, it's a very polished, proven, stable server that's been around for over 10 years, and the Java performance myth has to be abolished once and for all.

Bottom line: it's very likely that you need to optimize your Solr instance (both the client-side querying and the server-side index and performance settings). Solr powers some of the biggest websites so it's quite likely that it can handle your load as well.

Answer 2

I have no expertise with Haystack or PySolr but just looking at Solr makes me think that MySQL might be a better choice. I know that MySQL can scale to very large applications if it is setup correctly.

Apache Solr just on Tomcat. Tomcat can be a bit of a resource hog and can run slowly. MySQL runs from compiled binaries. This should provide a bit of a boost. The server that you run this off of will also make a large difference. I would say that if you have the ability go ahead and try and setup the MySQL system, and see if you get any difference.