Best text search engine for integrating with custom web app?

Question

We have a web app that allows users to upload documents, create their own documents, and so on. Uploaded files are stored on Amazon S3, created information is stored in a MySQL database. What I'm looking for is some sort of search engine, where I feed it all of our text documents, each with a unique ID, and it builds an index or whatever. Later, I can give it search queries, and it will pull out the best matching documents (via their ID), along with snippets of matching text.

Basically we want to allow our users to search through their repository of uploaded stuffs, along with anything that other users have marked as public. The solution should run on a standard Linux server, and ideally it would be open source, but I'll also consider paid solutions if they aren't outrageously priced.

So far, I've found three potential candidates:

1. MySQL Full Text Search - some reports I've read are that it's very slow
2. Apache Lucene - unfortunately written in Java, but I'll use it if I have to. Supposedly fast
3. Sphinx - doesn't seem to be as popular, ideally whatever solution I find will have lots of community support.

Please let me know if there are any other good choices that I've overlooked, or if you have experience with any of the above.

Answer 1

Take a look at Solr. It's based on Lucene, so it's very fast, and it's really easy to use from any platform.

Answer 2

Sphinx may be worth your consideration, as it works well with several common RDMS (notably MySQL)

Answer 3

There is also Xapian which is fast and is quite customizable.

It has support for custom indexers allowing one to index data that is not stored in a database which might be useful for your documents stored on S3.

Answer 4

I imagine that Google will have a solution that meets your needs. Start here: Google Enterprise

Answer 5

There is a Ruby port of Lucene called "Ferret". In addition to the Ruby API, you can get at the underlying c implementation called "cFerret".

Answer 6

Lucene is very good. And although it was originally written in java there is a php implementation http://framework.zend.com/manual/en/zend.search.lucene.html