Fast at selects/joining MySQL storage engines?

Question

I have some very large databases (some up to 150M rows) I'm working with & after initially inserting the data there isn't much INSERT's going on; just a lot of SELECT's & usage of JOINS.

I've been messing around with InfoBright a lot (the community version) & whilst I believe it is a good engine, I personally have been having some problems with it getting it to run like it should (fast).

So I was wondering if anyone else could recommend any other fast free storage engine for MySQL?

I'm just now checking out tokudb; is there anything else out there to check out as well?

Answer 1

Have you looked at Sphinx at all? While it is a search engine, it also supports query-less searches, which is similar to standard SELECT queries with indexes. I found it to be a huge help when dealing with large datasets. It's very fast, and is used heavily in high-traffic forums who are up in the millions (or hundreds of millions) of posts arena.

There is also a plugin for MySQL called SphinxSE which allows it to act as a MySQL storage engine which makes integration very easy to set up. You build your indexes by supplying the indexer program a query, and then once it's all set up, you can query it as if it was a normal table.

http://sphinxsearch.com/docs/2.0.1/sphinxse-overview.html (note, I haven't used it much since pre 1.0)

Answer 2

You should look at InfiniDB too. http://infinidb.org/ (one of the fastest)

There are a lot of considerations you need to make before benchmarking any engine. Hardware stuff like multicore processors, memory, configuration. Design stuff related to your schema etc etc. and how all this impacts the engine performance.

Do check this blog out for how they do benchmarking of engines (it names other engine types) - http://www.mysqlperformanceblog.com/2010/01/07/star-schema-bechmark-infobright-infinidb-and-luciddb/

Note that this comparison is for a star schema design. If a columnar db engine doesn't suit your requirements, you can look into XtraDB , which is an extended version of InnoDB (not the fastest, but is ACID compliant).

ps - Always track the properties (important to you) of each engine - like referential integrity checks, ACID compliance etc. Sometimes these limitations can be bigger deal breakers as compared to a 10% increase in query performance

Answer 3

Besides taking into consideration which DBMS you use, you should also focus on optimizing your tables, indices and queries. Whenever you have multiple joins, join first on the most selective relation and then on the less selective. Analyze your query execution plans. Create indices on columns that are hit often in your QEPs.

Answer 4

Brett - When using Infobright, you get the best performance gains by: 1) Utilizing the Knowledge Grid as much as possible 2) reducing joins 3) creating 'lookup'

Since the Knowledge-Grid is in-memory, you can kill off a lot of query time just by adding additional filters. Also, consider using a nested select instead of a join. By doing so, you can use an already-created knowledge node (instead of generating a pack-to-pack node on the fly).

If you have some queries that you think should be faster, post them, and I can help with potentially modifying the query to make it run faster.

Cheers,

Jeff