How do I properly perform a MySQL bake-off?

Question

I want to performance test (aka bake-off) MySQL server rpm against some other forks such as Percona server, MariaDB, and possibly some others. I'm hoping that by asking this question I can better understand the methodology behind setting up a proper performance test. I planned on using sysbench to run my actual test, but I'm open to anything.

1. What steps should I be taking to ensure that the test results in a apples-to-apples comparison and that only the RDBMS is the variant?
2. Where do I get started?
3. How do I evaluate results?
4. What advice can you give me?

Answer 1

Here is the first thing I thought about

1. Setup 3 DB servers (identical HW/OS configs) each installed with

   • MySQL
   • Percona
   • MariaDB

2. Get a Fourth Server installed with MONyog (eval version lasts 30 days)

3. Register the 3 DB servers in MONyog

4. Use the Performance Metrics Charts in MONyog or Set Up Your Own Charts in MONyog

5. Use SysBench against all 3 DB Servers at your discretion

This is just a basic outline. Believe me, you can use Monyog right out of the box. Check out my testimonial at MonYog's Website (Look for my name on the page). I eventaully convinced my company to purchase the Utlimate Version outright after my testimonial, which they watched with me as it happened.

Caveat : Do no get Monyog until you have the 3 DB servers and Sysbench Ready to get the best 30-day usage of MonYog.

UPDATE 2011-06-20 13:18 EDT

If you keep the InnoDB Buffer Pool Small (default for innodb_buffer_pool_size is 8M) and keep the binary logs disabled (do no include log-bin in my.cnf), then these metrics should be measured

• Innodb_buffer_pool_pages_flushed
• Innodb_buffer_pool_write_requests
• Innodb_row_lock_time_avg
• Innodb_os_log_fsyncs
• Queries, Questions
• SELECTs ((Com_select + Qcache_hits) / Questions)
• INSERTs ((Com_insert + Com_replace) / Questions)
• UPDATEs (Com_update / Questions)
• DELETEs (Com_delete / Questions)

Many of these are default graphs in MONyog's Monitors/Advisors Section, and more graphs can be created to check other desired MySQL metrics. MONyog also lets you export the graphs as CSV data.

UPDATE 2011-10-12 14:17 EDT

Percona performed a bake-off among multiple releases of MySQL

• MySQL 4.1
• MySQL 5.0
• MySQL 5.1 (with built-in InnoDB)
• MySQL 5.1 with InnoDB-plugin
• MySQL 5.5
• MySQL 5.6

All tests were performed with MySQL unconfigured. The results?

• MySQL 4.1 performs the best single-threaded
• MySQL 5.1 with InnoDB plug-in scales on multiple cores better than 5.1 InnoDB built, 5.5 and 5.6

What do I get out of this? Your must configure MySQL 5.5/5.6 to get multicore enhancements engaged.

Answer 2

This should be very helpful to you as this is the defacto industry benchmark for databases, TPC-C, http://www.tpc.org/tpcc/spec/tpcc_current.pdf

Even if you do not use execute this benchmark, the information contained within the test definition document should provide you with tons of insight into setting up your own benchmark for testing your database options.