How do I properly perform a MySQL bake-off?

I want to performance test (aka bake-off) MySQL server rpm against some other forks such as Percona server, MariaDB, and possibly some others. I'm hoping that by asking this question I can better understand the methodology behind setting up a proper performance test. I planned on using sysbench to run my actual test, but I'm open to anything.

• What steps should I be taking to

ensure that the test results in a
apples-to-apples comparison and that
only the RDBMS is the variant?

• Where do I get started?

• How do I evaluate results?

• What advice can you give me?

Here is the first thing I thought about

-
Setup 3 DB servers (identical HW/OS configs) each installed with

• MySQL

• Percona

• MariaDB

-
Get a Fourth Server installed with MONyog (eval version lasts 30 days)

-
Register the 3 DB servers in MONyog

-
Use the Performance Metrics Charts in MONyog or Set Up Your Own Charts in MONyog

-
Use SysBench against all 3 DB Servers at your discretion

This is just a basic outline. Believe me, you can use Monyog right out of the box. Check out my testimonial at MonYog's Website (Look for my name on the page). I eventaully convinced my company to purchase the Utlimate Version outright after my testimonial, which they watched with me as it happened.

Caveat : Do no get Monyog until you have the 3 DB servers and Sysbench Ready to get the best 30-day usage of MonYog.

UPDATE 2011-06-20 13:18 EDT

If you keep the InnoDB Buffer Pool Small (default for innodb_buffer_pool_size is 8M) and keep the binary logs disabled (do no include log-bin in my.cnf), then these metrics should be measured

• Innodb_buffer_pool_pages_flushed

• Innodb_buffer_pool_write_requests

• Innodb_row_lock_time_avg

• Innodb_os_log_fsyncs

• Queries, Questions

• SELECTs ((Com_select + Qcache_hits) / Questions)

• INSERTs ((Com_insert + Com_replace) / Questions)

• UPDATEs (Com_update / Questions)

• DELETEs (Com_delete / Questions)

Many of these are default graphs in MONyog's Monitors/Advisors Section, and more graphs can be created to check other desired MySQL metrics. MONyog also lets you export the graphs as CSV data.

UPDATE 2011-10-12 14:17 EDT

Percona performed a bake-off among multiple releases of MySQL

• MySQL 4.1

• MySQL 5.0

• MySQL 5.1 (with built-in InnoDB)

• MySQL 5.1 with InnoDB-plugin

• MySQL 5.5

• MySQL 5.6

All tests were performed with MySQL unconfigured. The results?

• MySQL 4.1 performs the best single-threaded

• MySQL 5.1 with InnoDB plug-in scales on multiple cores better than 5.1 InnoDB built, 5.5 and 5.6

What do I get out of this? Your must configure MySQL 5.5/5.6 to get multicore enhancements engaged.

StackExchange Database Administrators Q#3375, answer score: 14