MySQL Community – what do you want in a load testing framework?

So I’ve been doing a fair number of automated load tests these past six months. Primarily with Sysbench, which is a fine, fine tool. First I started using some simple bash based loop controls to automate my overnight testing, but as usually happens with shell scripts they grew unwieldy and I rewrote them in python. Now I have some flexible and easily configurable code for sysbench based MySQL benchmarking to offer the community. I’ve always been a fan of giving back to such a helpful group of people – you’ll never hear me complain about “my time isn’t free”. So, let me know what you want in an ideal testing environment (from a load testing framework automation standpoint) and I’ll integrate it into my existing framework and then release it via the BSD license. The main goal here is to have a standardized modular framework, based on sysbench, that allows anyone to compare their server performance via repeatable tests. It’s fun to see other people’s benchmarks but it’s often difficult to repeat and compare since most tests aren’t fully documented in their blog posts – this could be a solution to that.

Currently I have the harness doing iterations based on:

  • incrementing (choose a global dynamic variable, ie: sync_binlog=0-1000) system values
  • storage engine vs storage engine for the same workload
  • thread quantity increments for read-only or read+write
  • N-nodes in a cluster workloads with WRR traffic distribution (need to code WLC and others)
  • QPS testing for connection pool vs open/close connection
  • multi-table vs single-table workloads

Outputs available: CSV, XML, JSON for easy integration into any number of the various graphing frameworks available. I’ll probably code up a light weight python http server preloaded with Highcharts and Sparklines so you can see your benchmarks easily without having to roll your own graphs.

Quick now, tell me what you’d like me to code for you!

2 thoughts on “MySQL Community – what do you want in a load testing framework?

  1. George says:

    Hi Matt thanks for the offer and your time :)

    I recently started playing with sysbench http://vbtechsupport.com/tag/mariadb-5-2-5/ and found it time consuming to run all the tests and collate them for charts even with simple bash scripts and yes the scripts end up growing in size as you progress!

    Just read Baron’s blog over at http://www.xaprb.com/blog/2011/05/10/whats-a-good-benchmark/ about benchmarking needing to be fully measurable i.e. I/O and CPU usage, memory usage etc. That would be part missing during my sysbench tests, so that would be my suggestion to add to your load testing framework (if it hasn’t already). So the outputs in CSV, XML, JSON would provide easier way to chart the sysbench results as well as system stats to be charted. Being able to turn off/on the system measured stats from testing and output would be nice to see how much impact and overhead the measuring has on results.

    thanks for listening :)

    George

  2. admin says:

    Great ideas George. OS level monitoring is definitely a key metric to collect while doing any DB level tests. I usually correlate with the cacti graphs in my monitoring server but for a standalone monitoring framework they will be included via SAR or SNMP metrics.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>