Archive for November, 2009

Dirty growth-over-time query

I’ve been messing around with the Kontrollbase schema for the last couple of days, writing various queries for the daily reporting scripts that will eventually be an automated pdf report. I’ll give you examples of two of the queries, the first being overall environment stats, and the second being single-host growth over time.

Overall environment stats
select ((((MAX(os_mem_used)) / 1024 ) / 1024) / 1024) max_os_mem_used, ((((MIN(os_mem_used)) / 1024 ) / 1024) / 1024) min_os_mem_used, ((((AVG(os_mem_used)) / 1024 ) / 1024) / 1024) avg_os_mem_used, ((((STDDEV_POP(os_mem_used)) / 1024 ) / 1024) / 1024) stdev_os_mem_used, ((((MAX(length_data + length_index)) / 1024 ) / 1024) / 1024) max_size, ((((MIN(length_data + length_index)) / 1024 ) / 1024) / 1024) min_size, ((((AVG(length_data + length_index)) / 1024 ) / 1024) / 1024) avg_size, ((((STDDEV_POP(length_data + length_index)) / 1024 ) / 1024) / 1024) stdev_size, MAX(num_connections) max_connections, MIN(num_connections) min_connections, AVG(num_connections) avg_connections, STDDEV_POP(num_connections) stdev_connections, MAX(queries_per_second) max_qps, MIN(queries_per_second) min_qps, AVG(queries_per_second) avg_qps, STDDEV_POP(queries_per_second) stdev_qps from server_statistics;
*************************** 1. row ***************************
max_os_mem_used: 50.743488311768
min_os_mem_used: 0.023044586182
avg_os_mem_used: 1.5759627057922952
stdev_os_mem_used: 2.4064208226596184
max_size: 283.815660957247
min_size: 0.000492287800
avg_size: 10.8213909777686940
stdev_size: 39.9443980717105447
max_connections: 435
min_connections: 0
avg_connections: 16.9734
stdev_connections: 37.3269
stdev_os_mem_used: 2.4064208226596184
max_qps: 9243.6533203125
min_qps: 0.00011409764556447
avg_qps: 216.421064774444
stdev_qps: 1071.72792986232
1 row in set (0.00 sec)

single-host growth over time
mysql> select (select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and CURDATE() < = Creation_time order by Creation_time asc limit 1) as 0_day_size_mb, (select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and DATE_SUB(CURDATE(),INTERVAL 30 DAY) <= Creation_time order by Creation_time asc limit 1) as 30_day_size_mb, ( (select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and CURDATE() <= Creation_time order by Creation_time asc limit 1) - (select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and DATE_SUB(CURDATE(),INTERVAL 30 DAY) <= Creation_time order by Creation_time asc limit 1)) as difference, ( select (select ((select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and CURDATE() <= Creation_time order by Creation_time asc limit 1) - (select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and DATE_SUB(CURDATE(),INTERVAL 30 DAY) <= Creation_time order by Creation_time asc limit 1)) as difference) / (select(select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and CURDATE() <= Creation_time order by Creation_time asc limit 1) as 0_day_size_mb ) * 100) as percent_growth;
+---------------+----------------+--------------+----------------+
| 0_day_size_mb | 30_day_size_mb | difference | percent_growth |
+---------------+----------------+--------------+----------------+
| 9100.43986130 | 8199.39263916 | 901.04722214 | 9.901139240197 |
+---------------+----------------+--------------+----------------+
1 row in set (0.01 sec)

IRC is the best support method and Open Source rules

I was working on a server today that was not hooked up to our usual monitoring systems for one reason or another and I needed to generate a database tuning report. Typically I use Matthew Montgomery’s ‘tuning-primer.sh’ script for this since it’s command line based, simple to use, and generates a number of useful items for tuning recommendations. It’s a great starting point before delving into the deeper aspects of MySQL and the OS.

I ran into an issue with it on this server that was running the MySQL 5.0.77-percona-highperf-b13-log x86_64 build. The error was:
./tuning-primer.sh.1: line 517: 5.000000: syntax error in expression (error token is ".000000")

There were three options to fix this issue

  1. Dive into the code and modify it cowboy style
  2. Use our typical monitoring against the client’s wishes
  3. Contact the developer to get a fix

I hopped on IRC.freenode.net to the #mysql channel and found Matthew online. Long story short he narrowed it down to the version of MySQL being from Percona that was changing the format of the variable in question vs the typical MySQL variable type. After discussing the matter he released a new build to address this change and I was on my way to generating reports again.That is quality support that you won’t see from the likes of Microsoft or Apple or any big closed-source player.

If there’s on thing to take away from this story it’s the Open Source Software will ALWAYS be better than closed source because of the hands on attitude and direct contact you can get with the developers or, at the minimum, with the large user community that is willing and able to help troubleshoot. Why are people able to help? Because the code is open and free – and people like free software that they can improve and fix themselves – and those people like to help others because we’ve all needed help at some point or another, no matter how much of an expert you are.

If there are two things to take away it’s to remember that IRC is a wealth of useful information and support for Open Source applications. You can find me on the #mysql, ##php, #perl, #extjs, and #codeigniter channels under various usernames – or idling in the #kontrollbase channel for supporting my own application.

Closed source apps are the old way to do business, the rotting steel skeletons from the industrial age of computing… Open Source is the brainy kid down the street that doesn’t want to rip you off for something that some nameless big corporation designed overseas for pennies on the dollar just to turn a profit and sell you something with crappy support and non-auditable code. Long live OSS!

Tags: , , ,

Vote for Kontrollbase on MySQL Forge

Just a reminder to all of those users that are enjoying Kontrollbase – if you get a minute in your day please go to the MySQL Forge site and put your vote in on Kontrollbase. It’s a simple star based vote on the right side of the page located here: http://forge.mysql.com/projects/project.php?id=318

Tags: , ,

Interesting links that predict the future of data storage

Ok, they aren’t like the 2012 nonsense or anything but they are interesting articles that will be important topics in the near future for expanding our knowledge and usefulness in the job market. Being a DBA does not limit you to relational database systems. I expect to see more requests for non-relational or No-SQL type of data stores as they can have many advantages over an RDBMS. Stability, scalability, simplification of administration, higher performance on lesser hardware for larger datasets, and many other items make no-sql engines very interesting. Here are some useful links:

http://en.wikipedia.org/wiki/NoSQL

http://www.linux-mag.com/cache/7579/1.html

http://www.computerworld.com/s/article/9135086/No_to_SQL_Anti_database_movement_gains_steam_

http://www.viget.com/extend/nosql-misconceptions/

How can I pass up talking about the various Cloud offerings again. Here are some interesting news stories and a shameless link to the cloud service that my employer runs.

http://aws.typepad.com/aws/2009/10/two-new-ec2-instance-types-additional-memory.html

http://www.opsourcecloud.net/

http://virtualization.sys-con.com/node/770174

This is not related to no-sql or the cloud but I’m going to post it anyway: http://www.continuent.com/community/all-projects

How the Cloud saved my electric bill

The following is a pretty good example of a direct impact on the changes in the economy and the cost of energy. I’ve been paying from $80 to $150 per month to run the development servers in my server rack in the garage for the last four years and you can see that the cost of electricity has almost doubled. For the previous two years I’ve lived in Nevada. Nevada gets hot in the summer and without running a dedicated A/C unit for the garage I’ve been forced to power down the rack during the last summer months due to 100+ degree heat.

What is a computer geek to do without development servers? Well the first summer in Nevada I kept one very old Compaq DL360G1 server powered on during work hours with several fans circulating air in the rack. It melted by mid August. After that the solution was to double the RAM and get a quad core CPU in my workstation so I could run dev in virtual servers using VMware Server 2. It got the job done but was far from ideal. Then autumn came and the rack was back on. Many things have changed since last winter and I’ll sum this story up now – the Apple site had a dual core 13″ Macbook Pro with my name on it for traveling and using remote resources to test development code. It will run my external monitors and crunch away on VMs for dev work when I don’t have an internet connection.

What happened to the rack of servers? I sold them to friends that don’t live in the desert and replaced the resources with servers in The Cloud. The beautiful thing about this is that I can spec out servers with 8 or more cores and 32GB of ram and only pay for the processing time I use. No more leased servers or expensive hardware sitting idle and eating up power while they wait for me to crunch some numbers. So, now I have a unibody travelable system that will be running Linux and I get to save up to $150 every month by using the Cloud.