On the reporting engine, and a new name – Kontrollbase

2008/12/05 by admin in Kontrollbase, Monolith | 3 Comments

Update for all of those that are following the progress of Monolith. I’ve been busy this week coding the reporting engine – the code that analyses the hundreds of variables that are being collected – and I’m pleased to say that Perl was the correct choice for this application.Installation on the client servers will be a breeze, and relatively few non-standard modules are required for the server application.

I’ve been silently wondering for some time if the reporting engine equations should be part of a pluggable system (stored in the database) or if they should be hard coded. After writing up all of the equations, I’ve come to the conclusion that, for a quicker release, the equations and reporting engine will be hard-coded this time around for the default system alerts. Howeve -> user-defined alerts will be pluggable. Revisions ahead of 2.0 might feature pluggable main equations but I want v2.0 out asap.

Second bit of news on the application is that any users that were requesting a simple web-based install with a single config file have now gotten their wish. It’s a much streamlined process this time around.

And the final news is that I have decided to change the name of Monolith. There’s an app on the MySQL site that is from the Monolith company and it does monitoring along the lines of Zenoss – as in site wide monitoring that isn’t MySQL specific like my software. As such, to keep the projects easier to differentiate – my Monolith app will be released under the name of Kontrollbase. Kontroll is Swedish for control, and seeing as MySQL is originally Swedish and that the app helps you control the performance of MySQL… there it is. So, from now on Monolith will be referred to as Kontrollbase.

Things are looking up for release 2 and it’s coming along with good speed. More later… back to coding.

Read More

Parallel mysqldump backup script available. Testers wanted.

2008/10/07 by admin in Backups, Monolith, Scripts | 1 Comment

Large databases, long mysqldump times, long waits for globally locked tables. These problems basically never go away when you rely on mysqldump with –all-databases or a list of databases, as it dumps schemas serially. I’m not going to explain serial vs parallel processing here since that’s a larger topic. Suffice to say that in these days of multi-core / multi-cpu servers we only make use of one processor’s core when we serially export databases using mysqldump. So, I have a new script that attempts to alleviate those issues and now I need testers to provide feedback/improvements.

 

In order to keep some sanity when dealing with hundreds of database servers, the script takes care of the following:

  1. low global locking time requirements: solved by parallel tasks / forked processes
  2. backup file checking: with mysqldump files; it checks for “–Dump completed” at the end of the sql file
  3. ability to backup all-databases or an array of databases
  4. if backing up all databases, the ability to skip schema names like information_schema, lost&found, test, etc that can be in the data directory but aren’t really schemas
  5. compression of sql files
  6. size reporting of plain text and compressed sql files
  7. overall time and single export elapsed time reporting for each schema backup
  8. debug level logging of sub routines for troubleshooting if needed
  9. file pruning to keep 28 days of backups, so you don’t have to manually remove files to keep the backups partition from filling up
  10. talkback reporting for Monolith monitoring system / email reporting as well
  11. logs to /var/log/messages with final report so we get information in network based syslog collection
  12. process state exit code reporting in the case of various issues like “mysqldump failed because binlog not enabled when using –master-data=x”

So, that all said, if you want to help test this script email me at ” the matt reid at g m a i l . c o m”

It’s a bit over 800 lines of perl so I’m not going to paste it here, therefore email me :)

Read More