Quick How-To for DRBD + MySQL + LVS

2011/05/20 by admin in MySQL Server, open source | 2 Comments

I wrote this up a while ago and decided that I didn’t want to lose it in a shuffle of documents during my transition to a new workstation. It’s the basics of setting up Heartbeat (LVS) + DRBD (block replication between active/passive master servers) + MySQL. This should give you the basics of a H/A system without the benefits of SAN but also without the associated cost. The validity of this setup for H/A purposes is highly dependent on your workload and environment. You should know the ins and outs of your H/A solution before deciding to blame the system for not performing as expected. As with all production systems you should test, test, test and test some more before going live.

When I get around to it later I’ll post my How-To for setting up RHCS + SAN + MySQL. You can download the DRBD document PDF here: DRBD_LVS_Install-Configure_HowTo

Read More

MySQL Community – what do you want in a load testing framework?

2011/05/09 by admin in MySQL Server, open source, Programming, Python | 2 Comments

So I’ve been doing a fair number of automated load tests these past six months. Primarily with Sysbench, which is a fine, fine tool. First I started using some simple bash based loop controls to automate my overnight testing, but as usually happens with shell scripts they grew unwieldy and I rewrote them in python. Now I have some flexible and easily configurable code for sysbench based MySQL benchmarking to offer the community. I’ve always been a fan of giving back to such a helpful group of people – you’ll never hear me complain about “my time isn’t free”. So, let me know what you want in an ideal testing environment (from a load testing framework automation standpoint) and I’ll integrate it into my existing framework and then release it via the BSD license. The main goal here is to have a standardized modular framework, based on sysbench, that allows anyone to compare their server performance via repeatable tests. It’s fun to see other people’s benchmarks but it’s often difficult to repeat and compare since most tests aren’t fully documented in their blog posts – this could be a solution to that.

Currently I have the harness doing iterations based on:

  • incrementing (choose a global dynamic variable, ie: sync_binlog=0-1000) system values
  • storage engine vs storage engine for the same workload
  • thread quantity increments for read-only or read+write
  • N-nodes in a cluster workloads with WRR traffic distribution (need to code WLC and others)
  • QPS testing for connection pool vs open/close connection
  • multi-table vs single-table workloads

Outputs available: CSV, XML, JSON for easy integration into any number of the various graphing frameworks available. I’ll probably code up a light weight python http server preloaded with Highcharts and Sparklines so you can see your benchmarks easily without having to roll your own graphs.

Quick now, tell me what you’d like me to code for you!

Read More

Win a free book at the February Python Book Contest

2011/02/07 by admin in open source, Programming, Python, Uncategorized | 23 Comments

This month is a special month. It’s not because of Valentines day or even the exciting day where we see groundhogs. No, this month is special because I’m have a book contest where you, the reader, get to win something free for doing absolutely nothing more than posting a comment saying that you want one of the several books I have available in the contest.

So without getting into boring details I’ll keep this short. I’ve been reviewing a lot of books lately and I think it’s time to get some books into people’s hands to enjoy themselves. This month the giveaways are all Python oriented.

So, all you have to do is take a look at the following titles and post a comment here saying that you want one of them. At the end of the month two readers will be chosen via a random list sorting python script I’ve whipped up for just this purpose. You will then get an email from the publisher who will send a brand new e-copy of the book free of charge. I’ll also be reviewing these books at a later date for those that do not win the contest.



Python Text Processing with NLTK 2.0 Cookbook


Python 2.6 Text Processing: Beginners Guide


Python 2.6 Graphics Cookbook

Post a comment now and tell me which book you want!

Read More

Review: MySQL for Python by Albert Lukaszewski

2011/01/22 by admin in open source, Programming, Python, Reviews | Leave a comment

Packt Publishing recently sent me a copy of MySQL for Python to review and after reading through the book I must say that I’m rather impressed at the variety of topics that the book covers.

It starts off with the basics of setting up MySQL for your testing/development needs by going over several of the common installation and configuration methods. After that it’s a quick intro for connection methods and simple error reporting for connections. The author gives a quick intro to CRUD and how it relates to databases and python before heading into the common tasks of simple queries. I was surprised to see some database profiling discussion; which is rather handy for a new coder or a person new to MySQL. Once the basics of Inserts/Selects/Updates/Deletes are covered, which is a rather quick read, there is a welcome discussion of transactions and commit methods – if you do not read this section and are new to MySQL then believe me, you’re missing a very important topic. Most people will gloss over the basics and head right to the more advanced chapters that feature exception handling, the all too common “the mysql server has gone away” error, date&time functions, aggregate functions, and metadata queries. These chapters were the most interesting to me as they covered some great code for python that I have not yet played around with. Previously I’ve done a lot of work on those topics with perl and php so seeing how they were done in python was a great treat. The code is concise, easy to read, and well explained.

A number of topics cover the time saving solutions that no one should be without. Namely, bulk data inserting, data formatting, row iteration, and CSV parsing. Logging methods for access and changes to the database are also covered, and in the end will save your development cycle a lot of time when you are troubleshooting app-to-db interaction.

Two chapters will be of interest to DBAs in particular, and possibly not as interesting to pure developers, of which these are the Disaster Recovery and MySQL Administration topics. The author covers offline backups as well as online hot backups, two sections that no DBA should be without. The code for this type of work is covered in a decent amount of discussion but, along with the other chapters in the book, the theory and background of the topic is also discussed which gives the new reader an understanding of “why” and not just left with the “how”. The administration section of the book covers user creation and permissions management, along with a bit of background on security involved with that task, and also goes into quite a lot of coverage on web-based GUI administration and command line interaction for admin purposes.

Overall I enjoyed the contents of the book and would recommend taking a look if you are new to Python and MySQL or are even looking for a quick reference to the common tasks of database driven application development. This book does not cover the common ORM database interactions you’re likely to see in an app like Django or Pylons, but it will give you a solid foundation on how python and MySQL interact without an abstraction layer. If you are writing quick admin code or building your own database interaction layer, then this book would do well to be in your collection.

You can find the book at Amazon or directly from Packt.

Read More

How to get colored output from ‘ls’ on Solaris10

2010/04/26 by admin in MySQL Server, open source, Solaris | Leave a comment

For all of those linux users out there that have moved over to, or tried out, Solaris10 or OpenSolaris because they heard the tales of how MySQL is faster on Solaris… or perhaps you wanted to learn how to use Sol10 for the great features of Zones or the ZFS filesystem? Regardless of why you’re on it you are probably wondering why Linux has colored output of filenames and directories but Solaris does not. The question of ‘why?’ isn’t important, but how to enable colors is. It’s very simple, and here’s how I fixed it. This is a result of digging through multiple semi-related links on Google.

  1. Download all packages from SunFreeware.com
    • dependency: libintl-3.4.0-sol10-x86-local
    • dependency: libiconv-1.13.1-sol10-x86-local
    • dependency: gmp-4.2.1-sol10-x86-local
    • dependency: gcc-3.4.6-sol10-x86-local or libgcc-3.4.6-sol10-x86-local depending on your system needs
    • coreutils-8.4-sol10-x86-local
  2. Install ‘coreutils’ dependency packages using the command “pkgadd -d [package_name]
  3. Install ‘coreutils’ packages using the command “pkgadd -d coreutils-8.4-sol10-x86-local
  4. Enable color aliases in your rc file: “alias ls=’/usr/local/bin/ls –color=auto’”
Read More

Reviewed: Managing Software Development with SVN and Trac

2010/03/18 by admin in open source, Programming, Reviews | Leave a comment

I’ve recently been migrating my wiki/documentation for Kontrollbase to Trac. For those that are not aware, Trac is a web-based documentation/wiki/Subversion tool that is used by countless number of software projects. Subversion, of course, is a software collaboration and code management repository that manages branches/tags/trunk files with revision control. It’s one of the most heavily used open-source code repositories available. Given that I use SVN (subversion) for all of my software applications and am now using Trac, the book “Managing Software Development with Trac and Subversion” by David J Murphy comes as a useful and great resource for integrating these two useful tools.

The book is not a thousand plus pages type of computer manual. It gets the important points out in a very readable and organized method without winding the user through overly extensive examples. What you get with this book is everything you need to know about combining SVN with Trac without the fluff of other manuals.

In regard to the application of this book, it fits several groups of users; managers that need to understand how their software development team is managing code – or wants to implement a more productive system for their software team, the software developers themselves that need to get up to speed on these most important of technologies, as well as up and coming technologists that want to learn about the code development and management process. Its well roundedness is one of the best features.

The content is broken up into several sections: content management basics, Trac and SVN basics, trac and svn setup and integration, documentation creation and management, code management, and then very useful appendixes for the actual install process of apache, SVN, and Trac. Each section has everything you need to get started and finished in quick order. The author is showing that the processes involved in code management do not have to be stressful or difficult to learn.

The author stresses several key ideas before getting into the meat of the content. These being “everything is a task”, “small steps are better”, “communication is key”, “content management is what software development requires for success.” Overall the two main division of labor is between task management and communication between team members.

Overall this is a very useful book on the subject of code management and documentation. I would recommend it to anyone looking to learn about the processes involved as well as anyone that wants a reference manual for growing their technology bookshelf.

I’ve recently been migrating my wiki/documentation for Kontrollbase to Trac. For those that are not aware, Trac is a web-based documentation/wiki/Subversion tool that is used by countless number of software projects. Subversion, of course, is a software collaboration and code management repository that manages branches/tags/trunk files with revision control. It’s one of the most heavily used open-source code repositories available. Given that I use SVN (subversion) for all of my software applications and am now using Trac, the book “Managing Software Development with Trac and Subversion” by David J Murphy comes as a useful and great resource for integrating these two useful tools.

The book is not a thousand plus pages type of computer manual. It gets the important points out in a very readable and organized method without winding the user through overly extensive examples. What you get with this book is everything you need to know about combining SVN with Trac without the fluff of other manuals.

In regard to the application of this book, it fits several groups of users; managers that need to understand how their software development team is managing code – or wants to implement a more productive system for their software team, the software developers themselves that need to get up to speed on these most important of technologies, as well as up and coming technologists that want to learn about the code development and management process. Its well roundedness is one of the best features.

The content is broken up into several sections: content management basics, Trac and SVN basics, trac and svn setup and integration, documentation creation and management, code management, and then very useful appendixes for the actual install process of apache, SVN, and Trac. Each section has everything you need to get started and finished in quick order. The author is showing that the processes involved in code management do not have to be stressful or difficult to learn.

The author stresses several key ideas before getting into the meat of the content. These being “everything is a task”, “small steps are better”, “communication is key”, “content management is what software development requires for success.” Overall the two main division of labor is between task management and communication between team members.

Overall this is a very useful book on the subject of code management and documentation. I would recommend it to anyone looking to learn about the processes involved as well as anyone that wants a reference manual for growing their technology bookshelf. You can find the book at the Packt Publishing website here.

Read More

RESTful PHP Web Services – reviewed

2010/01/21 by admin in open source, PHP, Programming | Leave a comment

I’ve been using a lot of RESTful services these days and have been waiting for a good book that is dedicated to the topic. I recently received a copy of ‘RESTful PHP Web Services’, which does a successful job of outlining proven concepts in current web technology. If you want to learn the methods for creating and consuming RESTful services then you will find many examples in this book. From the architectural plans to well thought out code samples, the book covers a lot of ground in a relatively quick read.

The first chapter gives the reader a quick introduction to RESTful services and the most common PHP frameworks in use at the time of writing. I particularly enjoyed the section on the Zend framework due to the explanation of benefits over the other frameworks. The chapter also covers the very basics which include a detailed look at exactly what RESTful services means and what technologies are required to use and benefit from a RESTful architecture. The second chapter gives a quick run down of the various methods in use for consumption of data; these being Curl, several HTTP methods, processing data with XML, DOM, and SimpleXML. After those are covered there is a simple example of consuming services like Flickr using the previous methods. This transitions into many more examples of consuming real world services that any developer would find interesting and exciting for data mashups.

The real meat of the book starts in chapter four where we get into designing the resource utilization systems and then the resource clients in chapter five. Those topics basically go over the nuts and bolts of gathering data, manipulating it, updating it, as well as creating fresh data. We get more instruction and usage examples on the Zend framework in chapter seven where the author gives us information on the controllers, models, and view (MVC model). This would not be too useful without knowing how to debug the code that we’re using so there is, thankfully, a chapter dedicated to debugging XML building and parsing errors. A couple of short appendixes cover the author’s own WSO2 web service framework as well as REST Client Classes which should prove useful for writing your own reusable classes.

Overall this book covers the majority of topics that a new developer needs to understand in order to start developing and deploying RESTful code and web services in PHP. From frameworks to consumable service samples, and everything in between, RESTful PHP Web Services comes through in a concise and enjoyable style that will not disappoint. I highly recommend this book for developers that are new to this topic or experienced developers that need a quick refresher course.

Read More