TheMattReid

If you own a Nokia N900 cellular device you might be interested in the ability to control all of your IM accounts from the command line. For those that do not know, the N900 runs Maemo Linux and is capable of running MySQL embedded if you so choose. Here’s a quick script I wrote to provide that functionality for IM accounts. It’s at the bottom of the page, called “im-connections”.

wiki: http://wiki.maemo.org/N900_Mission_Control#Set_all_SIP_accounts_to_online_or_offline
pastebin: http://pastebin.com/qAC57E1N

A couple of question I get a lot from MySQL customers is “how will this hardware upgrade improve my transactions per second (TPS)” and “what level of TPS will MySQL perform on this hardware if I’m running ACID settings?” Running sysbench against MySQL with different values for per-thread and global memory buffer sizes, ACID settings, and other settings gives me concrete values to bring to the customer to show the impact that more RAM, faster CPUs, faster disks, or cnf changes have on the server. Here are some examples for a common question: “If I’m using full ACID settings vs non-ACID settings what performance am I going to get from this server?”

Let’s find out by running sysbench with the following settings (most are self explanatory – if not the man page can explain them):

• sysbench –test=oltp –db-driver=mysql –oltp-table-size=1000000 –mysql-engine-trx=yes –oltp-test-mode=complex –oltp-read-only=off –oltp-dist-type=special –max-requests=0 –num-threads=8 –max-time=120 –init-rng=on run

MySQL Settings:

In the first test MySQL is set to the following ACID related settings. This will give us results for TPS performance without full ACID compliance – very common settings on a server that is handling blogs, ad serving, general business websites, and other roles where full ACID is not required and performance is valued over the benefits of full ACID. These are important settings when we look at the difference in performance when we change to full ACID in the second test.

• innodb_flush_log_at_trx_commit = 0
• sync_binlog=0
• transaction-isolation=REPEATABLE-READ

System configuration and InnoDB buffer pool size:

• XEON E5345 Series 2.33ghz 8-core, 16GB RAM, Local SATA 7.2K disks
• innodb_buffer_pool_size = 10G

Full result set from sysbench:

Summary OLTP test statistics:

• queries performed:
• transactions:                        172426 (1436.83 per sec.)
• read/write requests:                 3276664 (27304.51 per sec.)
• other operations:                    344882 (2873.91 per sec.)

Non-ACID results:

We can simplify the results by looking at the following TPS results for this non-ACID test:

• transactions:                        172426 (1436.83 per sec.)

Full ACID results:

Let’s go ahead and run the test again with different ACID settings. This will give us the TPS results for full ACID compliance:

• innodb_flush_log_at_trx_commit = 1
• sync_binlog=1
• transaction-isolation=REPEATABLE-READ

We get the following results for TPS:

• transactions:                     3197   (26.58 per sec.)
• read/write requests:                 60743  (505.04 per sec.)
• other operations:                    6394   (53.16 per sec.)

Final Results:

So as you can see the difference between full ACID settings and not (on the same server with only those values on the cnf being changed) results in a huge difference in performance on this standard database server. We can now hand this data to the customer and they will know what impact the settings will have on their application’s performance and what to expect when running full ACID vs non-ACID.

More info on using sysbench here: http://sysbench.sourceforge.net

I guess they got tired of people sending angry emails about data transfer fees:

“Amazon provides an online calculator to help customers decide whether it makes financial sense to ship data via mail rather than uploading over the Internet. You plug in the number of terabytes, devices, average file size, return shipping information and other factors, and find out how much the data transfer would cost via mail compared to standard Internet uploads.

For example, transferring data from a single device containing 2TB would require 26 hours of data loading time and cost $144.74. Uploading the same amount of data over the Internet would cost $204.80. The calculator does not show how long the Internet transfer would take.”

http://www.networkworld.com/news/2010/061010-amazon-cloud-fedex.html

If you’ve been reading up on the various NoSQL offerings and have wanted to try out one but don’t know how to get started, this is one of the easiest ways. I chose MongoDB for this example because I’m going to start using it for a project that needs features that MySQL isn’t as fast at: namely denormalized data with billions of rows. MongoDB has plenty of drivers for other scripting and high-level languages but I’ll focus on the PHP driver today. If there is interest I can do a write up on Python usage later. This example is limited to CentOS, Fedora, and Redhat 5 servers that use the yum package management system. For more information you can reference their download page: http://www.mongodb.org/display/DOCS/Downloads

First install the prerequisites:

• sudo yum install gcc php php-pear

Then install the mogo php extension via pecl and add the extension to the ini file. (Increase the php.ini memory_limit to 32M if necessary). Then we restart apache to have the extension loaded. You can check your extensions via the phpinfo(); command in php.

• sudo pecl install mongo
• sudo echo “extension=mongo.so” >> /etc/php.ini
• sudo /etc/init.d/httpd restart

For the last part we add the yum repo and get the actual MongoDB server and client installed

• cd /etc/yum.repos.d
• sudo emacs mongodb.10gen.repo
• paste in the following. This uses the repo for 64bit servers. See their download page for other architecture repo settings.

[10gen]
name=10gen Repository
baseurl=http://downloads.mongodb.org/distros/centos/5.4/os/x86_64/
gpgcheck=0

• sudo yum install mongo-stable-server
• sudo /etc/init.d/monod start
• chkconfig mongod on

That’s it, you’re ready to start using Mongo! You can check the logfile to make sure things are running smoothly: “sudo cat /var/log/mongo/mongod.log”

For all of those linux users out there that have moved over to, or tried out, Solaris10 or OpenSolaris because they heard the tales of how MySQL is faster on Solaris… or perhaps you wanted to learn how to use Sol10 for the great features of Zones or the ZFS filesystem? Regardless of why you’re on it you are probably wondering why Linux has colored output of filenames and directories but Solaris does not. The question of ‘why?’ isn’t important, but how to enable colors is. It’s very simple, and here’s how I fixed it. This is a result of digging through multiple semi-related links on Google.

1. Download all packages from SunFreeware.com
   • dependency: libintl-3.4.0-sol10-x86-local
   • dependency: libiconv-1.13.1-sol10-x86-local
   • dependency: gmp-4.2.1-sol10-x86-local
   • dependency: gcc-3.4.6-sol10-x86-local or libgcc-3.4.6-sol10-x86-local depending on your system needs
   • coreutils-8.4-sol10-x86-local
2. Install ‘coreutils’ dependency packages using the command “pkgadd -d [package_name]
3. Install ‘coreutils’ packages using the command “pkgadd -d coreutils-8.4-sol10-x86-local
4. Enable color aliases in your rc file: “alias ls=’/usr/local/bin/ls –color=auto’”

I’ve recently had the pleasure of reading “Python Testing: An easy and convenient approach to testing your python projects” from Packt Publishing. It’s been a quick read but a solid set of instructions on the different methods for the subject.

The book starts out very quickly with details about the various methods that are available, the means of automation for testing, and of course the environment you’d want to be in for working on the subjects that the book covers. It then, in the second chapter, moves into the guts of testing by describing the basics of doctest via syntax and some simple examples, and then moves on to a real world example via the AVL tree. It’s all very basic testing until chapter three where the author gets into unit testing, which is probably the most useful method in my opinion, and he goes to prove it’s usefulness with examples of it’s use in different parts and stages of the development process. Later in the book the python mocker is used to separate unit sections, and then the actual unittest framework is discussed with more examples and a enough details that if you don’t understand it by then, you may never. By chapter six we are into the Nose app that drives the unittest, which is very useful of course.

The most useful part of the book comes toward the end where the author discusses and the walks through the method used to create a test-driven application and then even shows examples via the whole chapter dedicated towards making a testable web application frontend. Very impressive for such a quick read. Integration testing and System testing is also covered, thankfully. The final chapter covers some useful tools and techniques of which I particularly enjoyed the section on version control hooks. If you are not using version control in your development process you need to start now, as such the hooks for integration with the test framework are rather useful to know.

Overall this is a very nice book that discusses python application testing from the ground up. It’s perfect for a beginner or an intermediate python programmer that has little to no experience in automated testing methods. More advanced programmers that have already used these methods will probably not find the book too useful except for the last chapter that covers extra tools and techniques that they might not have seen before. If I didn’t have this book and needed to learn about python testing, it would be my first choice and my only recommendation so far. Well written and very useful.

If there is one thing I do not like about the book, it would be the reliance on the python CLI for running commands. I am a CLI kind of person and I keep lots of terminals open at the same time, so I prefer to write my code in an editor or IDE in one term tab, then switch to another and execute the script; I do not use the python command line to do much of anything. So following some of the steps in the book require that you follow the CLI method and that gets old for me. It’s a personal preference but one worth noting as there is a lot of it in the book. That’s the only thing I did not enjoy in a book that was otherwise basically perfect for the subject.

Saw this interesting paper about highly concurrent programming methods and figured the word should be spread! It’s not new material but it’s a good read. See the full article here: http://www.usenix.org/events/hotos03/tech/full_papers/vonbehren/vonbehren_html/

“Highly concurrent applications such as Internet servers and transaction processing databases present a number of challenges to application designers. First, handling large numbers of concurrent tasks requires the use of scalable data structures. Second, these systems typically operate near maximum capacity, which creates resource contention and high sensitivity to scheduling decisions; overload must be handled with care to avoid thrashing. Finally, race conditions and subtle corner cases are common, which makes debugging and code maintenance difficult.

Threaded servers have historically failed to meet these challenges, leading many researchers to conclude that event-based programming is the best (or even only) way to achieve high performance in highly concurrent applications. “

I was facing an issue today: the old G4 Mac that I own has Photoshop on it. The new mac I use – an intel macbook pro cannot install the PPC version of Photoshop that I own. The old G4 died last week, so no using PS. I downloaded Photoshop Elements to demo it and get one image converted from a large size to a small size and then do some gradient work. PS Elements refused to install with various errors. I searched Google and came across an application called Pixelmator. It’s one of the best photo editing tools I’ve used in a long time at this price point. In fact it seems to have all of the same features and tools as my old version of Photoshop CS, and it’s only $59. I’ll continue to demo it for the 30 day trial. Great app. I highly recommend it if you don’t feel like spending hundreds on the competition.

I’ve recently been migrating my wiki/documentation for Kontrollbase to Trac. For those that are not aware, Trac is a web-based documentation/wiki/Subversion tool that is used by countless number of software projects. Subversion, of course, is a software collaboration and code management repository that manages branches/tags/trunk files with revision control. It’s one of the most heavily used open-source code repositories available. Given that I use SVN (subversion) for all of my software applications and am now using Trac, the book “Managing Software Development with Trac and Subversion” by David J Murphy comes as a useful and great resource for integrating these two useful tools.

The book is not a thousand plus pages type of computer manual. It gets the important points out in a very readable and organized method without winding the user through overly extensive examples. What you get with this book is everything you need to know about combining SVN with Trac without the fluff of other manuals.

In regard to the application of this book, it fits several groups of users; managers that need to understand how their software development team is managing code – or wants to implement a more productive system for their software team, the software developers themselves that need to get up to speed on these most important of technologies, as well as up and coming technologists that want to learn about the code development and management process. Its well roundedness is one of the best features.

The content is broken up into several sections: content management basics, Trac and SVN basics, trac and svn setup and integration, documentation creation and management, code management, and then very useful appendixes for the actual install process of apache, SVN, and Trac. Each section has everything you need to get started and finished in quick order. The author is showing that the processes involved in code management do not have to be stressful or difficult to learn.

The author stresses several key ideas before getting into the meat of the content. These being “everything is a task”, “small steps are better”, “communication is key”, “content management is what software development requires for success.” Overall the two main division of labor is between task management and communication between team members.

Overall this is a very useful book on the subject of code management and documentation. I would recommend it to anyone looking to learn about the processes involved as well as anyone that wants a reference manual for growing their technology bookshelf.

I’ve recently been migrating my wiki/documentation for Kontrollbase to Trac. For those that are not aware, Trac is a web-based documentation/wiki/Subversion tool that is used by countless number of software projects. Subversion, of course, is a software collaboration and code management repository that manages branches/tags/trunk files with revision control. It’s one of the most heavily used open-source code repositories available. Given that I use SVN (subversion) for all of my software applications and am now using Trac, the book “Managing Software Development with Trac and Subversion” by David J Murphy comes as a useful and great resource for integrating these two useful tools.

The book is not a thousand plus pages type of computer manual. It gets the important points out in a very readable and organized method without winding the user through overly extensive examples. What you get with this book is everything you need to know about combining SVN with Trac without the fluff of other manuals.

In regard to the application of this book, it fits several groups of users; managers that need to understand how their software development team is managing code – or wants to implement a more productive system for their software team, the software developers themselves that need to get up to speed on these most important of technologies, as well as up and coming technologists that want to learn about the code development and management process. Its well roundedness is one of the best features.

The content is broken up into several sections: content management basics, Trac and SVN basics, trac and svn setup and integration, documentation creation and management, code management, and then very useful appendixes for the actual install process of apache, SVN, and Trac. Each section has everything you need to get started and finished in quick order. The author is showing that the processes involved in code management do not have to be stressful or difficult to learn.

The author stresses several key ideas before getting into the meat of the content. These being “everything is a task”, “small steps are better”, “communication is key”, “content management is what software development requires for success.” Overall the two main division of labor is between task management and communication between team members.

Overall this is a very useful book on the subject of code management and documentation. I would recommend it to anyone looking to learn about the processes involved as well as anyone that wants a reference manual for growing their technology bookshelf. You can find the book at the Packt Publishing website here.

This is a simple matter with a simple solution that might help someone save time and confusion. Emacs wasn’t coloring my comments correctly so I went ahead and had it change them to red-italic. If you are having similar issues you can drop the following into your home directory’s .emacs file. Enjoy. Keep in mind that if you are using emacs in a terminal session as opposed to the X-server gui then you will not see the italics.


(global-font-lock-mode 1)
(custom-set-variables
'(gud-gdb-command-name "gdb --annotate=1")
'(large-file-warning-threshold nil))
(custom-set-faces
'(font-lock-comment-face ((((class color) (background light)) (:foreground "red" :slant italic)))))