Easy Python: MySQL connection and iteration

If you’ve been looking for a simple python script to use with MySQL that you can use to expand upon for your next project, check this one out. It has error handling for the connection, error handling for the sql call, and loop iteration for the rows returned.

#!/usr/bin/python
import sys
import MySQLdb

my_host = "localhost"
my_user = "user"
my_pass = "password"
my_db = "test"

try:
    db = MySQLdb.connect(host=my_host, user=my_user, passwd=my_pass, db=my_db)
except MySQLdb.Error, e:
     print "Error %d: %s" % (e.args[0], e.args[1])
     sys.exit (1)

cursor = db.cursor()
sql = "select column1, column2 from table";
cursor.execute(sql)    
results = cursor.fetchall()
for row in results:
    column1 = row[0]
    column2 = row[1]
    print "column1: %s, column2: %s"%(column1,column2)

db.close()

How to: rotate wordpress posts into headline/feature status

If you’re using the new Arthemia theme for WordPress you might notice that there are two areas of the theme that can have articles promoted to; namely Headline and Featured sections. This is controlled by category association. Basically you have a post and if you want it in the Headline area of the theme you attach the category “headline” to it, similarly for the featured section. Now, let’s say you don’t want to manually change this all the time since it can be time consuming to promote posts to those categories if you want rotating content.

Here’s a simple solution. In this bash script I connect to MySQL and remove the current associations from posts and then randomly choose posts to be promoted to the Headline and Featured categories. This can be modified for other ideas you might have involving categories/posts/randomized associations in WordPress.

The queries contain IDs for the Headline and Featured categories. In my installation, which will be different than yours, has the Headline category as ID=’103′ and Featured as ID=’104′ – replace as needed. I’m also doing some matching (see the WHERE sections) so that I don’t promote posts with certain IDs that are specific to the site for this script. You’ll want to customize the queries as needed for your site. You can find the script here: http://pastebin.com/1QqiM5rh

How to install MongoDB on CentOS 5.4 / RHEL5 and interface with PHP 5

If you’ve been reading up on the various NoSQL offerings and have wanted to try out one but don’t know how to get started, this is one of the easiest ways. I chose MongoDB for this example because I’m going to start using it for a project that needs features that MySQL isn’t as fast at: namely denormalized data with billions of rows. MongoDB has plenty of drivers for other scripting and high-level languages but I’ll focus on the PHP driver today. If there is interest I can do a write up on Python usage later. This example is limited to CentOS, Fedora, and Redhat 5 servers that use the yum package management system. For more information you can reference their download page: http://www.mongodb.org/display/DOCS/Downloads

First install the prerequisites:

  • sudo yum install gcc php php-pear

Then install the mogo php extension via pecl and add the extension to the ini file. (Increase the php.ini memory_limit to 32M if necessary). Then we restart apache to have the extension loaded. You can check your extensions via the phpinfo(); command in php.

  • sudo pecl install mongo
  • sudo echo “extension=mongo.so” >> /etc/php.ini
  • sudo /etc/init.d/httpd restart

For the last part we add the yum repo and get the actual MongoDB server and client installed

  • cd /etc/yum.repos.d
  • sudo emacs mongodb.10gen.repo
  • paste in the following. This uses the repo for 64bit servers. See their download page for other architecture repo settings.
  • [10gen]
    name=10gen Repository
    baseurl=http://downloads.mongodb.org/distros/centos/5.4/os/x86_64/
    gpgcheck=0

  • sudo yum install mongo-stable-server
  • sudo /etc/init.d/monod start
  • chkconfig mongod on

That’s it, you’re ready to start using Mongo! You can check the logfile to make sure things are running smoothly: “sudo cat /var/log/mongo/mongod.log”

How to get colored output from ‘ls’ on Solaris10

For all of those linux users out there that have moved over to, or tried out, Solaris10 or OpenSolaris because they heard the tales of how MySQL is faster on Solaris… or perhaps you wanted to learn how to use Sol10 for the great features of Zones or the ZFS filesystem? Regardless of why you’re on it you are probably wondering why Linux has colored output of filenames and directories but Solaris does not. The question of ‘why?’ isn’t important, but how to enable colors is. It’s very simple, and here’s how I fixed it. This is a result of digging through multiple semi-related links on Google.

  1. Download all packages from SunFreeware.com
    • dependency: libintl-3.4.0-sol10-x86-local
    • dependency: libiconv-1.13.1-sol10-x86-local
    • dependency: gmp-4.2.1-sol10-x86-local
    • dependency: gcc-3.4.6-sol10-x86-local or libgcc-3.4.6-sol10-x86-local depending on your system needs
    • coreutils-8.4-sol10-x86-local
  2. Install ‘coreutils’ dependency packages using the command “pkgadd -d [package_name]
  3. Install ‘coreutils’ packages using the command “pkgadd -d coreutils-8.4-sol10-x86-local
  4. Enable color aliases in your rc file: “alias ls=’/usr/local/bin/ls –color=auto’”

Reviewed: Python Testing by Daniel Arbuckle

I’ve recently had the pleasure of reading “Python Testing: An easy and convenient approach to testing your python projects” from Packt Publishing. It’s been a quick read but a solid set of instructions on the different methods for the subject.

The book starts out very quickly with details about the various methods that are available, the means of automation for testing, and of course the environment you’d want to be in for working on the subjects that the book covers. It then, in the second chapter, moves into the guts of testing by describing the basics of doctest via syntax and some simple examples, and then moves on to a real world example via the AVL tree. It’s all very basic testing until chapter three where the author gets into unit testing, which is probably the most useful method in my opinion, and he goes to prove it’s usefulness with examples of it’s use in different parts and stages of the development process. Later in the book the python mocker is used to separate unit sections, and then the actual unittest framework is discussed with more examples and a enough details that if you don’t understand it by then, you may never. By chapter six we are into the Nose app that drives the unittest, which is very useful of course.

The most useful part of the book comes toward the end where the author discusses and the walks through the method used to create a test-driven application and then even shows examples via the whole chapter dedicated towards making a testable web application frontend. Very impressive for such a quick read. Integration testing and System testing is also covered, thankfully. The final chapter covers some useful tools and techniques of which I particularly enjoyed the section on version control hooks. If you are not using version control in your development process you need to start now, as such the hooks for integration with the test framework are rather useful to know.

Overall this is a very nice book that discusses python application testing from the ground up. It’s perfect for a beginner or an intermediate python programmer that has little to no experience in automated testing methods. More advanced programmers that have already used these methods will probably not find the book too useful except for the last chapter that covers extra tools and techniques that they might not have seen before. If I didn’t have this book and needed to learn about python testing, it would be my first choice and my only recommendation so far. Well written and very useful.

If there is one thing I do not like about the book, it would be the reliance on the python CLI for running commands. I am a CLI kind of person and I keep lots of terminals open at the same time, so I prefer to write my code in an editor or IDE in one term tab, then switch to another and execute the script; I do not use the python command line to do much of anything. So following some of the steps in the book require that you follow the CLI method and that gets old for me. It’s a personal preference but one worth noting as there is a lot of it in the book. That’s the only thing I did not enjoy in a book that was otherwise basically perfect for the subject.

Event based programming vs threading by Rob von Behren, Jeremy Condit and Eric Brewer

Saw this interesting paper about highly concurrent programming methods and figured the word should be spread! It’s not new material but it’s a good read. See the full article here: http://www.usenix.org/events/hotos03/tech/full_papers/vonbehren/vonbehren_html/

“Highly concurrent applications such as Internet servers and transaction processing databases present a number of challenges to application designers. First, handling large numbers of concurrent tasks requires the use of scalable data structures. Second, these systems typically operate near maximum capacity, which creates resource contention and high sensitivity to scheduling decisions; overload must be handled with care to avoid thrashing. Finally, race conditions and subtle corner cases are common, which makes debugging and code maintenance difficult.

Threaded servers have historically failed to meet these challenges, leading many researchers to conclude that event-based programming is the best (or even only) way to achieve high performance in highly concurrent applications. “

Reviewed: Managing Software Development with SVN and Trac

I’ve recently been migrating my wiki/documentation for Kontrollbase to Trac. For those that are not aware, Trac is a web-based documentation/wiki/Subversion tool that is used by countless number of software projects. Subversion, of course, is a software collaboration and code management repository that manages branches/tags/trunk files with revision control. It’s one of the most heavily used open-source code repositories available. Given that I use SVN (subversion) for all of my software applications and am now using Trac, the book “Managing Software Development with Trac and Subversion” by David J Murphy comes as a useful and great resource for integrating these two useful tools.

The book is not a thousand plus pages type of computer manual. It gets the important points out in a very readable and organized method without winding the user through overly extensive examples. What you get with this book is everything you need to know about combining SVN with Trac without the fluff of other manuals.

In regard to the application of this book, it fits several groups of users; managers that need to understand how their software development team is managing code – or wants to implement a more productive system for their software team, the software developers themselves that need to get up to speed on these most important of technologies, as well as up and coming technologists that want to learn about the code development and management process. Its well roundedness is one of the best features.

The content is broken up into several sections: content management basics, Trac and SVN basics, trac and svn setup and integration, documentation creation and management, code management, and then very useful appendixes for the actual install process of apache, SVN, and Trac. Each section has everything you need to get started and finished in quick order. The author is showing that the processes involved in code management do not have to be stressful or difficult to learn.

The author stresses several key ideas before getting into the meat of the content. These being “everything is a task”, “small steps are better”, “communication is key”, “content management is what software development requires for success.” Overall the two main division of labor is between task management and communication between team members.

Overall this is a very useful book on the subject of code management and documentation. I would recommend it to anyone looking to learn about the processes involved as well as anyone that wants a reference manual for growing their technology bookshelf.

I’ve recently been migrating my wiki/documentation for Kontrollbase to Trac. For those that are not aware, Trac is a web-based documentation/wiki/Subversion tool that is used by countless number of software projects. Subversion, of course, is a software collaboration and code management repository that manages branches/tags/trunk files with revision control. It’s one of the most heavily used open-source code repositories available. Given that I use SVN (subversion) for all of my software applications and am now using Trac, the book “Managing Software Development with Trac and Subversion” by David J Murphy comes as a useful and great resource for integrating these two useful tools.

The book is not a thousand plus pages type of computer manual. It gets the important points out in a very readable and organized method without winding the user through overly extensive examples. What you get with this book is everything you need to know about combining SVN with Trac without the fluff of other manuals.

In regard to the application of this book, it fits several groups of users; managers that need to understand how their software development team is managing code – or wants to implement a more productive system for their software team, the software developers themselves that need to get up to speed on these most important of technologies, as well as up and coming technologists that want to learn about the code development and management process. Its well roundedness is one of the best features.

The content is broken up into several sections: content management basics, Trac and SVN basics, trac and svn setup and integration, documentation creation and management, code management, and then very useful appendixes for the actual install process of apache, SVN, and Trac. Each section has everything you need to get started and finished in quick order. The author is showing that the processes involved in code management do not have to be stressful or difficult to learn.

The author stresses several key ideas before getting into the meat of the content. These being “everything is a task”, “small steps are better”, “communication is key”, “content management is what software development requires for success.” Overall the two main division of labor is between task management and communication between team members.

Overall this is a very useful book on the subject of code management and documentation. I would recommend it to anyone looking to learn about the processes involved as well as anyone that wants a reference manual for growing their technology bookshelf. You can find the book at the Packt Publishing website here.

Is emacs not coloring your Python comments?

This is a simple matter with a simple solution that might help someone save time and confusion. Emacs wasn’t coloring my comments correctly so I went ahead and had it change them to red-italic. If you are having similar issues you can drop the following into your home directory’s .emacs file. Enjoy. Keep in mind that if you are using emacs in a terminal session as opposed to the X-server gui then you will not see the italics.


(global-font-lock-mode 1)
(custom-set-variables
'(gud-gdb-command-name "gdb --annotate=1")
'(large-file-warning-threshold nil))
(custom-set-faces
'(font-lock-comment-face ((((class color) (background light)) (:foreground "red" :slant italic)))))

A simple webpage test script in Python

Looking around on Google for a webpage test script returns a lot of results. Some of them are useful, some are not. In particular, for Python, the scripts on the first page of results are minimal and lacking a useful copy and paste / ready to go script that will answer the question “is my webpage available?”. So I decided to write a quick one that will give you the return code and email you as an alert if the page does not return with a 200 code (successful). You can find the script here. Update: the webserver was trying to execute the script as a .py file so I just changed it to .txt – for it to work you will want to change the .txt extension to a .py extension after you download it.

If you are familiar with Python scripting, this script could easily be modified to post to a form so that you can test a MySQL transaction (or other transactional DB) to ensure your stack is running as needed. Note: wordpress is clobbering my tab indents on the code so if you opt to copy/paste this code into an editor please tab it out correctly or just download the script from the link above.


#!/usr/bin/python
################################################################################
## Kontrollkit
## NAME: kt-url-monitor.py
## DATE: 2010-02-24
## AUTHOR: Matt Reid
## WEBSITE: http://kontrollsoft.com
## EMAIL: support@kontrollsoft.com
## LICENSE: BSD http://www.opensource.org/licenses/bsd-license.php
################################################################################
## Copyright 2010-present Matt Reid
## All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# 3. The name of the author may not be used to endorse or promote products
# derived from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
# OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
# IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
# NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
# THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
##################################################################################
## WHAT THIS SCRIPT DOES #########################################################
## Purpose: Checks URL for valid page and emails if failed.
##################################################################################
## EDIT THE FOLLOWING AS NEEDED
email = "email@email.com" ##Email address to send alert to
host = "hostname.com" ##Hostname base URL without http://
port = "80" ##Port - 80 for http, 443 for https
url = "/index.html" ##Page to look for on host
SENDMAIL = "/usr/sbin/sendmail" ##Binary location for sendmail
logfile = "/var/log/kt-url-monitor.log" ##Logfile to write actions to
## END OF EDITABLE OPTIONS
##################################################################################
import sys
import os
import os.path
import datetime
from httplib import HTTP

def wfile(detail):
fstate = os.path.exists(logfile)
if(fstate == 0):
print "Logfile does not exist. Attempting to create..."
try:
file = open(logfile,"a+")
file.writelines(detail)
file.close()

except IOError:
print "Cannot open logfile for writing. Exiting."
sys.exit(1)

else:
print "Logfile exists, writing...\n"
file = open(logfile,"a+")
file.writelines(detail)
file.close()

def get(host,port,url):
concaturl = host+url
print "Checking Host:",concaturl
h = HTTP(host, port)
h.putrequest('GET', url)
h.putheader('Host', host)
h.putheader('User-agent', 'python-httplib')
h.endheaders()

(returncode, returnmsg, headers) = h.getreply()
if returncode != 200:
print returncode,returnmsg
return returncode

else:
f = h.getfile()
# return f.read() #returns content of page for further parsing
print returncode, returnmsg
return returncode

if __name__ == '__main__':
now = datetime.datetime.now()
date = now.strftime("%Y-%m-%d %H:%M:%S")
dmsg = date+" - URL verification starting - "
wfile(dmsg)
concaturl = "http://"+host+url
state = get(host,port,url)

if(state != 200):
msg = "Link (%s) is broken or unavailable. Return code: %s" % (concaturl,state)+"\n"
wfile(msg)
to = "To: "+email+"\n"
print msg,"\n"
p = os.popen("%s -t" % SENDMAIL, "w")
p.write(to)
p.write("Subject: Demo Server Status\n")
p.write("\n")
p.write(msg)
sts = p.close()
if sts != 0:
print "Sendmail exit status", sts
else:
msg = "Link (%s) is good. Return code: %s" % (concaturl,state)+"\n"
wfile(msg)
print msg+"\n"

Reviewed: RESTful PHP Web Services

I’ve been using a lot of RESTful services these days and have been waiting for a good book that is dedicated to the topic. I recently received a copy of ‘RESTful PHP Web Services’, which does a successful job of outlining proven concepts in current web technology. If you want to learn the methods for creating and consuming RESTful services then you will find many examples in this book. From the architectural plans to well thought out code samples, the book covers a lot of ground in a relatively quick read.

The first chapter gives the reader a quick introduction to RESTful services and the most common PHP frameworks in use at the time of writing. I particularly enjoyed the section on the Zend framework due to the explanation of benefits over the other frameworks. The chapter also covers the very basics which include a detailed look at exactly what RESTful services means and what technologies are required to use and benefit from a RESTful architecture. The second chapter gives a quick run down of the various methods in use for consumption of data; these being Curl, several HTTP methods, processing data with XML, DOM, and SimpleXML. After those are covered there is a simple example of consuming services like Flickr using the previous methods. This transitions into many more examples of consuming real world services that any developer would find interesting and exciting for data mashups.

The real meat of the book starts in chapter four where we get into designing the resource utilization systems and then the resource clients in chapter five. Those topics basically go over the nuts and bolts of gathering data, manipulating it, updating it, as well as creating fresh data. We get more instruction and usage examples on the Zend framework in chapter seven where the author gives us information on the controllers, models, and view (MVC model). This would not be too useful without knowing how to debug the code that we’re using so there is, thankfully, a chapter dedicated to debugging XML building and parsing errors. A couple of short appendixes cover the author’s own WSO2 web service framework as well as REST Client Classes which should prove useful for writing your own reusable classes.

Overall this book covers the majority of topics that a new developer needs to understand in order to start developing and deploying RESTful code and web services in PHP. From frameworks to consumable service samples, and everything in between, RESTful PHP Web Services comes through in a concise and enjoyable style that will not disappoint. I highly recommend this book for developers that are new to this topic or experienced developers that need a quick refresher course.