Is emacs not coloring your Python comments?

This is a simple matter with a simple solution that might help someone save time and confusion. Emacs wasn’t coloring my comments correctly so I went ahead and had it change them to red-italic. If you are having similar issues you can drop the following into your home directory’s .emacs file. Enjoy. Keep in mind that if you are using emacs in a terminal session as opposed to the X-server gui then you will not see the italics.


(global-font-lock-mode 1)
(custom-set-variables
'(gud-gdb-command-name "gdb --annotate=1")
'(large-file-warning-threshold nil))
(custom-set-faces
'(font-lock-comment-face ((((class color) (background light)) (:foreground "red" :slant italic)))))

Comments

A simple webpage test script in Python

Looking around on Google for a webpage test script returns a lot of results. Some of them are useful, some are not. In particular, for Python, the scripts on the first page of results are minimal and lacking a useful copy and paste / ready to go script that will answer the question “is my webpage available?”. So I decided to write a quick one that will give you the return code and email you as an alert if the page does not return with a 200 code (successful). You can find the script here. Update: the webserver was trying to execute the script as a .py file so I just changed it to .txt – for it to work you will want to change the .txt extension to a .py extension after you download it.

If you are familiar with Python scripting, this script could easily be modified to post to a form so that you can test a MySQL transaction (or other transactional DB) to ensure your stack is running as needed. Note: wordpress is clobbering my tab indents on the code so if you opt to copy/paste this code into an editor please tab it out correctly or just download the script from the link above.


#!/usr/bin/python
################################################################################
## Kontrollkit
## NAME: kt-url-monitor.py
## DATE: 2010-02-24
## AUTHOR: Matt Reid
## WEBSITE: http://kontrollsoft.com
## EMAIL: support@kontrollsoft.com
## LICENSE: BSD http://www.opensource.org/licenses/bsd-license.php
################################################################################
## Copyright 2010-present Matt Reid
## All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# 3. The name of the author may not be used to endorse or promote products
# derived from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
# OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
# IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
# NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
# THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
##################################################################################
## WHAT THIS SCRIPT DOES #########################################################
## Purpose: Checks URL for valid page and emails if failed.
##################################################################################
## EDIT THE FOLLOWING AS NEEDED
email = "email@email.com" ##Email address to send alert to
host = "hostname.com" ##Hostname base URL without http://
port = "80" ##Port - 80 for http, 443 for https
url = "/index.html" ##Page to look for on host
SENDMAIL = "/usr/sbin/sendmail" ##Binary location for sendmail
logfile = "/var/log/kt-url-monitor.log" ##Logfile to write actions to
## END OF EDITABLE OPTIONS
##################################################################################
import sys
import os
import os.path
import datetime
from httplib import HTTP

def wfile(detail):
fstate = os.path.exists(logfile)
if(fstate == 0):
print "Logfile does not exist. Attempting to create..."
try:
file = open(logfile,"a+")
file.writelines(detail)
file.close()

except IOError:
print "Cannot open logfile for writing. Exiting."
sys.exit(1)

else:
print "Logfile exists, writing...\n"
file = open(logfile,"a+")
file.writelines(detail)
file.close()

def get(host,port,url):
concaturl = host+url
print "Checking Host:",concaturl
h = HTTP(host, port)
h.putrequest('GET', url)
h.putheader('Host', host)
h.putheader('User-agent', 'python-httplib')
h.endheaders()

(returncode, returnmsg, headers) = h.getreply()
if returncode != 200:
print returncode,returnmsg
return returncode

else:
f = h.getfile()
# return f.read() #returns content of page for further parsing
print returncode, returnmsg
return returncode

if __name__ == '__main__':
now = datetime.datetime.now()
date = now.strftime("%Y-%m-%d %H:%M:%S")
dmsg = date+" - URL verification starting - "
wfile(dmsg)
concaturl = "http://"+host+url
state = get(host,port,url)

if(state != 200):
msg = "Link (%s) is broken or unavailable. Return code: %s" % (concaturl,state)+"\n"
wfile(msg)
to = "To: "+email+"\n"
print msg,"\n"
p = os.popen("%s -t" % SENDMAIL, "w")
p.write(to)
p.write("Subject: Demo Server Status\n")
p.write("\n")
p.write(msg)
sts = p.close()
if sts != 0:
print "Sendmail exit status", sts
else:
msg = "Link (%s) is good. Return code: %s" % (concaturl,state)+"\n"
wfile(msg)
print msg+"\n"

Comments

RESTful PHP Web Services – reviewed

I’ve been using a lot of RESTful services these days and have been waiting for a good book that is dedicated to the topic. I recently received a copy of ‘RESTful PHP Web Services’, which does a successful job of outlining proven concepts in current web technology. If you want to learn the methods for creating and consuming RESTful services then you will find many examples in this book. From the architectural plans to well thought out code samples, the book covers a lot of ground in a relatively quick read.

The first chapter gives the reader a quick introduction to RESTful services and the most common PHP frameworks in use at the time of writing. I particularly enjoyed the section on the Zend framework due to the explanation of benefits over the other frameworks. The chapter also covers the very basics which include a detailed look at exactly what RESTful services means and what technologies are required to use and benefit from a RESTful architecture. The second chapter gives a quick run down of the various methods in use for consumption of data; these being Curl, several HTTP methods, processing data with XML, DOM, and SimpleXML. After those are covered there is a simple example of consuming services like Flickr using the previous methods. This transitions into many more examples of consuming real world services that any developer would find interesting and exciting for data mashups.

The real meat of the book starts in chapter four where we get into designing the resource utilization systems and then the resource clients in chapter five. Those topics basically go over the nuts and bolts of gathering data, manipulating it, updating it, as well as creating fresh data. We get more instruction and usage examples on the Zend framework in chapter seven where the author gives us information on the controllers, models, and view (MVC model). This would not be too useful without knowing how to debug the code that we’re using so there is, thankfully, a chapter dedicated to debugging XML building and parsing errors. A couple of short appendixes cover the author’s own WSO2 web service framework as well as REST Client Classes which should prove useful for writing your own reusable classes.

Overall this book covers the majority of topics that a new developer needs to understand in order to start developing and deploying RESTful code and web services in PHP. From frameworks to consumable service samples, and everything in between, RESTful PHP Web Services comes through in a concise and enjoyable style that will not disappoint. I highly recommend this book for developers that are new to this topic or experienced developers that need a quick refresher course.

Comments

New book reviews in the works

Due to the interest in my previous review on the ExtJS 3.0 Cookbook, I’ve been requested to continue those efforts. Stay tuned for reviews of the following books:

Comments

ExtJS 3.0 Cookbook Review

There is a quantity of snow falling outside as I write this and that means it’s a good day to read programming books. I’ve recently been enjoying the ExtJS 3.0 Cookbook by Jorge Ramon that is published by the Packt group. I’ve been using ExtJS for the last year and have experienced the transition from version 2.0 to 3.0 and have been rather pleased with the results it has delivered for numerous web projects I’ve coded. After trying many other JS kits and frameworks including Jquery, Mootools, and several others I have generally settled on ExtJS to solve the layout and user interface parts of my applications because of its ease of use and flexible nature for expansion. The framework is an extension of the Yahoo javascript framework, otherwise known as YUI, but with many quality extensions. This book comes as a welcome guide to the general tasks that the framework will solve as well as some of the more unique and in-depth items that it can provide. Quite simply it’s a book with over a hundred examples of all the best features of the ExtJS framework. The examples are well explained with detailed write-ups that are fun to read. The examples also have immediately usable code for dropping into place or modifying for your own needs. If you’re interested in learning more about this amazing framework or are already acquainted and need a quick guide to expand your abilities, then this will be a good reference to own. Before I get going you can see the book here.

The book starts out with an overview of the way that ExtJS handles DOM and data types, which is all very basic stuff but required understanding as well. It’s a good start to understanding the base elements if you are a bit rusty. It’s also a good chapter to see how ExtJS handles them since it might not be what you are used to if coming from another framework. After enough pages to answer most typical questions of that topic the book dives into the UI building aspects of the framework. This is where things get really good as I’ve always enjoyed how the framework builds UI for the web. You get examples for containers and positioning, accordion functionality, card and tab based windowing, column and table layouts, and the windowing functions for full browser UI building. I’ve used many of these features all in one layout and the results are a effective and efficient manner of data distribution and information organization for content driven sites. The book even covers dynamic building of UI elements, which comes in handy for a user interface that grows and changes with the needs of the user.

After you build your UI you need methods to get content into the application and ways that the user can interact with the app. The book addresses these needs as well. Plenty of examples abound for all of the required manners that users will be using forms, date fields, combo boxes, remote data loading via JSON and XML, as well as covering grid panels and data tables. Of course it must be mentioned that there are numerous examples that discuss the AJAX features of ExtJS since this is one of it’s major strengths and successes. If there’s a way to use remote data calls via XML or JSON, it is covered in this book. We also get a nice run down of trees and tab data which is inevitably useful for many purposes. If you are interested in progress bars and customized progress circles the book covers those with over 20 pages of examples. Just before the topics are finished off Jorge covers the functions for building rich charts and graphs. Details involve static data display, remote data display, as well as auto-refresh of data for realtime charting. Patterns, class extensions, plugins, and ‘keeping state’ are the last topics of the book. They offer quality methods for customizing ExtJS with your own JS code and keeping the user’s data consistent between sessions.

Overall this is one of the best books I’ve read about ExtJS. It’s as if the author took all of the items that I had to research over the last year of using the framework and put the solutions all into a nicely readable form with quality code examples that are explained in a useful manner. If you are looking for a book that touches upon the most common and some of the more esoteric options of the ExtJS framework but don’t feel like scouring the online forums and hundreds of pages of documentation then this is the book for you. If you haven’t already built your own code for the features that the chapters discuss then the examples given will save you time and get your projects up and running fast and efficiently.

Comments (1)

Sick of table locks during backups?

I sure am! That is all. Please continue with whatever you were doing.

Comments

DDR-Drive gets 300,000 IOPS

Sure it’s expensive and hard to justify for most budgets but this is the future of drive technology happening now. http://www.ddrdrive.com

Comments

Dirty growth-over-time query

I’ve been messing around with the Kontrollbase schema for the last couple of days, writing various queries for the daily reporting scripts that will eventually be an automated pdf report. I’ll give you examples of two of the queries, the first being overall environment stats, and the second being single-host growth over time.

Overall environment stats
select ((((MAX(os_mem_used)) / 1024 ) / 1024) / 1024) max_os_mem_used, ((((MIN(os_mem_used)) / 1024 ) / 1024) / 1024) min_os_mem_used, ((((AVG(os_mem_used)) / 1024 ) / 1024) / 1024) avg_os_mem_used, ((((STDDEV_POP(os_mem_used)) / 1024 ) / 1024) / 1024) stdev_os_mem_used, ((((MAX(length_data + length_index)) / 1024 ) / 1024) / 1024) max_size, ((((MIN(length_data + length_index)) / 1024 ) / 1024) / 1024) min_size, ((((AVG(length_data + length_index)) / 1024 ) / 1024) / 1024) avg_size, ((((STDDEV_POP(length_data + length_index)) / 1024 ) / 1024) / 1024) stdev_size, MAX(num_connections) max_connections, MIN(num_connections) min_connections, AVG(num_connections) avg_connections, STDDEV_POP(num_connections) stdev_connections, MAX(queries_per_second) max_qps, MIN(queries_per_second) min_qps, AVG(queries_per_second) avg_qps, STDDEV_POP(queries_per_second) stdev_qps from server_statistics;
*************************** 1. row ***************************
max_os_mem_used: 50.743488311768
min_os_mem_used: 0.023044586182
avg_os_mem_used: 1.5759627057922952
stdev_os_mem_used: 2.4064208226596184
max_size: 283.815660957247
min_size: 0.000492287800
avg_size: 10.8213909777686940
stdev_size: 39.9443980717105447
max_connections: 435
min_connections: 0
avg_connections: 16.9734
stdev_connections: 37.3269
stdev_os_mem_used: 2.4064208226596184
max_qps: 9243.6533203125
min_qps: 0.00011409764556447
avg_qps: 216.421064774444
stdev_qps: 1071.72792986232
1 row in set (0.00 sec)

single-host growth over time
mysql> select (select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and CURDATE() < = Creation_time order by Creation_time asc limit 1) as 0_day_size_mb, (select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and DATE_SUB(CURDATE(),INTERVAL 30 DAY) <= Creation_time order by Creation_time asc limit 1) as 30_day_size_mb, ( (select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and CURDATE() <= Creation_time order by Creation_time asc limit 1) - (select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and DATE_SUB(CURDATE(),INTERVAL 30 DAY) <= Creation_time order by Creation_time asc limit 1)) as difference, ( select (select ((select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and CURDATE() <= Creation_time order by Creation_time asc limit 1) - (select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and DATE_SUB(CURDATE(),INTERVAL 30 DAY) <= Creation_time order by Creation_time asc limit 1)) as difference) / (select(select (((length_data + length_index) / 1024) / 1024) as curr_size from server_statistics where server_list_id='44' and CURDATE() <= Creation_time order by Creation_time asc limit 1) as 0_day_size_mb ) * 100) as percent_growth;
+---------------+----------------+--------------+----------------+
| 0_day_size_mb | 30_day_size_mb | difference | percent_growth |
+---------------+----------------+--------------+----------------+
| 9100.43986130 | 8199.39263916 | 901.04722214 | 9.901139240197 |
+---------------+----------------+--------------+----------------+
1 row in set (0.01 sec)

Comments

IRC is the best support method and Open Source rules

I was working on a server today that was not hooked up to our usual monitoring systems for one reason or another and I needed to generate a database tuning report. Typically I use Matthew Montgomery’s ‘tuning-primer.sh’ script for this since it’s command line based, simple to use, and generates a number of useful items for tuning recommendations. It’s a great starting point before delving into the deeper aspects of MySQL and the OS.

I ran into an issue with it on this server that was running the MySQL 5.0.77-percona-highperf-b13-log x86_64 build. The error was:
./tuning-primer.sh.1: line 517: 5.000000: syntax error in expression (error token is ".000000")

There were three options to fix this issue

  1. Dive into the code and modify it cowboy style
  2. Use our typical monitoring against the client’s wishes
  3. Contact the developer to get a fix

I hopped on IRC.freenode.net to the #mysql channel and found Matthew online. Long story short he narrowed it down to the version of MySQL being from Percona that was changing the format of the variable in question vs the typical MySQL variable type. After discussing the matter he released a new build to address this change and I was on my way to generating reports again.That is quality support that you won’t see from the likes of Microsoft or Apple or any big closed-source player.

If there’s on thing to take away from this story it’s the Open Source Software will ALWAYS be better than closed source because of the hands on attitude and direct contact you can get with the developers or, at the minimum, with the large user community that is willing and able to help troubleshoot. Why are people able to help? Because the code is open and free – and people like free software that they can improve and fix themselves – and those people like to help others because we’ve all needed help at some point or another, no matter how much of an expert you are.

If there are two things to take away it’s to remember that IRC is a wealth of useful information and support for Open Source applications. You can find me on the #mysql, ##php, #perl, #extjs, and #codeigniter channels under various usernames – or idling in the #kontrollbase channel for supporting my own application.

Closed source apps are the old way to do business, the rotting steel skeletons from the industrial age of computing… Open Source is the brainy kid down the street that doesn’t want to rip you off for something that some nameless big corporation designed overseas for pennies on the dollar just to turn a profit and sell you something with crappy support and non-auditable code. Long live OSS!

Comments (6)

Vote for Kontrollbase on MySQL Forge

Just a reminder to all of those users that are enjoying Kontrollbase – if you get a minute in your day please go to the MySQL Forge site and put your vote in on Kontrollbase. It’s a simple star based vote on the right side of the page located here: http://forge.mysql.com/projects/project.php?id=318

Comments