Simple Python: a job queue with threading

Every so often you need to use a queue to manage operations in an application. Python makes this very simple. Python also, as I’ve written about before, makes threading very easy to work with. So in this quick program I’ll describe via comments, how to make a simple queue where each job is processed by a thread. Integrating this code to read jobs from a mysql database would be trivial as well; simply replace the “jobs = [..." code with a database call to a row select query.

 
#!/usr/bin/env python
## DATE: 2011-01-20
## FILE: queue.py
## AUTHOR: Matt Reid
## WEBSITE: http://themattreid.com
from Queue import *
from threading import Thread, Lock

'''this function will process the items in the queue, in serial'''
def processor():
    if queue.empty() == True:
        print "the Queue is empty!"
        sys.exit(1)
    try:
        job = queue.get()
        print "I'm operating on job item: %s"%(job)
        queue.task_done()
    except:
        print "Failed to operate on job"

'''set variables'''
queue = Queue()
threads = 4
    
'''a list of job items. you would want this to be more advanced,
like reading from a file or database'''
jobs = [ "job1", "job2", "job3" ]

”’iterate over jobs and put each into the queue in sequence”’
for job in jobs:
     print “inserting job into the queue: %s”%(job)
     queue.put(job)

”’start some threads, each one will process one job from the queue”’
for i in range(threads):
     th = Thread(target=processor)
     th.setDaemon(True)
     th.start()

”’wait until all jobs are processed before quitting”’
queue.join() 

8 thoughts on “Simple Python: a job queue with threading

  1. [...] This post was mentioned on Twitter by 何李石, Python UK. Python UK said: Simple Python: a job queue with threading | {themattreid} http://bit.ly/eJF4I2 [...]

  2. Jay Pipes says:

    Hi Matt!

    Due to the GIL, threading in Python may not be the best solution. I’d recommend taking a look at both the multiprocessing module as well as Eventlet, which uses coroutines to handle massive concurrency and has a simpler interface than something like Twisted…

    Cheers!
    jay

  3. admin says:

    @Jay, that’s a great idea. I’ve been playing around with it for a bit, maybe I’ll post an example of that method next.

  4. dimios says:

    dows anyone have an example how to use this with real jobs?

  5. admin says:

    What kind of example do you want to see? I have plenty of code I can share.

  6. dimios says:

    thanks god. I am trying to anderstand how all these works about queues with python. I have to build a simple system which allows a user to submit jobs to a queue (or queues). Jobs from the queue should be executed on one of a number of worker machines in an appropriate order.

    The problem is that I cant find any information on internet relaited to that. I have found more informations about the network-part of this project and I have developed a server/client where they sent and receive messages, all in python, between two machines in the same network.

    I want to see an example of code as yours but where you can use it to do something real, not a test. i.e. renaming files or something else, I don’t know.

    I have much more question is there anyway we can chat on gmail or skype? thanks in advance

  7. dimios says:

    please any example or help?

  8. Marcus says:

    The example demonstrates the idea clear enuf. What real life example u folks want? Gee, if u cant understand these simple examples and use it in your actual apps, consider “not” being a programmer.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>