file locking using a context manager (with statement) in python

I needed a quick locking mechanism to prevent my daemons from stepping over each other. To have a sane daemon startup (and prevent multiple daemon spawns), we need to ensure that we have an exclusive lock before starting the program.  Googling around didn't lead to show any context managers that actually use the flock syscalls.

So here goes my attempt that seems to work:

Spinning off some python processes that utilise this context manager shows serialisation taking place:
And here's the output of lsof showing locking for the processes spun off above:

Playing with data from NOAA Global Forecast System using python

Bugger! Blogger barfs out an error when I attempt to upload my ipython notebook. You can find the notebook on github or as a rendered html here.

Here are some pretty generated images:

Horn of Africa zoomed in (Mar 27 2013)

Global 2M temperature in degrees C (Mar 27 2013)

Dirt cheap steganography

This steg module is based on code, ideas and images from http://blog.wolfram.com/2010/07/08/doing-spy-stuff-with-mathematica/ .
It requires numpy & matplotlib but can be reworked to depend on pypng instead of matplotlib

Sample usage

$ python2.7 simple_decode.py --help

Usage:

Hide/Unhide info in images as described in this post:

http://blog.wolfram.com/2010/07/08/doing-spy-stuff-with-mathematica/

usage: simple_decode.py [options]

Options:

-h, --help show this help message and exit

-v DEBUG, --verbose=DEBUG

Debug. Higher integers increases verbosity

-e ENCODED_FILE, --encoded_file=ENCODED_FILE

Encoded PNG file to work with. Default is

steg_chicken_secret.png

-d DATA_FILE, --data_file=DATA_FILE

File with decoded data or with data to encode.

Example decode of data hidden in the wolfram blog images

$ python2.7 simple_decode.py -e steg_chicken_secret.png -d chicken.txt
$ cat chicken.txt ;echo

This is a secret message.



$ python2.7 simple_decode.py -e alice.png -d alice.txt

$ file alice.txt

alice.txt: UTF-8 Unicode English text, with very long lines, with no line terminators



$ python2.7 simple_decode.py -e finalImage.png -d data.jpg

$ file data.jpg

data.jpg: JPEG image data, JFIF standard 1.01, comment: "Created by Wolfram Mathematica "

XMLRPC Register instance is stateful. Do not want that

I have an XMLserver that uses the method SimpleXMLRPCServer.register_instance to register a class and it's methods. Something like:

....    
    inst = ServiceClass(init_params)
    server.register_instance(inst)
    server.register_introspection_functions()

    try:
        server.serve_forever()
    finally:
        server.server_close()

The serviceclass looks like:

class ServiceClass(object):
    """ServiceClass calls the xxxx executable and returns stdout & stderr"""
    def __init__(self, params):
        super(ServiceClass, self).__init__()
        self.params = params
        print "Init with %s" % self.params

    def _shell_exec(self, cmd=[]):
        .......
        ....
...
    def dbupdate(self, sometext=""):
        """Given an object, try call the xxxx binary and
        return the output of the command
        Keyword arguments:                                                                                                                 
        params:                                                                                                                            
            sometext -- text of the object
        """
        if not len('sometext'):
            raise MissingObjectError("No text passed for command")

        # Create a tempfile
        (tmpfd, fname) = tempfile.mkstemp()
        os.write(tmpfd,  sometext)
        os.close(tmpfd)
        # Execute
        cmd_exe = self.params['cmd']                                                                                                       
        cmd_exe.extend([fname])
        ret_stdout, ret_stderr = self._shell_exec(cmd_exe)

        # Cleanup file
        os.unlink(fname)

        print "cmd: %s\nself.cmd: %s\nfname: %s\n" %(cmd_exe, self.params['cmd'], fname)
    
        return (ret_stdout, ret_stderr)

Turns out that python keeps state since... the instance is created only once at server startup. All subsequent calls share this instance. And that affects the class variables. Notice that tempfiles from a prior run are visible to a later run which might be from a different client. :(

:!python some_webservice_xmlrpc.py -d 1 -p 8000                                                                                        
Listening on :8000...
Service URI is http://:8000/
Use Control-C to exit
CMD: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpkCXRS-']
cmd: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpkCXRS-']
self.cmd: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpkCXRS-']
fname: /tmp/tmpkCXRS-

machine1.example.net - - [21/Apr/2011 14:53:08] "POST / HTTP/1.0" 200 -
CMD: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpkCXRS-', '/tmp/tmpBGDHHU']
cmd: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpkCXRS-', '/tmp/tmpBGDHHU']
self.cmd: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpkCXRS-', '/tmp/tmpBGDHHU']

I tried different solutions from del cmd prior to return to initializing all variables to None|"" at the start of the method. Didn't work.. till I remembered shallow copying.   With a deep copy,

....                                                   
#        cmd_exe.extend([fname])
#Replace with
        cmd_exe = copy.deepcopy(self.params['cmd'])
....

things behave as expected:

Listening on :8000...
Service URI is http://:8000/
Use Control-C to exit
Init with {'debug': 1, 'cmd': ['/path/to/executable', '-c', '/path/to/config.conf', '-f']}
CMD: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpLTR0-S']
cmd: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpLTR0-S']
self.cmd: ['/path/to/executable', '-c', '/path/to/config.conf', '-f']
fname: /tmp/tmpLTR0-S

machine1.example.net - - [21/Apr/2011 14:58:19] "POST / HTTP/1.0" 200 -
CMD: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpxdCOtw']
cmd: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpxdCOtw']
self.cmd: ['/path/to/executable', '-c', '/path/to/config.conf', '-f']
fname: /tmp/tmpxdCOtw

machine1.example.net - - [21/Apr/2011 14:58:26] "POST / HTTP/1.0" 200 -

Importing old data into munin

This python script tries to figure out the original rrdtool create parameters that were used to create a given rrd. It's very basic (handles only RRA:Average afaik) so don't expect magic from it. It expects rrdtool to be your path.

Sample invocation:

lmwangi@jaghpu:~/rrd$ python rrdinfo-parser.py -f all.rrd

rrdtool create all.rrd --start 978300900 --step 300 \

DS:a:COUNTER:600:U:U \

DS:c:DERIVE:600:U:U \

DS:b:GAUGE:600:U:U \

DS:d:ABSOLUTE:600:U:U \

RRA:AVERAGE:0.5:1:10 \

lmwangi@jaghpu:~/rrd$ python rrdinfo-parser.py -f test.rrd

rrdtool create test.rrd --start 920804400 --step 300 \

DS:speed:COUNTER:600:U:U \

RRA:AVERAGE:0.5:1:24 \

RRA:AVERAGE:0.5:6:10 \

So say we have a Munin derived rrd that we need to import old data into. First we run the script to extract the schema, then we edit the --start parameter to an epoch timestamp that predates your data and finally recreate the rrd.

rrdtool create service-logs-total-g.rrd --start 1108598400 --step 300 \                                                   
       DS:42:GAUGE:600:U:U \
       RRA:AVERAGE:0.5:1:576 \
       RRA:MIN:0.5:1:576 \
       RRA:MAX:0.5:1:576 \
       RRA:AVERAGE:0.5:6:432 \
       RRA:MIN:0.5:6:432 \
       RRA:MAX:0.5:6:432 \
       RRA:AVERAGE:0.5:24:540 \
       RRA:MIN:0.5:24:540 \
       RRA:MAX:0.5:24:540 \
       RRA:AVERAGE:0.5:288:450

Then we import the processed data (in my case, log file summaries) which looks like:

head 2010.data
1262296800:4241:221:173:276
1262297400:3920:197:155:231
1262298000:4171:184:226:208
1262298600:3700:197:159:244
1262299200:3350:195:166:227

where the fields are ts:total:errors:unknowns:etc.

Using a simple awk script we extract field one and two (timestamp and total) and import them into the rrd and when we are done, overwrite the original rrd.

FILE=service-logs-total-g.rrd
for record in $(cat 2010.data|awk -F: '{print $1":"$2}');
do 
 rrdupdate $FILE $record;
done

Google Translate using Python

A simple google translator for your console: It uses the labs api described here

http://code.google.com/apis/language/translate/v2/using_rest.html. The latest code is here.

#!/usr/bin/env python
# vim: tabstop=4 expandtab shiftwidth=4 softtabstop=4 textwidth=79 autoindent

"""
Python source code

Simple translator using the google translate API.
Example run:
    $ python translate1.py  -t sw "Hello. Are we meeting today?"
    Hello. Sisi ni mkutano wa leo?
"""
import urllib2
import json
import optparse

# API KEY
API_KEY = "Replace with your Key"
API_URL = "https://www.googleapis.com/language/translate/v2?\
key=%s&q=%s&source=%s&target=%s&prettyprint=false"

#Defaults
DEFAULT_SRC_LANG = "en"
DEFAULT_DEST_LANG = "fr"


class Translate(object):

"""Translate: Uses the google api to translate a string from one language
    to another
    """
def __init__(self):
super(Translate, self).__init__()
self.langs = ["af", "sq", "ar", "be", "bg", "ca", "zh-CN", "zh-TW",
"hr", "cs", "da", "nl", "en", "et", "tl", "fi", "fr",
"gl", "de", "el", "ht", "iw", "hi", "hu", "is", "id",
"ga", "it", "ja", "lv", "lt", "mk", "ms", "mt", "no",
"fa", "pl", "pt", "ro", "ru", "sr", "sk", "sl", "es",
"sw", "sv", "th", "tr", "uk", "vi", "cy", "yi"]

self.uri = API_URL

def translate(self, params):
"""Translates texts
        keywords:
            params - Dictionary
                src_text - String
                src_lang - 2 letter iso code for language
                dest_lang - 2 letter iso code for language
        """
req_uri = self.uri % (API_KEY, urllib2.quote(params['src_text']),
params['src_lang'],
params['dest_lang'])

hdl = urllib2.urlopen(req_uri)
resp = hdl.read()
hdl.close()
j = json.loads(resp)
try:
return j['data']['translations'][0]['translatedText']
except TypeError:
return ""


def check_lang(option, opt_str, value, parser):
""" Callback for optparse. Verifies that value is an item in a list"""
translator = Translate()
langs = translator.langs
if value not in langs:
raise optparse.OptionValueError(
"Invalid option: %s.\nLanguage not in %s" % (opt_str, langs))

setattr(parser.values, option.dest, value)


def main():
"""Main function. Called when this file is a shell script"""
translator = Translate()
usage = "usage: %prog [options] 'Text to translate'"
parser = optparse.OptionParser(usage)
parser.add_option("-f", "--from", action="callback",
callback=check_lang, dest="src_lang",
default=DEFAULT_SRC_LANG, type="string",
help="Translate from this language. Default is %default")

parser.add_option("-t", "--to", action="callback",
default=DEFAULT_DEST_LANG, type="string",
callback=check_lang, dest="dest_lang",
help="Translate to this language. Default is %default")

(options, args) = parser.parse_args()

params = {}
params['src_lang'] = options.src_lang
params['dest_lang'] = options.dest_lang
params['src_text'] = args[0]
dest_text = translator.translate(params)
print dest_text

if __name__ == '__main__':
main()

VirtualEnv and CherryPy with MySQL-python

Make some directories
$ mkdir -p ~/project/src ~/project/builds && cd ~/project/src

Grab virtualenv
$ wget http://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.5.1.tar.gz#md5=3daa1f449d5d2ee03099484cecb1c2b7

Build virtualenv
$ cd ../builds/

$ tar zxf ../src/virtualenv-1.5.1.tar.gz

$ cd virtualenv-1.5.1/

$ python virtualenv.py ~/project/virtual_python
New python executable in /home/laban/project/virtual_python/bin/python
Installing setuptools.............done.

$ ls ~/project/virtual_python
bin include lib lib64


So let's install CherryPy
$ ~/project/virtual_python/bin/easy_install cherrypy
Searching for cherrypy
Reading http://pypi.python.org/simple/cherrypy/
Reading http://www.cherrypy.org
Reading http://download.cherrypy.org/cherrypy/3.1.0/
Reading http://sourceforge.net/project/showfiles.php?group_id=56099
Reading http://download.cherrypy.org/cherrypy/3.1.0rc1/
Reading http://download.cherrypy.org/cherrypy/3.0.1/
Reading http://download.cherrypy.org/cherrypy/3.1.0beta3/
Reading http://download.cherrypy.org/cherrypy/3.0.3/
Reading http://download.cherrypy.org/cherrypy/3.0.0/
Reading http://download.cherrypy.org/cherrypy/2.2.1/
Reading http://download.cherrypy.org/cherrypy/3.1.2/
Reading http://download.cherrypy.org/cherrypy/3.1.1/
Reading http://download.cherrypy.org/cherrypy/3.0RC1/
Reading http://trac.cherrypy.org/cgi-bin/trac.cgi/wiki/CherryPyDownload
Reading http://download.cherrypy.org/cherrypy/3.1beta/
Reading http://download.cherrypy.org/cherrypy/3.0.2/
Reading http://download.cherrypy.org/cherrypy/2.3.0/
Reading http://download.cherrypy.org/cherrypy/3.0.4/
Best match: CherryPy 3.1.2
Downloading http://download.cherrypy.org/cherrypy/3.1.2/CherryPy-3.1.2.zip
Processing CherryPy-3.1.2.zip
Running CherryPy-3.1.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-B4VfG0/CherryPy-3.1.2/egg-dist-tmp-EFtfdH
zip_safe flag not set; analyzing archive contents...
cherrypy._cptree: module references __file__
cherrypy._cpmodpy: module references __file__
cherrypy.lib.profiler: module references __file__
cherrypy.lib.covercp: module references __file__
cherrypy.process.plugins: module references __file__
cherrypy.test.test_states: module references __file__
cherrypy.test.test_logging: module references __file__
cherrypy.test.test_core: module references __file__
cherrypy.test.checkerdemo: module references __file__
cherrypy.test.test_misc_tools: module references __file__
cherrypy.test.test_routes: module references __file__
cherrypy.test.modpy: module references __file__
cherrypy.test.benchmark: module references __file__
cherrypy.test.test_config: module references __file__
cherrypy.test.test_tidy: module references __file__
cherrypy.test.test_wsgiapps: module references __file__
cherrypy.test.test: module references __file__
cherrypy.test.test_virtualhost: module references __file__
cherrypy.test.modwsgi: module references __file__
cherrypy.test.test_session: module references __file__
cherrypy.test.modfcgid: module references __file__
cherrypy.test.helper: module references __file__
cherrypy.test.test_caching: module references __file__
cherrypy.test.test_static: module references __file__
cherrypy.scaffold.__init__: module references __file__
cherrypy.tutorial.tut09_files: module references __file__
cherrypy.tutorial.tut06_default_method: module references __file__
cherrypy.tutorial.tut07_sessions: module references __file__
cherrypy.tutorial.tut02_expose_methods: module references __file__
cherrypy.tutorial.tut01_helloworld: module references __file__
cherrypy.tutorial.tut03_get_and_post: module references __file__
cherrypy.tutorial.tut05_derived_objects: module references __file__
cherrypy.tutorial.tut04_complex_site: module references __file__
cherrypy.tutorial.tut10_http_errors: module references __file__
cherrypy.tutorial.tut08_generators_and_yield: module references __file__
Adding CherryPy 3.1.2 to easy-install.pth file
Installing cherryd script to /home/laban/project/virtual_python/bin

Installed /home/laban/project/virtual_python/lib/python2.4/site-packages/CherryPy-3.1.2-py2.4.egg
Processing dependencies for cherrypy
Finished processing dependencies for cherrypy

OK, something a little more complex

$ ~/project/virtual_python/bin/easy_install MySQL-python
Searching for MySQL-python
Reading http://pypi.python.org/simple/MySQL-python/
Reading http://sourceforge.net/projects/mysql-python/
Reading http://sourceforge.net/projects/mysql-python
Best match: MySQL-python 1.2.3
Downloading http://download.sourceforge.net/sourceforge/mysql-python/MySQL-python-1.2.3.tar.gz
Processing MySQL-python-1.2.3.tar.gz
Running MySQL-python-1.2.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-PvF-1-/MySQL-python-1.2.3/egg-dist-tmp-MOqY_x
warning: no files found matching 'MANIFEST'
warning: no files found matching 'ChangeLog'
warning: no files found matching 'GPL'
In file included from _mysql.c:29:
pymemcompat.h:10:20: error: Python.h: No such file or directory
_mysql.c:30:26: error: structmember.h: No such file or directory
....

Fixed by installing the python-devel rpm
Second try :

$ ~/project/virtual_python/bin/easy_install MySQL-python
Searching for MySQL-python
Reading http://pypi.python.org/simple/MySQL-python/
Reading http://sourceforge.net/projects/mysql-python/
Reading http://sourceforge.net/projects/mysql-python
Best match: MySQL-python 1.2.3
Downloading http://download.sourceforge.net/sourceforge/mysql-python/MySQL-python-1.2.3.tar.gz
Processing MySQL-python-1.2.3.tar.gz
Running MySQL-python-1.2.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-30_Yax/MySQL-python-1.2.3/egg-dist-tmp-O4DYQW
warning: no files found matching 'MANIFEST'
warning: no files found matching 'ChangeLog'
warning: no files found matching 'GPL'
In file included from /usr/include/python2.4/Python.h:8,
from pymemcompat.h:10,
from _mysql.c:29:
/usr/include/python2.4/pyconfig.h:6:25: error: pyconfig-64.h: No such file or directory
In file included from /usr/include/python2.4/Python.h:55,
from pymemcompat.h:10,
from _mysql.c:29:
/usr/include/python2.4/pyport.h:612:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
_mysql.c: In function ‘_mysql_ConnectionObject_affected_rows’:
_mysql.c:724: warning: implicit declaration of function ‘PyLong_FromUnsignedLongLong’
_mysql.c:724: warning: return makes pointer from integer without a cast
_mysql.c: In function ‘_mysql_ConnectionObject_insert_id’:
_mysql.c:1704: warning: return makes pointer from integer without a cast
_mysql.c: In function ‘_mysql_ResultObject_num_rows’:
_mysql.c:1774: warning: return makes pointer from integer without a cast
error: Setup script exited with error: command 'gcc' failed with exit status 1


Turns out that I had a 32bit python-devel package instead of the 64 bit.
One more try
$ ~/project/virtual_python/bin/easy_install MySQL-python
Processing MySQL-python-1.2.3.tar.gz
Running MySQL-python-1.2.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-E9CKk4/MySQL-python-1.2.3/egg-dist-tmp-tO6Qjb
warning: no files found matching 'MANIFEST'
warning: no files found matching 'ChangeLog'
warning: no files found matching 'GPL'
zip_safe flag not set; analyzing archive contents...
Adding MySQL-python 1.2.3 to easy-install.pth file

Installed /home/laban/project/virtual_python/lib/python2.4/site-packages/MySQL_python-1.2.3-py2.4-linux-x86_64.egg
Processing dependencies for MySQL-python==1.2.3
Finished processing dependencies for MySQL-python==1.2.3

Now on to some coding :)

python-ldap error

I kept running into a

{'desc': 'Bad parameter to an ldap routine'}

error while trying to add a record from django. Turns out that in the form validation, I had a few fields defined as IntegerFields which means i was passing

('uidNumber', 5001)

instead of

('uidNumber', '5001')

to python-ldap. Casting to an ascii string works. Unicode doesn't