Read a snippet of a file using vim

:0r ! sed -n 1,22p .procmailrc

• Don't insert a newline prior to the read (0)
• Read
• Use sed to get an address range of a file

And there was much celebration

Dirt cheap steganography

This steg module is based on code, ideas and images from http://blog.wolfram.com/2010/07/08/doing-spy-stuff-with-mathematica/ .
It requires numpy & matplotlib but can be reworked to depend on pypng instead of matplotlib

Sample usage

$ python2.7 simple_decode.py --help

Usage:

Hide/Unhide info in images as described in this post:

http://blog.wolfram.com/2010/07/08/doing-spy-stuff-with-mathematica/

usage: simple_decode.py [options]

Options:

-h, --help show this help message and exit

-v DEBUG, --verbose=DEBUG

Debug. Higher integers increases verbosity

-e ENCODED_FILE, --encoded_file=ENCODED_FILE

Encoded PNG file to work with. Default is

steg_chicken_secret.png

-d DATA_FILE, --data_file=DATA_FILE

File with decoded data or with data to encode.

Example decode of data hidden in the wolfram blog images

$ python2.7 simple_decode.py -e steg_chicken_secret.png -d chicken.txt
$ cat chicken.txt ;echo

This is a secret message.



$ python2.7 simple_decode.py -e alice.png -d alice.txt

$ file alice.txt

alice.txt: UTF-8 Unicode English text, with very long lines, with no line terminators



$ python2.7 simple_decode.py -e finalImage.png -d data.jpg

$ file data.jpg

data.jpg: JPEG image data, JFIF standard 1.01, comment: "Created by Wolfram Mathematica "

RPKI tools

An ISC implementation of the RPKI drafts can be found here http://www.rpki.net/.
Since all objects published in RPKI repository are in ASN1, dumpasn1 can be used to quickly peek at the objects. Here's an example.

Sample asn1 dump

RPKI Products: ROA

Current described by this draft, Route Origin Authorizations (ROAs) associate an AS number and a list prefixes in an object which is then CMS signed by a the owner of the prefixes. By generating a ROA, the owner of the prefix is stating that the as number in the ROA is allowed to originate the prefixes listed in the ROA. Third parties can then fetch ROAs, verify their cms signature and then use validation rules as per http://tools.ietf.org/wg/sidr/draft-ietf-sidr-roa-validation/. ROAs encapsulate end entity certificates within them. A sample ROA is shown below.

Sample ROA

RPKI Manifests

Manifests are currently documented by this sidr draft. Manifest are signed objects that have a list of all the objects in a repository (except the manifest itself) and their corresponding hash. A valid manifest helps to ascertain that the objects in a publication point are fresh (latest issued) and complete i.e. none are missing.
Similar to ROAs, manifests embed an End Entity (EE) which enforces the manifest validity period. A sample manifest is shown below.

Sample Manifest

RPKI CRLs

Each engine (rpki instance with a certificate) is expected to maintain an update to date CRL that is generated at regular intervals. The CRL records any keys revoked due to product expiry or security incidences. Each certificate points to it's parent CRL by using the X509v3 CRL Distribution Points attribute (It's the parent that issued a certificate. Consequently, revocation responsibility lies with the parent).
An example CRL is shown below. The important bits:

• Authority Key Identifier and Issuer point to the issuer (certificate in previous article).
• CRL number which is incremented with each issue.
• Last update and next update time constraints.
• Revocation number and date of each certificate identified by a serial number. 

Sample CRL

Certificate fingerprints

If you have self signed certs, you may run across some programs that require you to include the fingerprint of the remote server cert somewhere in your config.
An example application is mercurial (as shown below):

$ hg clone https://a.b.c.net/public/application/
abort: error: _ssl.c:497: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

The fix is easy:
Get the server certificate fingerprint (sha1 fingerprint of certificate)

$ echo quit |openssl s_client -connecta.b.c.net:443  2>/dev/null|awk '/---BEG/,/--END/' |openssl x509 -fingerprint -noout
SHA1 Fingerprint=69:C7:E4:E7:10:F8:8F:19:F9:5D:92:F3:8D:EC:CF:4F:7B:3E:04:0A

Then edit your ~/.hgrc

...
 [hostfingerprints]
a.b.c.net = 69:C7:E4:E7:10:F8:8F:19:F9:5D:92:F3:8D:EC:CF:4F:7B:3E:04:0A                                            
.....

That should fix things up.

Same thing can be done for fetchmail:

$ echo quit |openssl s_client -connect mail.xyz.net:143 -starttls imap 2>/dev/null|awk '/---BEG/,/--END/' |openssl x509 -md5 -fingerprint -noout
MD5 Fingerprint=AA:87:67:1A:21:16:50:57:3F:6C:D0:C8:E8:02:19:7C

then fixing your ~/.fetchmailrc

poll mail.xyz.net protocol imap
        user "user1" password "somepass123"       
        sslfingerprint  "AA:87:67:1A:21:16:50:57:3F:6C:D0:C8:E8:02:19:7C"                                                    
        mda "/usr/bin/procmail -d %T "

RPKI certificates

Resource certificates are defined in the SIDR resource certificate profile specification. In this second part of the series, we'll start analysing a set of certificates from the AfriNIC repository (rsync://rpki.afrinic.net) from the top. We'll visit the trust anchor locator, trust anchor certificate, RIR certificate and an LIR certificate. An important point to note is that each certificate is a CA certificate with the exception of embedded end entity certificates.

Trust Anchor Locator

A trust anchor as a trusted entry point of any PKI is expected to be stable over time. However, the current practice has RIRs running their own roots with their resources specified in the root cert.

If an RIR receives an allocation, they have to issue a new certificate that contains the new resources which will have to be propagated to validators. Taking the case of IPV4, 17 blocks were allocated in a span of a year (2009). This corresponds to 1 trust anchor change every 21 days on average. On top of this, certificates have time constraints which will induce certificate re-issues due to expiry.


The TAL format is specified in the SIDR TA draft to work around these issues. Essentially, the TAL contains the rsync URI of the root certificate with rfc3779 extensions and the public key component of that certificate. These two are separated using a [CR]LF According to the specification, the public key is ``a base 64-encoded, DER-encoded X.509 subjectPublicKeyInfo [RFC5280]''.

It can easily be generated by using shell utilities and OpenSSL.
Since the TAL doesn't have resources and time constraints, It can be long lived requiring change when the key is rolled.

Here is the AfriNIC TAL.

rsync://rpki.afrinic.net/repository/AfriNIC.cer
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxsAqAhWIO+ON2Ef9oRDM
pKxv+AfmSLIdLWJtjrvUyDxJPBjgR+kVrOHUeTaujygFUp49tuN5H2C1rUuQavTH
vve6xNF5fU3OkTcqEzMOZy+ctkbde2SRMVdvbO22+TH9gNhKDc9l7Vu01qU4LeJH
k3X0f5uu5346YrGAOSv6AaYBXVgXxa0s9ZvgqFpim50pReQe/WI3QwFKNgpPzfQL
6Y7fDPYdYaVOXPXSKtx7P4s4KLA/ZWmRL/bobw/i2fFviAGhDrjqqqum+/9w1hEl
L/vqihVnV18saKTnLvkItA/Bf5i11Yhw2K7qv573YWxyuqCknO/iYLTR1DToBZcZ
UQIDAQAB

Root certificate

The root certificate sits at the root of the Resource PKI. It contains all the resources suballocated in the tree below it. In an ideal world, there would be the one root certificate and it's number resources would be:

• 0/0 for IPV4
• 0/0 for IPV6
• 1-4294967296 for AS numbers

Unfortunately, in the current structure, each RIR is running it's own root and encoding it's resources only. So for example, the AfriNIC certificate will only contain IANA's allocation to AfriNIC.

In the figure on the first post, the first dashed rectangle represents the root repository. It contains all the certificates issued by the root, a CRL and a manifest for this repository. This structure is replicated across the entire PKI tree that is, each certificate has a repository that is accessible over rsync and contains: 

• A CRL.
• A manifest.
• Zero or more products (ROAs, AAAs...) in it's repository.
• Certificates for sub CAs.

An example certificate can be seen below. It's highlights are:

• Self signed which implies that it's a root certificate.
• The location of the repository and manifest are encoded in the certificate.
• The certificate has x509 CA extensions (Green sections).
• Has rfc3779 extensions (sbgp-* sections). 

AfriNIC root certificate

Resource certificate

The root certificate is a special case of the resource certificate. A non root resource certificate has:

• A parent - Authority Information Access in a certificate.
• Zero or more children - Subject Information Access is a pointer to a store with this certificate's children.
• Zero or more - Subject Information Access also points to a store with this certificate's products. products. 

 An example tree is shown below. In this case, the root certificate shown above issues the RIR certificate in figure below which in turn issues the LIR certificate in the last figure. Please note that relationship between the certificates as indicated by the colour coding.

RIR Certificate

LIR Certificate

Suppose I wanted to block facebook.

Suppose you have access to your firewall and you want to block Facebook.
Then you need to get their prefixes which you can find out their ASN.

You can find out their asn from traceroute:

$  traceroute -n -A www.facebook.com
....
14  4.69.149.82 [AS3356]  630.713 ms  633.180 ms 4.69.149.146 [AS3356]  629.633 ms
15  4.53.112.58 [AS3356]  578.959 ms  580.703 ms  579.624 ms
16  204.15.20.120 [AS32934/AS10753]  577.604 ms  557.247 ms  636.807 ms
17  74.119.76.67 [AS32934]  633.454 ms 74.119.76.186 [AS32934]  602.556 ms  603.583 ms
18  74.119.76.197 [AS32934]  634.461 ms 74.119.76.205 [AS32934]  634.431 ms 74.119.77.55 [AS32934]  702.079 ms
19  69.63.190.10 [AS32934/AS10753]  701.423 ms  700.644 ms  700.937 ms

Which you can verify with tools such as these

$ dig +short AS32934.asn.cymru.com TXT
"32934 | US | arin | 2004-08-24 | FACEBOOK - Facebook, Inc."

Then you'd get their prefixes using a lookup service

$ lynx --dump https://www.dan.me.uk/bgplookup?asn=32934 2>&1 | grep AS32934
   IPv4 Prefixes seen at AS32934:
     204.15.20.0/22       [AS32934]
     2620:0:1c00::/40     [AS32934]
     66.220.144.0/21      [AS32934]
     66.220.152.0/21      [AS32934]
     66.220.159.0/24      [AS32934]
     69.171.224.0/20      [AS32934]
     69.171.239.0/24      [AS32934]
     69.171.240.0/20      [AS32934]
     69.171.255.0/24      [AS32934]
     69.63.176.0/21       [AS32934]
     69.63.184.0/21       [AS32934]
     74.119.76.0/22       [AS32934]

You can then feed the prefixes into your firewall.
Of course, If I was a determined user, I could use any of the free proxies out there rendering this whole post moot (DPI anyone?).

Screen capture in X

This is the cheapest screen capture in town. It's also probably installed right now and you can get it to work in seconds.

Recipe:
Use xwd which is the X Window Dumper to capture your desktop/window.
Use ImageMagick's convert to convert the xwd into a png/jpg etc
Collate the images into a video using mencoder



xwd is found in  x11-apps under debian.
convert is part of the ImageMagick suite.
mencoder is part of the mplayer suite.

So open a terminal, cd into a work dir and start capturing screenshots at 5fps.

while :; 
do 
    xwd -root |convert - blah-$(date +%s.%N).png; 
    sleep 0.2; 
done
  

Then go do your stuff... Much later ctrl-c the while loop to terminate it and create your capture

mencoder "mf://*.png" -mf fps=5 -o output.avi -ovc lavc -lavcopts vcodec=mpeg4

You can also speed up your screencap by faking the fps. Make it 50 in the mencoder command above and see.

It should be pretty trivial to record a separate audio stream and add it to your video

Libvirt cheatsheet

Remote connection over ssh

virt-manager -c qemu+ssh://root@10.0.0.1/system

In the installation/cloning examples below, I precreated LV using lvcreate. The path's given point to the LVs (For example /dev/xen02/member.root)
Installing without virtio might end up with a slow VM.

virt-install --name=member \
--ram 1024 \
--os-type linux --os-variant debianlenny \
--file /dev/xen02/member.root \
--network network=default \
--vnc \
--noautoconsole \
--hvm \
--cdrom /home/lmwangi/debian-505-amd64-netinst.iso
--force yes

Virtio speeds up things a lot.

virt-install --name=nms01 \
--ram 512 \
--os-type linux --os-variant debianlenny \
--disk path=/dev/dm-8,bus=virtio \
--network network=default,model=virtio \
--vnc \
--noautoconsole \
--cdrom /home/lmwangi/debian-505-amd64-netinst.iso
--virt-type kvm \
--force yes

Cloning is easy. Create/Install a template VM and clone it to create your workhorses.

virt-clone \
--prompt \
--connect qemu:///system \
--original template \
--name "champ" \
--file /dev/xen02/champ.root

By default, your default network should be started.

virsh net-start default && virsh net-autostart default

You can also view an existing VM's definition.

# virsh net-dumpxml default

default
900a487e-d5a6-b731-7bb0-3997fc0455fb







.....

You can also hotplug disks. The VM wasn't stable though :(

virsh # detach-disk
error: command 'detach-disk' requires option
error: command 'detach-disk' requires option
virsh # detach-disk nosql vdb
Disk detached successfully

virsh # attach-disk nosql /dev/xen02/nosql.storage vdb
Disk attached successfully

On the VM, you may need to

modprobe acpiphp

Occasionally, I have legacy VMs that were not properly created (Someone bypassed libvirt.) Cloning is a pain. In essence, I create a similar LV (same size), define a VM using the new LV and then dd over the LV

[root@vm01 ~]# lvdisplay cnix/box1_root
--- Logical volume ---
LV Name /dev/cnix/box1_root
VG Name cnix
LV UUID xxxx-xxxx.....
LV Write Access read/write
LV Status available
# open 1
LV Size 19.53 GB
Current LE 5000
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:12

[root@vm01 ~]# lvcreate -L 19.53G -n signer sdbvg
Rounding up size to full physical extent 19.53 GB
Logical volume "signer" created

[root@vm01 ~]# virt-clone --prompt --original legacy1 --name signer --file /dev/sdbvg/signer.root --file /dev/sdbvg/signer.swap
This will overwrite the existing path '/dev/sdbvg/signer.root'!
Do you really want to use this disk (yes or no) yes
This will overwrite the existing path '/dev/sdbvg/signer.swap'!

Do you really want to use this disk (yes or no) yes
Cloning /dev/cnix/legacy1_root | 20 GB 02:11
Cloning /dev/cnix/legacy1_swap | 2.0 GB 00:08

Clone 'signer' created successfully.

Sometimes this is not always flawless

virt-clone --prompt --original master --name master1 --file /dev/sdbvg/master1.root --file /dev/sdbvg/master1.swap
ERROR Domain 'master' was not found.
What is the name of the original virtual machine? master
ERROR Domain 'master' was not found.
What is the name of the original virtual machine? legacy1
ERROR Domain 'legacy1' was not found.
What is the name of the original virtual machine? ERROR

So we have to dd the image to the new hdd

[root@vm01 ~]# dd if=/dev/cnix/legacy1_root of=/dev/sdbvg/master1.root
dd if=/dev/cnix/legacy2_root of=/dev/sdbvg/member.root
40960000+0 records in
40960000+0 records out
20971520000 bytes (21 GB) copied, 1237.21 seconds, 17.0 MB/s

And boot the new VM

Precedence for IPv4 vs IPv6

You can use /etc/gai.conf to set up your IPV4/IPV6 precedence as documented here and here.


Say we have two hosts www.he.net and www.ripe.net

$ host www.he.net
www.he.net is an alias for he.net.
he.net has address 216.218.186.2
he.net has IPv6 address 2001:470:0:76::2


$ host www.ripe.net
www.ripe.net has address 193.0.6.139
www.ripe.net has IPv6 address 2001:67c:2e8:22::c100:68b

Case 1: Prefer IPV4
Append the following to /etc/gai.conf

precedence ::ffff:0:0/96  100

then we have:

$ telnet www.ripe.net 80
Trying 193.0.6.139...
^C
$ telnet www.he.net 80
Trying 216.218.186.2...

Case 1: Prefer IPV6 for specific hosts
If we append

precedence 2001:470::/32 100

then we have

$ telnet www.ripe.net 80
Trying 193.0.6.139...
^C
$ telnet www.he.net 80
Trying 2001:470:0:76::2...
^C

So we seem to prefer that network for ipv6 and ipv4 everywhere else.


Case 3: Prefer ipv4 for specific hosts
Wondering if we invert the mask the reverse will be true.

It's all in the stars

Astrology is the prostitution of mathematics.

Galileo character - Galileo's Dream

Decrypting SSL using tshark/wireshark

I was banging my head on the keyboard last week trying to figure out why wireshar/tshark was not decrypting my SSL packets. Turns out that you cannot do this if the key exchange is Diffie-Hellman. You can get around this by telling the server to ignore DH, In apache's case, changing the SSLCipherSuite to something like:

SSLCipherSuite !DH:!DH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW

The server hello changes from

Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)

to

Cipher Suite: TLS_RSA_WITH_RC4_128_SHA (0x0005)

Q'apla!

 tshark   -o "ssl.desegment_ssl_records: TRUE" -o "ssl.desegment_ssl_application_data: TRUE" -o "ssl.keys_list:10.0.0.1,443,http,/path/to/server.key.pem" -o "ssl.debug_file: /tmp/wireshark-log" -r /path/to/example.pcap  -V| awk '/^Frame.*/;/Source: 2/;/Secure Socket Layer.*/,/^$/;' |tail -n 40

....

     Version: TLS 1.0 (0x0301)
        Length: 61
        Encrypted Application Data: 6C6423B17A7211C8F8E9E60B5F726D0C7C40B5645B234A36...
    SSL segment data (41 bytes)
[Reassembled SSL Segments (227 bytes): #60(186), #60(41)]
    [Frame: 60, payload: 0-185 (186 bytes)]
    [Frame: 60, payload: 186-226 (41 bytes)]
Hypertext Transfer Protocol
    HTTP/1.1 200 Script output follows\r\n
        [Expert Info (Chat/Sequence): HTTP/1.1 200 Script output follows\r\n]
            [Message: HTTP/1.1 200 Script output follows\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Version: HTTP/1.1
        Response Code: 200
    Date: Mon, 16 May 2011 13:41:11 GMT\r\n
    Server: Apache/2.2.8 (CentOS)\r\n
    Content-Length: 41\r\n
        [Content length: 41]
    Connection: close\r\n
    Content-Type: application/mercurial-0.1\r\n
    \r\n
Media Type
    Media Type: application/mercurial-0.1 (41 bytes)

Dammit, read the output...

I keep forgetting to read the output of  commands

$ gdb apache2 /tmp/gdb/core
GNU gdb (GDB) 7.0.1-debian
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
...
     
warning: The current binary is a PIE (Position Independent Executable), which
GDB does NOT currently support.  Most debugger features will fail if used
in this session.
     
Reading symbols from /usr/sbin/apache2...Reading symbols from /usr/lib/debug/usr/sbin/apache2-mpm-prefork...done.
(no debugging symbols found)...done.
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f9779b25c4f in ?? ()
(gdb) bt
#0  0x00007f9779b25c4f in ?? ()
#1  0x00007fc05b93c8b0 in ?? ()
#2  0x0000000000000007 in ?? ()
#3  0x0000000000000000 in ?? ()
(gdb) where
#0  0x00007f9779b25c4f in ?? ()
#1  0x00007fc05b93c8b0 in ?? ()
#2  0x0000000000000007 in ?? ()
#3  0x0000000000000000 in ?? ()
(gdb) quit
root@acme:~#
root@acme:~# apt-get install apache2-mpm-worker

Turns out that squeeze's gdb doesn't support PIE, pulling in SID's copy works

$ gdb -p 21164
GNU gdb (GDB) 7.2-debian
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
.                                                                                                   Attaching to process 21164
Reading symbols from /usr/lib/apache2/mpm-worker/apache2...Reading symbols from /usr/lib/debug/usr/lib/apache2/mpm-worker/apache2-mpm-worke
r...done.
done.
Reading symbols from /lib/libpcre.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib/libpcre.so.3
Reading symbols from /usr/lib/libaprutil-1.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libaprutil-1.so.0
Reading symbols from /usr/lib/libapr-1 .....

Introduction to RPKI

What is RPKI

Resource Public Key Infrastructure (RPKI) is the name of the game.
For the Internet to work, we need BGP¹ which in turn requires IP prefixes and AS numbers.  When an LIR or any multi homed organization wants to be reachable on the Internet, they have to peer using these number resources.

No security framework exists that authenticates whether a peer does indeed own the resource set that they are advertising. Consequently, rogue announcements² may appear on the internet that will impact on the reachability of legitimate sites. RPKI allows one to take these resources, add them into a certificate and then use these resource aware certificates to sign Route Origin Authorizations (ROA) and a set of other objects that may be defined in the future (The underlying specification documents are still in draft status For the latest and up to date specifications, please see the sidr ietf page).

Number resources are issued by IANA to the 5 RIRs. These RIRs then issue resources to LIRs/NIRs which issue their resources to their customers. Their customers may issue their resources... PKIs do follow a similar tree structure where there's a root (self signed CA certificate) issues other certificates (Which may be CA certificates that subissue ...). Consequently, every resource owner can and should have a resource certificate.


These two hierarchical structures are similar and this allows them to be tightly integrated.

An example allocation structure:

Table 1: Resource Allocation table

ORG IPV4 IPV6 ASN Function IANA 0/0 0/0 1-2**32 Issues resources to RIRS AFRINIC 196/8 2001:4200/23 327680-328703 Issues resources to LIRS LIR C 196.1/16 2001:4200/32 327680-327690 Large ISP. ISP X 196.1.0/23 2001:4200/48 327685 Small ISP.

An example PKI structure:

What is RPKI...Really?

Well:

• It allows a resource holder to prove that they do actually owns a set of resources (Binds number resources to the subject of the certificate).
• A resource holder can provide a PKI based authorisation attesting that a peer is allowed to originate a set of prefixes on their behalf (In simple terms, I am a small company X who buys bandwidth with A & C. A ROA can be created to attest that only A & C can advertise X's prefixes)
• CRLs are also published and their distribution points listed in certificates.
• Certificates, CRLs and products such as ROAs for each authority publication point are listed along with their hashes in a signed structure that is published. This structure is called a manifest(mft).
• CA certs, CRLs, MFTs, ROAs are published in a repository to the world over rsync and optionally https.
• Validation tools exist that can validate an entire repository tree

Next, A quick peek at sample RPKI certificates.

XMLRPC Register instance is stateful. Do not want that

I have an XMLserver that uses the method SimpleXMLRPCServer.register_instance to register a class and it's methods. Something like:

....    
    inst = ServiceClass(init_params)
    server.register_instance(inst)
    server.register_introspection_functions()

    try:
        server.serve_forever()
    finally:
        server.server_close()

The serviceclass looks like:

class ServiceClass(object):
    """ServiceClass calls the xxxx executable and returns stdout & stderr"""
    def __init__(self, params):
        super(ServiceClass, self).__init__()
        self.params = params
        print "Init with %s" % self.params

    def _shell_exec(self, cmd=[]):
        .......
        ....
...
    def dbupdate(self, sometext=""):
        """Given an object, try call the xxxx binary and
        return the output of the command
        Keyword arguments:                                                                                                                 
        params:                                                                                                                            
            sometext -- text of the object
        """
        if not len('sometext'):
            raise MissingObjectError("No text passed for command")

        # Create a tempfile
        (tmpfd, fname) = tempfile.mkstemp()
        os.write(tmpfd,  sometext)
        os.close(tmpfd)
        # Execute
        cmd_exe = self.params['cmd']                                                                                                       
        cmd_exe.extend([fname])
        ret_stdout, ret_stderr = self._shell_exec(cmd_exe)

        # Cleanup file
        os.unlink(fname)

        print "cmd: %s\nself.cmd: %s\nfname: %s\n" %(cmd_exe, self.params['cmd'], fname)
    
        return (ret_stdout, ret_stderr)

Turns out that python keeps state since... the instance is created only once at server startup. All subsequent calls share this instance. And that affects the class variables. Notice that tempfiles from a prior run are visible to a later run which might be from a different client. :(

:!python some_webservice_xmlrpc.py -d 1 -p 8000                                                                                        
Listening on :8000...
Service URI is http://:8000/
Use Control-C to exit
CMD: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpkCXRS-']
cmd: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpkCXRS-']
self.cmd: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpkCXRS-']
fname: /tmp/tmpkCXRS-

machine1.example.net - - [21/Apr/2011 14:53:08] "POST / HTTP/1.0" 200 -
CMD: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpkCXRS-', '/tmp/tmpBGDHHU']
cmd: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpkCXRS-', '/tmp/tmpBGDHHU']
self.cmd: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpkCXRS-', '/tmp/tmpBGDHHU']

I tried different solutions from del cmd prior to return to initializing all variables to None|"" at the start of the method. Didn't work.. till I remembered shallow copying.   With a deep copy,

....                                                   
#        cmd_exe.extend([fname])
#Replace with
        cmd_exe = copy.deepcopy(self.params['cmd'])
....

things behave as expected:

Listening on :8000...
Service URI is http://:8000/
Use Control-C to exit
Init with {'debug': 1, 'cmd': ['/path/to/executable', '-c', '/path/to/config.conf', '-f']}
CMD: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpLTR0-S']
cmd: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpLTR0-S']
self.cmd: ['/path/to/executable', '-c', '/path/to/config.conf', '-f']
fname: /tmp/tmpLTR0-S

machine1.example.net - - [21/Apr/2011 14:58:19] "POST / HTTP/1.0" 200 -
CMD: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpxdCOtw']
cmd: ['/path/to/executable', '-c', '/path/to/config.conf', '-f', '/tmp/tmpxdCOtw']
self.cmd: ['/path/to/executable', '-c', '/path/to/config.conf', '-f']
fname: /tmp/tmpxdCOtw

machine1.example.net - - [21/Apr/2011 14:58:26] "POST / HTTP/1.0" 200 -

recursion && tail recursion

A look at the difference between plain old recursion and tail recursion.

• Compile recursion.c
• Fire up GDB

$ gdb recursion
GNU gdb (GDB) 7.2-debian
.....
(gdb)

• Set up breakpoints at recursion termination points

(gdb)  break recursion.c:50
Breakpoint 1 at 0x4005b9: file /home/lmwangi/learning/algo/recursions/recursion.c, line 50.
(gdb)  break recursion.c:68
Breakpoint 2 at 0x400601: file /home/lmwangi/learning/algo/recursions/recursion.c, line 68.

• Run till the first breakpoint - Good old recursion. Notice that we have to step 4 times.

(gdb) run
Starting program: /home/lmwangi/learning/algo/recursions/recursion

Breakpoint 1, factorial (n=1) at /home/lmwangi/learning/algo/recursions/recursion.c:50
50                      return 1;
(gdb) bt
#0  factorial (n=1) at /home/lmwangi/learning/algo/recursions/recursion.c:50
#1  0x00000000004005cd in factorial (n=2) at /home/lmwangi/learning/algo/recursions/recursion.c:53
#2  0x00000000004005cd in factorial (n=3) at /home/lmwangi/learning/algo/recursions/recursion.c:53
#3  0x00000000004005cd in factorial (n=4) at /home/lmwangi/learning/algo/recursions/recursion.c:53
#4  0x00000000004005cd in factorial (n=5) at /home/lmwangi/learning/algo/recursions/recursion.c:53
#5  0x0000000000400544 in main () at /home/lmwangi/learning/algo/recursions/recursion.c:30
(gdb) step
56      }
(gdb) step
56      }
(gdb) step
56      }
(gdb) step
56      }
(gdb) step
56      }
(gdb) step
main () at /home/lmwangi/learning/algo/recursions/recursion.c:32
32              printf("Factorial %d\n", result);

• Tail recursion. Notice that we step only once since the function has all the results from prior operations

(gdb) cont
Continuing.
Factorial 120

Breakpoint 2, tail_factorial (n=1, a=120) at /home/lmwangi/learning/algo/recursions/recursion.c:68
68                      return a;
(gdb) bt
#0  tail_factorial (n=1, a=120) at /home/lmwangi/learning/algo/recursions/recursion.c:68
#1  0x000000000040061c in tail_factorial (n=2, a=60) at /home/lmwangi/learning/algo/recursions/recursion.c:71
#2  0x000000000040061c in tail_factorial (n=3, a=20) at /home/lmwangi/learning/algo/recursions/recursion.c:71
#3  0x000000000040061c in tail_factorial (n=4, a=5) at /home/lmwangi/learning/algo/recursions/recursion.c:71
#4  0x000000000040061c in tail_factorial (n=5, a=1) at /home/lmwangi/learning/algo/recursions/recursion.c:71
#5  0x000000000040056d in main () at /home/lmwangi/learning/algo/recursions/recursion.c:34
(gdb) step
73      }
(gdb) step
main () at /home/lmwangi/learning/algo/recursions/recursion.c:36
36              printf("Tail Factorial %d\n", result);
(gdb) 

Unix Permission modes

This is a long post covering unix permission modes. A file's mode is stored in it's inode structure. Most of this post is inline in the following shell interaction snippet(plain text here). The quirky bits are:

1. Permissions are analysed in increasing generality i.e User, Group, Others. The first match is accepted whether positive or negative
2. Executing a setuid binary is equivalent to executing that binary as the user owning the binary
3. Executing a setgid binary is equivalent to executing the binary as the group owning that binary
4. Binaries require setting the x bit only while shell scripts require r & x bits to be set.
5. If you setgid a directory and create a file in that directory, the file's group will be set to the group of the directory (Group permissions become inherited regardless of whether you were in that group).
6. Sticky bits on a directory restrict deletion to the owner of the directory, the creator of a file and the superuser. That's how /tmp works! It's also known as the restricted deletion flag.

# Lets make a couple of tests
$ test_read () { $(cat  2>/dev/null test.file > /dev/null ); if [ $? -eq 0 ];then echo "Read : True "; else echo "Read : False"; fi; }
$ test_write () { $(echo 2>/dev/null -e '#!/bin/sh\necho hello' > test.file ); if [ $? -eq 0 ];then echo "Write: True "; else echo "Write: False"; fi; }
$ test_exec () { $( ./test.file 2>/dev/null 1>/dev/null); if [ $? -eq 0 ];then echo "Exec : True "; else echo "Exec : False"; fi; }

# Touch a test.file
$ ls -l test.file
-rwx------ 1 lmwangi lmwangi 21 Apr 14 16:31 test.file

# Test mode as user 
$ for mode in $(seq -w 0 100 700); do echo "Mode $mode"; touch test.file; chmod $mode test.file; test_read; test_write; test_exec;echo;  done
Mode 000
Read : False
Write: False
Exec : False

Mode 100
Read : False
Write: False
Exec : False

Mode 200
Read : False
Write: True 
Exec : False

Mode 300
Read : False
Write: True 
Exec : False

Mode 400
Read : True 
Write: False
Exec : False

Mode 500
Read : True 
Write: False
Exec : True 

Mode 600
Read : True 
Write: True 
Exec : False

Mode 700
Read : True 
Write: True 
Exec : True 


# File permissions like firewall acls stop at the first match. If I am the owner of the file, matching stops with the user bits. Groups/others are never consulted... So we chgrp of file to a group you are not in say root and test.
$ sudo chgrp root test.file 
$ ls -l
total 4
----rwx--- 1 lmwangi root 21 Apr 14 16:36 test.file

$ for mode in $(seq -w 0 010 070); do echo "Mode $mode"; touch test.file; chmod $mode test.file; test_read; test_write; test_exec;echo;  done
Mode 000
Read : False
Write: False
Exec : False

Mode 010
Read : False
Write: False
Exec : False

Mode 020
Read : False
Write: False
Exec : False

Mode 030
Read : False
Write: False
Exec : False

Mode 040
Read : False
Write: False
Exec : False

Mode 050
Read : False
Write: False
Exec : False

Mode 060
Read : False
Write: False
Exec : False

Mode 070
Read : False
Write: False
Exec : False

# It get's interesting if I am not the owner of the file but I am in a group that owns the file.
$ id
uid=1000(lmwangi) gid=1000(lmwangi) groups=1000(lmwangi),20(dialout),24(cdrom),25(floppy),29(audio),44(video),46(plugdev),112(netdev),114(fuse),119(libvirt),1003(packetcapture)
$ ls -l
total 4
----rwx--- 1 root lmwangi 21 Apr 14 16:42 test.file

for mode in $(seq -w 0 010 070); do echo "Mode $mode"; sudo chmod $mode test.file; test_read; test_write; test_exec;echo;  done
Mode 000
Read : False
Write: False
Exec : False

Mode 010
Read : False
Write: False
Exec : False

Mode 020
Read : False
Write: True 
Exec : False

Mode 030
Read : False
Write: True 
Exec : False

Mode 040
Read : True 
Write: False
Exec : False

Mode 050
Read : True 
Write: False
Exec : True 

Mode 060
Read : True 
Write: True 
Exec : False

Mode 070
Read : True 
Write: True 
Exec : True 

# The same principle applies. Others mode only applies if I am neither the file owner nor in a group that owns the file.
$ for mode in $(seq -w 0 001 007); do echo "Mode $mode"; sudo chmod $mode test.file; test_read; test_write; test_exec;echo;  done
Mode 000
Read : False
Write: False
Exec : False

Mode 001
Read : False
Write: False
Exec : False

Mode 002
Read : False
Write: True 
Exec : False

Mode 003
Read : False
Write: True 
Exec : False

Mode 004
Read : True 
Write: False
Exec : False

Mode 005
Read : True 
Write: False
Exec : True 

Mode 006
Read : True 
Write: True 
Exec : False

Mode 007
Read : True 
Write: True 
Exec : True 

# It gets better with executables :)
# Let's make a small template executable
$  echo -e '#include <stdio.h>\nint main(){printf("hello\\n"); return 0;}' > hello.c && gcc hello.c -o hello
$ ./hello 
hello
# Now watch mode 100 compare with the a shell script of mode 100. Do the same for 300, 500 & 700
Mode 000
Read : False
Write: False
Exec : False

Mode 100
Read : False
Write: False
Exec : True 
Mode 200
Read : False
Write: True 
Exec : False

Mode 300
Read : False
Write: True 
Exec : True 

Mode 400
Read : True 
Write: False
Exec : False

Mode 500
Read : True 
Write: False
Exec : True 

Mode 600
Read : True 
Write: True 
Exec : False

Mode 700
Read : True 
Write: True 
Exec : True 

# As can be seen. You must have r & x to exec a shell script while your require only x for a binary.
# Recap again
$ for mode in $(seq -w 0 100 700); do echo $mode; chmod $mode test.file; ls -l test.file |grep test.file; ./test.file; done
000
---------- 1 lmwangi lmwangi 6695 Apr 14 17:06 test.file
-bash: ./test.file: Permission denied
100
---x------ 1 lmwangi lmwangi 6695 Apr 14 17:06 test.file
hello
200
--w------- 1 lmwangi lmwangi 6695 Apr 14 17:06 test.file
-bash: ./test.file: Permission denied
300
--wx------ 1 lmwangi lmwangi 6695 Apr 14 17:06 test.file
hello
400
-r-------- 1 lmwangi lmwangi 6695 Apr 14 17:06 test.file
-bash: ./test.file: Permission denied
500
-r-x------ 1 lmwangi lmwangi 6695 Apr 14 17:06 test.file
hello
600
-rw------- 1 lmwangi lmwangi 6695 Apr 14 17:06 test.file
-bash: ./test.file: Permission denied
700
-rwx------ 1 lmwangi lmwangi 6695 Apr 14 17:06 test.file
hello


# Now let's look at the  setuid/setgid/sticky bits. 
1 = Sticky - if set not relevant in a binary in my system [Linux]
2 = SetGID - if set the progran runs with eGID set to the binary group
4 = SetUID - if set the program runs with euid set tot the binary owner.

# Let's make a program that shows us the uid, effective uid, gid and effective gid
$ echo -e '#include <stdio.h>\n#include <unistd.h>\nint main(){printf("hello\\nUID:%d\\nEUID:%d\\nGID:%d\\nEGID:%d\\n",getuid(),geteuid(),getgid(),getegid()); return 0;}' > hello.c && gcc hello.c -o hello                                                      

$ ./hello 
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

$ id
uid=1000(lmwangi) gid=1000(lmwangi) groups=1000(lmwangi),20(dialout),24(cdrom),25(floppy),29(audio),44(video),46(plugdev),112(netdev),114(fuse),119(libvirt),1003(packetcapture)

$ for owner in lmwangi:lmwangi lmwangi:root root:lmwangi root:root; do echo ">>>> $owner <<<<"; for mode in 1777 2777 4777 6777; do echo "mode $mode"; cp hello test.file ; sudo chown $owner test.file; sudo chmod $mode test.file; ls -l test.file; ./test.file;echo; done;  done
>>>> lmwangi:lmwangi <<<<
mode 1777
-rwxrwxrwt 1 lmwangi lmwangi 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

mode 2777
-rwxrwsrwx 1 lmwangi lmwangi 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

mode 4777
-rwsrwxrwx 1 lmwangi lmwangi 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

mode 6777
-rwsrwsrwx 1 lmwangi lmwangi 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

>>>> lmwangi:root <<<<
mode 1777
-rwxrwxrwt 1 lmwangi root 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

mode 2777
-rwxrwsrwx 1 lmwangi root 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:0

mode 4777
-rwsrwxrwx 1 lmwangi root 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

mode 6777
-rwsrwsrwx 1 lmwangi root 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:0

>>>> root:lmwangi <<<<
mode 1777
-rwxrwxrwt 1 root lmwangi 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

mode 2777
-rwxrwsrwx 1 root lmwangi 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

mode 4777
-rwsrwxrwx 1 root lmwangi 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:0
GID:1000
EGID:1000

mode 6777
-rwsrwsrwx 1 root lmwangi 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:0
GID:1000
EGID:1000

>>>> root:root <<<<
mode 1777
-rwxrwxrwt 1 root root 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

mode 2777
-rwxrwsrwx 1 root root 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:0

mode 4777
-rwsrwxrwx 1 root root 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:0
GID:1000
EGID:1000

mode 6777
-rwsrwsrwx 1 root root 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:0
GID:1000
EGID:0

# Sticky bit (RESTRICTED DELETION FLAG) on directories 

$ mkdir test; sudo chown root:root test; sudo chmod 1777 test; ls -l |grep test
drwxrwxrwt 2 root    root    4096 Apr 14 17:39 test
-rwsrwsrwx 1 root    root    7205 Apr 14 17:31 test.file

$ echo lmwangi > test/by_lmwangi
$ ls -l test/
total 4
-rw-r--r-- 1 lmwangi lmwangi 8 Apr 14 17:41 by_lmwangi

# No one can remove the file
$ sudo su guest -c "rm test/by_lmwangi"
rm: remove write-protected regular file `test/by_lmwangi'? y
rm: cannot remove `test/by_lmwangi': Operation not permitted

# Root can  override though or the owner of the dir
$ sudo rm test/by_lmwangi 

$ rm -rf test
$ mkdir test
$ chmod 1777 test
$ sudo su guest -c "echo hello > test/by_guest"
$ ls -l test
total 4
-rw-r--r-- 1 guest guest 6 Apr 14 17:38 by_guest
$ ls -l |grep test
drwxrwxrwt 2 lmwangi lmwangi 4096 Apr 14 17:38 test
-rwsrwsrwx 1 root    root    7205 Apr 14 17:31 test.file
$ cat test/by_guest 
hello
$ rm test/by_guest 
rm: remove write-protected regular file `test/by_guest'? y

$ ls -l test
total 0

# And we go out with a bang
# setuid on dir == no change on files                                                                  
$ rm -rf test && mkdir test; sudo chown root:root test; sudo chmod 4777 test; ls -l|grep test
$ cd test/
$ touch aha
$ ls -l
total 0
-rw-r--r-- 1 lmwangi lmwangi 0 Apr 14 17:46 aha
$ cd ..

# But setgid on dir == changes any file created in it to the group owning the dir
$ rm -rf test && mkdir test; sudo chown root:root test; sudo chmod 2777 test; ls -l|grep test
drwxrwsrwx 2 root    root    4096 Apr 14 17:47 test
-rwsrwsrwx 1 root    root    7205 Apr 14 17:31 test.file
$ cd test/
$ touch aha
$ ls -l
total 0
-rw-r--r-- 1 lmwangi root 0 Apr 14 17:47 aha
$ id
uid=1000(lmwangi) gid=1000(lmwangi) groups=1000(lmwangi),20(dialout),24(cdrom),25(floppy),29(audio),44(video),46(plugdev),112(netdev),114(fuse),119(libvirt),1003(packetcapture)

Unix is loads of fun :)