Kenya DOB density plot

Here is a density plot of a sample of Kenyans  based on their year of birth. It's interesting to see the dips correlate with turbulent times; political and military. In the diagram below, the largest unexplained dip is in the early 1970s. Anyone care to explain it; Google and Wikipedia don't return any results?
Click here for a larger image

EDIT:
Could it be the 1969 Kisumu incident?



 


Here is the code used to produce the graph in R

Hotfix for leaking filedescriptors

I just came across this neat gdb trick. Say you have a badly written program that's leaking file descriptors all over.. and it's in prod (the horror!) and it's imperative that you let it limp along till the patched binary comes around... GDB can close the file descriptors for you!

Here's how:

Leaky Program


Lsof tells it all. Take note of the two open descriptors to /etc/passwd


Playing around with variables in GDB. Can you guess why we chose the magic value 2147483646? For a hint, look at the last gist in this post


What does lsof think?


Bulk close of fds


Shell output from leaky

Gem of a fortune - I. M. Banks.

"Good afternoon, madam.  How may I help you?"

"Good afternoon.  I'd like a FrintArms HandCannon, please."

"A--?  Oh, now, that's an awfully big gun for such a lovely lady.  I
mean, not everybody thinks ladies should carry guns at all, though I
say they have a right to.  But I think... I might... Let's have a look
down here.  I might have just the thing for you.  Yes, here we are!
Look at that, isn't it neat?  Now that is a FrintArms product as well,
but it's what's called a laser -- a light-pistol some people call
them.  Very small, as you see; fits easily into a pocket or bag; won't
spoil the line of a jacket; and you won't feel you're lugging half a
tonne of iron around with you.  We do a range of matching accessories,
including -- if I may say so -- a rather saucy garter holster.  Wish I
got to do the fitting for that!  Ha -- just my little joke.  And
there's *even*... here we are -- this special presentation pack: gun,
charged battery, charging unit, beautiful glider-hide shoulder holster
with adjustable fitting and contrast stitching, and a discount on your
next battery.  Full instructions, of course, and a voucher for free
lessons at your local gun club or range.  Or there's the *special*
presentation pack; it has all the other one's got but with *two*
charged batteries and a night-sight, too.  Here, feel that -- don't
worry, it's a dummy battery -- isn't it neat?  Feel how light it is?
Smooth, see?  No bits to stick out and catch on your clothes, *and*
beautifully balanced.  And of course the beauty of a laser is, there's
no recoil.  Because it's shooting light, you see?  Beautiful gun,
beautiful gun; my wife has one.  Really.  That's not a line, she
really has.  Now, I can do you that one -- with a battery and a free
charge -- for ninety-five; or the presentation pack on a special
offer for one-nineteen; or this, the special presentation pack, for
one-forty-nine."

"I'll take the special."

"Sound choice, madam, *sound* choice.  Now, do--?"

"And a HandCannon, with the eighty-mill silencer, five GP clips, three
six-five AP/wire-flechettes clips, two bipropellant HE clips, and a
Special Projectile Pack if you have one -- the one with the embedding
rounds, not the signalers.  I assume the night-sight on this toy is
compatible?"

"Aah... yes,  And how does madam wish to pay?"

She slapped her credit card on the counter.  "Eventually."

        -- Iain M. Banks, "Against a Dark Background"

Automatic number plate detection

ALPR stands for automatic license plate recognition. The process typically consists of:

• Identifying potential license plate candidates from an image.
• Ranking the potential plate candidates and selecting the most likely one.
•  Identifying alphanumeric characters and converting them to text.

The hardest part of this process is reliably acquiring the plate candidate image with variables such as:

• Lighting conditions.
• Camera angle.
• Image noise
• Variations in the type of plates.
• Distance to plate; and consequently, the size of the plate in the image

In this article, I'll be going through a gist that works for a few images. I'll cover the literature review in a followup article.

Please note that this implementation has been developed and tested with very few samples; overfitting will probably reduce it's effectiveness.

1. Load an image using opencv and python (Line 33)
2. Run mean shift segmentation (I haven't evaluated the effectiveness of this) and save the image.
3. Use the segmented image as the camera image.
4. Convert the camera image (defaults as a numpy array) into OpenCV Mat and an IPL image (CV2 can work with numpy arrays. Plain old cv requires an IPL or Mat object).
5. Convert the camera image into grayscale.
6. Conduct thresholding on the image (Line 76). This means that pixels that are above the threshold become white and below black. This process is also called binarization.
7. Create a mask that will be used for blob extraction that covers the entire image; setting all the pixels to 1 means that the mask will be applied everywhere.
8. Invert the binarized image. License plates are mostly white with a few characters in black. Inversion means that the plate becomes mostly black with a few white character strokes (Line 94).
9. Blob detection parameters. We don't want very small or very large blobs.
10. Do the actual blob detection (Line 115).
11. Iterate over the grayscale image adding an RGB color << this is a bug
12. Mark potential blobs as black in a white image (Line 131).
13. Another threshold step << This is probably unnecessary.
14. Dilation and erosion to clean up the image << probably unnecessary.
15. Find contours over the black images.
16. For each contour, try and estimate it's overall shape. triangle? rectangle? n-gon (Line 196)? after eliminating very small and very large contours.
17. Draw a bounding box.
18. Calculate an estimate of the aspect ratio. Plates have a W/H ratio of 1 to 5
19. Off the cuff heuristic. Largest rectangle that has an acceptable aspect ratio and is a rectangle is probably a plate. This is not always true. You might much a window or a large rectangular object...
20. For the selected plate candidate, draw a bounding rect

Here are the results:

Segmented image

Thresholded image

Grayscale conversion

Blobs on a white image

Effects of erosion on an image

Effects of dilation on an image

Plate candidates - from contours

Min area box over plate candidate

Bounding box over plate candidate

And here are more images from a few more tests in picassa:

• Failures
• Successes

OSX: Y U No like 3G?

I recently came across an issue where my laptop would consume copious amounts of bandwidth. On average, the idle laptop would slurp up about 8MB of traffic/hour. In a day, that's ~ 100 MB of data gone.

In S. Africa and on 3G, this is prohibitively expensive. How expensive?
I am on Telkom 8ta, which charges

~ 40 Rand/100MB == 6USD/100MB. 

Scaling up this cost to a month adds up to

180USD/Month for ~ 3GB of data. 

 Obviously, this is a problem!

 So let's find out what's going on....

Tcpdump

Hmm, 5 similar queries? Obviously, there's something stuck in a dns lookup loop. This is not cool at all. If we do some math, the dns traffic adds up to the extra traffic that my 3G dongle is reporting. So how do we find out which process generates this data? We can use little snitch or better, use dtrace which comes in bundled with OSX.

DTrace

Dtrace allows you to hook up a probe that get's called once your kernel function is called. For example, say you know that all udp traffic in your box is generated via a specific syscall, you can 'watch' this syscall and generate some statistics around this.. (Basing this description on systemtap). So let's find out how a typical dns lookup works. The easiest program I can think that does dnslookup is nslookup. So let's truss up nslookup
The system call we are after is sendmsg and recvmsg (UDP send and udp recv). The exact packets sent and receiver are also unraveled in the packet tree below the dtruss output. The ardent reader will notice that the length of the UDP packet - 8 bytes == the syscall return output for sendmsg and recvmsg. For example, sendmsg has a return value of 32 bytes (40 UDP header - 2 bytes src port - 2 bytes dst port - 2 bytes length - 2bytes checksum). See line 32, 36, 101 and 172 of the gist above.

So we now know without a doubt that we ought to watch for a process that's generating a lot of recvmsg and sendmsg (At a rate of 10/sec for either syscall). Let's fire up another dtrace script

Fini

So let's try and validate our findings by stopping the buggy process. Phew! And that's it! A sample of 100 packets has 100 DNS requests when opendirectoryd is running. The 100 packets are captured in 3.5 seconds. With opendirectoryd stopped, it takes 23 seconds to capture 100 packets; DNS constitutes 30% of the sample. It's obvious that something is wrong with opendirectoryd. Stopping the process will save me tons of bucks until Apple issues a fix.

xargs and race conditions

I tend to use xargs as a quick paralleling tool. It works great. However, you may notice that due to the multiple processes running, output from one process will overlap with another process creating an unreadable mess.  As an example, I want to ping 6 addresses at the same time. I can do it serially at 1 ping/sec for a total of 6 seconds. Linear growth or O(n):

How do you get around it? We can run everything in parallel using xargs

Note that the time taken is markedly reduced but.. Output from each process is interleaved with other commands.

How do you get around that? Well, locking.. Luckily, the shell has a nifty locking utility that is just perfect for this. Enter flock..

Still as fast and neater. Man flock for more awesomeness. The flock utility uses the flock syscall to acquire a lock on a file structure. In the example above, I show that you can use a file that you are reading as a lock file. flock doesn't prevent you from conducting other file ops on your fd.

Digests using OpenSSL. Shell and perl editions,

Here's a small snippet that allows one to generate a digest.
Short of it: Perl

my $key_string = read_file("secretkey.pem");
my $key = Crypt::OpenSSL::RSA->new_private_key($key_string);
my $filestring = read_file("somefile");
$key->use_md5_hash();
my $signature = $key->sign($filestring);
print encode_base64($signature)."\n";

Short of it: shell

openssl dgst -md5 -sign secretkey.pem  <  somefile |base64 | tee somefile.sig_from_shell
cat somefile.sig_from_shell | base64 -d > somefile.sig_from_shell.raw
openssl dgst -md5 -verify public.pem  -signature somefile.sig_from_perl.raw <  somefile

In it's goriness: