Calculating the start time of a process

A quick script that calculate the start time of a process in Linux:

EXT4 err: couldn't mount because of unsupported optional features

Backward & Forwards compatibility is great! Until it bites you.

The extN branch of filesystem is the pretty much the standard on your average GNU/Linux installation. Migrations tend to work well if N is increasing i.e. from ext2->ext3->ext4. However, things are not so rosy if you want to mount your ext4 disk on your old PC.
Ext4 which made it into mainline in 2.6.27 introduces new features such as extents that are incompatible and unknown by older kernels. Attempting to mount an incompatible extN fs on your old kernel will fail and err out with logs similar to:

EXT3-fs: sda1: couldn't mount because of unsupported optional features (240).

EXT2-fs: sda1: couldn't mount because of unsupported optional features (240).

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,1)

The kernel here attempts to probe and mount the using the most recent extN. If that fails it tries the next ext filesystem and so on. In this case, this is a 2.6.16 kernel attempting to mount an ext4 fs with new features.

# dumpe2fs /tmp/img.ext4
.....
Filesystem volume name: /
Last mounted on: /mnt/tmp
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash

 The solution in this case is to use a newer kernel that understands ext4 or recompile your kernel to support ext4.

Linux kernel network backdoor

The ksplice blog has a very nice entry on hosting backdoors in hardware.
The quick summary of this backdoor is:

1. Register a protocol handler for an unused IP protocol number .
2. Call usermodhelper to execute the payload of the packet (skb->data).
3. Remote system now executes any command that you send it as root.

Unfortunately, it looks like the code is either out of date and/or buggy. Attempting to modprobe the backdoor module generates the following kernel call trace:

Further investigations reveal that this is due to us calling a sleepy method from an atomic one... call_usermodhelper will eventually call wait_for_common which sleeps.  You do not want to sleep in an ISR routine.

The fix for this is to use a deferrable; we need to stop working in an interrupt context and schedule the non atomic work for future processing.

One possible solution is to use work queues for deferrable work. Here's an example implementation in github using work queues.

And here's an example session:

Linux: The pagecache and the loop back fs

Linux has a mechanism that allows you to create a block device that is backed by a file.  Most commonly, this is used to provide an encrpyted fs. All this is fine and dandy. However, you have to factor in that the Linux OS (And practically any other OS) will want to cache contents for block device in memory.

The reasoning here is. Accessing the contents of a file from a disk will cost you ~5ms. Rather than incur this cost on future reads, the OS caches the contents of the recently used file in a page cache. Future reads or writes to this file will hit the page cache which is orders of magnitude faster than your average disk.

This means that your writes will linger in memory until the your backing file's contents get evicted from the page cache. Using o_direct on files that are hosted within the loop backed fs won't help. You have to force pagecache evictions. The easiest way to do this is a call to fadvise and a sync to force pdflush to write your changes.

Here's an experiment:
I have 3 windows open.:

• One running blktrace  which shows VFS activity.
• One that has a oneshot dd.
• One that has a bunch of shell commands that poke around the loop mounted fs.

The experiment is executed in the following order:

1. An fadvise and a sync at the beginning to make sure the pagecache is clean and all writes are on the FS.
2. We also print out a couple of stats from /proc/meminfo.
3. We issue a dd call with direct I/O (oflag=direct).
4. Print stats from /proc/meminfo and take a look at the backing file using debugfs.
5. Show how much of the backing file is cached in the pagecache by using fincore.
6. Evict the pagecache using the fadvise command from linux-ftools.
7. Force a sync which wakes up pdflush to write out the dirty buffers.
8. Run debugfs to take a peek at the backing fs again.
9. Print out some more stats from meminfo.

Here's the script:
And the output generated:
The interesting bits to note here is that writes to a loop back filesystems in Linux are not guaranteed to be on disk until you evict the pagecache and force a sync.

If you are interested in further digging, here's debug output from:

blktrace:
Debugfs and dumpe2fs:

MYSQL load data: Stripping out zulu time and converting ip addresses to unsigned ints.

Snip for me: Storing IP addresses compactly in MySQL

RMySQL on OSX

I ran into an issue installing RMySQL on OSX. I am using a R from brew and MySQL from the Oracle download page. Trying to install RMySQL would throw up this error

Turns out that the right solution is really easy. Set your MySQL home R env var, make sure that libmysqlclient can be found by R, configure the RMySQL package with the proper inc & lib dirs for MySQL

Tests!

Vertical line markers in R using geom_vline

Say you have a two data sets that share a common dimension/axis only.  Here's an example:

# Event csv
timestamp,event
2013-04-03 22:59:05.061Z,A
2013-04-03 22:59:05.061Z,B
2013-04-03 22:59:07.109Z,C
2013-04-03 22:59:07.115Z,D
2013-04-03 22:59:07.209Z,E

# Performance data
hostname;interval;timestamp;CPU;user;nice;system;iowait;steal;idle
box1;1;2013-04-03 22:59:02 UTC;-1;10.53;0.00;2.01;0.50;0.00;86.97
box1;1;2013-04-03 22:59:03 UTC;-1;0.25;0.00;0.00;0.00;0.00;99.75
box1;1;2013-04-03 22:59:04 UTC;-1;0.00;0.00;0.25;0.25;0.00;99.50
box1;1;2013-04-03 22:59:05 UTC;-1;10.72;0.00;1.00;0.25;0.00;88.03
box1;1;2013-04-03 22:59:06 UTC;-1;10.67;0.00;10.67;0.00;0.25;78.41
box1;1;2013-04-03 22:59:07 UTC;-1;5.01;0.00;9.02;3.51;0.00;82.46
box1;1;2013-04-03 22:59:08 UTC;-1;12.28;0.00;11.53;4.26;0.25;71.68
box1;1;2013-04-03 22:59:09 UTC;-1;15.88;0.00;11.66;10.92;0.50;61.04

You'd like to plot these values on one graph; one overlaid over the other. Based on the data from the example above, you'd like to plot the CPU user against the timestamp metric, then you'd like to add in markers to show events over the chart.

Here's a gist:

 Obviously, we need a better legend, y axis label and a title for this graph.. That's left as an exercise.

R and GFS.

Playing with NOAA's GFS data with R.

Credit:

• http://braz.blogspot.com/2009/12/r-for-global-climate-forecasting.html 
• http://joewheatley.net/ncep-global-forecast-system/

Here are some sample images.

Pre-requisites

• Fetched data from NOAA (in particular this dataset).
• Installed wgrib2.
• Extracted the fields I was interested in from the grib2 file into netcdf

./grib2/wgrib2/wgrib2  -s  gfs.t12z.sfluxgrbf03.grib2  | grep :LAND | ./grib2/wgrib2/wgrib2 -i gfs.t12z.sfluxgrbf03.grib2 -netcdf land.netcdf
./grib2/wgrib2/wgrib2  -s  gfs.t12z.sfluxgrbf03.grib2  | grep TMP:2 | ./grib2/wgrib2/wgrib2 -i gfs.t12z.sfluxgrbf03.grib2 -netcdf temp2.netcdf
./grib2/wgrib2/wgrib2  -s  gfs.t12z.sfluxgrbf03.grib2  | grep VEG: | ./grib2/wgrib2/wgrib2 -i gfs.t12z.sfluxgrbf03.grib2 -netcdf vegetation.netcdf
./grib2/wgrib2/wgrib2  -s  gfs.t12z.sfluxgrbf03.grib2  | grep TCDC: | ./grib2/wgrib2/wgrib2 -i gfs.t12z.sfluxgrbf03.grib2 -netcdf cloud.netcdf

• You can also view the available fields using:

./grib2/wgrib2/wgrib2  -s  gfs.t12z.sfluxgrbf03.grib2  | less

• Or by visiting the description.

Having fun

Started R and installed netcdf and fields:

install.packages('ncdf')
install.packages('fields')

Then used the following gist in R.

Playing with data from NOAA Global Forecast System using python

Bugger! Blogger barfs out an error when I attempt to upload my ipython notebook. You can find the notebook on github or as a rendered html here.

Here are some pretty generated images:

Horn of Africa zoomed in (Mar 27 2013)

Global 2M temperature in degrees C (Mar 27 2013)

Kenya DOB density plot

Here is a density plot of a sample of Kenyans  based on their year of birth. It's interesting to see the dips correlate with turbulent times; political and military. In the diagram below, the largest unexplained dip is in the early 1970s. Anyone care to explain it; Google and Wikipedia don't return any results?
Click here for a larger image

EDIT:
Could it be the 1969 Kisumu incident?



 


Here is the code used to produce the graph in R

Hotfix for leaking filedescriptors

I just came across this neat gdb trick. Say you have a badly written program that's leaking file descriptors all over.. and it's in prod (the horror!) and it's imperative that you let it limp along till the patched binary comes around... GDB can close the file descriptors for you!

Here's how:

Leaky Program


Lsof tells it all. Take note of the two open descriptors to /etc/passwd


Playing around with variables in GDB. Can you guess why we chose the magic value 2147483646? For a hint, look at the last gist in this post


What does lsof think?


Bulk close of fds


Shell output from leaky

Gem of a fortune - I. M. Banks.

"Good afternoon, madam.  How may I help you?"

"Good afternoon.  I'd like a FrintArms HandCannon, please."

"A--?  Oh, now, that's an awfully big gun for such a lovely lady.  I
mean, not everybody thinks ladies should carry guns at all, though I
say they have a right to.  But I think... I might... Let's have a look
down here.  I might have just the thing for you.  Yes, here we are!
Look at that, isn't it neat?  Now that is a FrintArms product as well,
but it's what's called a laser -- a light-pistol some people call
them.  Very small, as you see; fits easily into a pocket or bag; won't
spoil the line of a jacket; and you won't feel you're lugging half a
tonne of iron around with you.  We do a range of matching accessories,
including -- if I may say so -- a rather saucy garter holster.  Wish I
got to do the fitting for that!  Ha -- just my little joke.  And
there's *even*... here we are -- this special presentation pack: gun,
charged battery, charging unit, beautiful glider-hide shoulder holster
with adjustable fitting and contrast stitching, and a discount on your
next battery.  Full instructions, of course, and a voucher for free
lessons at your local gun club or range.  Or there's the *special*
presentation pack; it has all the other one's got but with *two*
charged batteries and a night-sight, too.  Here, feel that -- don't
worry, it's a dummy battery -- isn't it neat?  Feel how light it is?
Smooth, see?  No bits to stick out and catch on your clothes, *and*
beautifully balanced.  And of course the beauty of a laser is, there's
no recoil.  Because it's shooting light, you see?  Beautiful gun,
beautiful gun; my wife has one.  Really.  That's not a line, she
really has.  Now, I can do you that one -- with a battery and a free
charge -- for ninety-five; or the presentation pack on a special
offer for one-nineteen; or this, the special presentation pack, for
one-forty-nine."

"I'll take the special."

"Sound choice, madam, *sound* choice.  Now, do--?"

"And a HandCannon, with the eighty-mill silencer, five GP clips, three
six-five AP/wire-flechettes clips, two bipropellant HE clips, and a
Special Projectile Pack if you have one -- the one with the embedding
rounds, not the signalers.  I assume the night-sight on this toy is
compatible?"

"Aah... yes,  And how does madam wish to pay?"

She slapped her credit card on the counter.  "Eventually."

        -- Iain M. Banks, "Against a Dark Background"

Automatic number plate detection

ALPR stands for automatic license plate recognition. The process typically consists of:

• Identifying potential license plate candidates from an image.
• Ranking the potential plate candidates and selecting the most likely one.
•  Identifying alphanumeric characters and converting them to text.

The hardest part of this process is reliably acquiring the plate candidate image with variables such as:

• Lighting conditions.
• Camera angle.
• Image noise
• Variations in the type of plates.
• Distance to plate; and consequently, the size of the plate in the image

In this article, I'll be going through a gist that works for a few images. I'll cover the literature review in a followup article.

Please note that this implementation has been developed and tested with very few samples; overfitting will probably reduce it's effectiveness.

1. Load an image using opencv and python (Line 33)
2. Run mean shift segmentation (I haven't evaluated the effectiveness of this) and save the image.
3. Use the segmented image as the camera image.
4. Convert the camera image (defaults as a numpy array) into OpenCV Mat and an IPL image (CV2 can work with numpy arrays. Plain old cv requires an IPL or Mat object).
5. Convert the camera image into grayscale.
6. Conduct thresholding on the image (Line 76). This means that pixels that are above the threshold become white and below black. This process is also called binarization.
7. Create a mask that will be used for blob extraction that covers the entire image; setting all the pixels to 1 means that the mask will be applied everywhere.
8. Invert the binarized image. License plates are mostly white with a few characters in black. Inversion means that the plate becomes mostly black with a few white character strokes (Line 94).
9. Blob detection parameters. We don't want very small or very large blobs.
10. Do the actual blob detection (Line 115).
11. Iterate over the grayscale image adding an RGB color << this is a bug
12. Mark potential blobs as black in a white image (Line 131).
13. Another threshold step << This is probably unnecessary.
14. Dilation and erosion to clean up the image << probably unnecessary.
15. Find contours over the black images.
16. For each contour, try and estimate it's overall shape. triangle? rectangle? n-gon (Line 196)? after eliminating very small and very large contours.
17. Draw a bounding box.
18. Calculate an estimate of the aspect ratio. Plates have a W/H ratio of 1 to 5
19. Off the cuff heuristic. Largest rectangle that has an acceptable aspect ratio and is a rectangle is probably a plate. This is not always true. You might much a window or a large rectangular object...
20. For the selected plate candidate, draw a bounding rect

Here are the results:

Segmented image

Thresholded image

Grayscale conversion

Blobs on a white image

Effects of erosion on an image

Effects of dilation on an image

Plate candidates - from contours

Min area box over plate candidate

Bounding box over plate candidate

And here are more images from a few more tests in picassa:

• Failures
• Successes

OSX: Y U No like 3G?

I recently came across an issue where my laptop would consume copious amounts of bandwidth. On average, the idle laptop would slurp up about 8MB of traffic/hour. In a day, that's ~ 100 MB of data gone.

In S. Africa and on 3G, this is prohibitively expensive. How expensive?
I am on Telkom 8ta, which charges

~ 40 Rand/100MB == 6USD/100MB. 

Scaling up this cost to a month adds up to

180USD/Month for ~ 3GB of data. 

 Obviously, this is a problem!

 So let's find out what's going on....

Tcpdump

Hmm, 5 similar queries? Obviously, there's something stuck in a dns lookup loop. This is not cool at all. If we do some math, the dns traffic adds up to the extra traffic that my 3G dongle is reporting. So how do we find out which process generates this data? We can use little snitch or better, use dtrace which comes in bundled with OSX.

DTrace

Dtrace allows you to hook up a probe that get's called once your kernel function is called. For example, say you know that all udp traffic in your box is generated via a specific syscall, you can 'watch' this syscall and generate some statistics around this.. (Basing this description on systemtap). So let's find out how a typical dns lookup works. The easiest program I can think that does dnslookup is nslookup. So let's truss up nslookup
The system call we are after is sendmsg and recvmsg (UDP send and udp recv). The exact packets sent and receiver are also unraveled in the packet tree below the dtruss output. The ardent reader will notice that the length of the UDP packet - 8 bytes == the syscall return output for sendmsg and recvmsg. For example, sendmsg has a return value of 32 bytes (40 UDP header - 2 bytes src port - 2 bytes dst port - 2 bytes length - 2bytes checksum). See line 32, 36, 101 and 172 of the gist above.

So we now know without a doubt that we ought to watch for a process that's generating a lot of recvmsg and sendmsg (At a rate of 10/sec for either syscall). Let's fire up another dtrace script

Fini

So let's try and validate our findings by stopping the buggy process. Phew! And that's it! A sample of 100 packets has 100 DNS requests when opendirectoryd is running. The 100 packets are captured in 3.5 seconds. With opendirectoryd stopped, it takes 23 seconds to capture 100 packets; DNS constitutes 30% of the sample. It's obvious that something is wrong with opendirectoryd. Stopping the process will save me tons of bucks until Apple issues a fix.