Making sense of /proc/buddyinfo

/proc/buddyinfo gives you an idea about the free memory fragments on your Linux box. You get to view the free fragments for each available order, for the different zones of each numa node. The typical /proc/buddyinfo looks like this:


This box has a single numa node. Each numa node is an entry in the kernel linked list pgdat_list. Each node is further divided into zones. Here are some example zone types:

• DMA Zone: Lower 16 MiB of RAM used by legacy devices that cannot address anything beyond the first 16MiB of RAM.
• DMA32 Zone (only on x86_64): Some devices can't address beyond the first 4GiB of RAM. On x86, this zone would probably be covered by Normal zone
• Normal Zone: Anything above zone DMA and doesn't require kernel tricks to be addressable. Typically on x86, this is 16MiB to 896MiB. Many kernel operations require that the memory being used be from this zone
• Highmem Zone (x86 only): Anything above 896MiB.

Each zone is further divided into power of 2 (also known as the order) page sized chunks by the buddy allocator. The buddy allocator attempts to satisfy an allocation request from a zone's free pool. Over time, this free pool will fragment and higher order allocations will fail. The buddyinfo proc file is generated on demand by walking all the free lists.

Say we have just rebooted the machine and we have a free pool of 16MiB (DMA zone). The most sensible thing to do would be to have the this memory split into largest contiguous blocks available. The largest order is defined at compile time to 11 which means that the largest slice the buddy allocator has is 4MiB block (2^10 * page_size). so the 16 MiB DMA zone would initially split into 4 free blocks.

Here's how we'll service an allocation request for 72KiB:

1. Round up the allocation request to the next power of 2 (128)
2. Split a 4MiB chunk into two 2MiB chunks
3. Split  one 2 MiB chunk into two MiB chunks
4. Continue splitting until we get a 128KiB chunk that we'll allocate.

 Allocation requests will over time split, merge, split... this pool until we get to a point where we might have to fail a request due to the lack of a contiguous memory block.

Here's an example of an allocation failure from a Gentoo bug report.

In such cases, the buddyinfo proc file will allow you to view the current fragmentation state of your memory.
Here's a quick python script that will make this data more digestible.

And sample output for the buddyinfo data pasted earlier on.

Calculating the start time of a process

A quick script that calculate the start time of a process in Linux:

EXT4 err: couldn't mount because of unsupported optional features

Backward & Forwards compatibility is great! Until it bites you.

The extN branch of filesystem is the pretty much the standard on your average GNU/Linux installation. Migrations tend to work well if N is increasing i.e. from ext2->ext3->ext4. However, things are not so rosy if you want to mount your ext4 disk on your old PC.
Ext4 which made it into mainline in 2.6.27 introduces new features such as extents that are incompatible and unknown by older kernels. Attempting to mount an incompatible extN fs on your old kernel will fail and err out with logs similar to:

EXT3-fs: sda1: couldn't mount because of unsupported optional features (240).

EXT2-fs: sda1: couldn't mount because of unsupported optional features (240).

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,1)

The kernel here attempts to probe and mount the using the most recent extN. If that fails it tries the next ext filesystem and so on. In this case, this is a 2.6.16 kernel attempting to mount an ext4 fs with new features.

# dumpe2fs /tmp/img.ext4
.....
Filesystem volume name: /
Last mounted on: /mnt/tmp
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash

 The solution in this case is to use a newer kernel that understands ext4 or recompile your kernel to support ext4.

Linux kernel network backdoor

The ksplice blog has a very nice entry on hosting backdoors in hardware.
The quick summary of this backdoor is:

1. Register a protocol handler for an unused IP protocol number .
2. Call usermodhelper to execute the payload of the packet (skb->data).
3. Remote system now executes any command that you send it as root.

Unfortunately, it looks like the code is either out of date and/or buggy. Attempting to modprobe the backdoor module generates the following kernel call trace:

Further investigations reveal that this is due to us calling a sleepy method from an atomic one... call_usermodhelper will eventually call wait_for_common which sleeps.  You do not want to sleep in an ISR routine.

The fix for this is to use a deferrable; we need to stop working in an interrupt context and schedule the non atomic work for future processing.

One possible solution is to use work queues for deferrable work. Here's an example implementation in github using work queues.

And here's an example session:

Linux: The pagecache and the loop back fs

Linux has a mechanism that allows you to create a block device that is backed by a file.  Most commonly, this is used to provide an encrpyted fs. All this is fine and dandy. However, you have to factor in that the Linux OS (And practically any other OS) will want to cache contents for block device in memory.

The reasoning here is. Accessing the contents of a file from a disk will cost you ~5ms. Rather than incur this cost on future reads, the OS caches the contents of the recently used file in a page cache. Future reads or writes to this file will hit the page cache which is orders of magnitude faster than your average disk.

This means that your writes will linger in memory until the your backing file's contents get evicted from the page cache. Using o_direct on files that are hosted within the loop backed fs won't help. You have to force pagecache evictions. The easiest way to do this is a call to fadvise and a sync to force pdflush to write your changes.

Here's an experiment:
I have 3 windows open.:

• One running blktrace  which shows VFS activity.
• One that has a oneshot dd.
• One that has a bunch of shell commands that poke around the loop mounted fs.

The experiment is executed in the following order:

1. An fadvise and a sync at the beginning to make sure the pagecache is clean and all writes are on the FS.
2. We also print out a couple of stats from /proc/meminfo.
3. We issue a dd call with direct I/O (oflag=direct).
4. Print stats from /proc/meminfo and take a look at the backing file using debugfs.
5. Show how much of the backing file is cached in the pagecache by using fincore.
6. Evict the pagecache using the fadvise command from linux-ftools.
7. Force a sync which wakes up pdflush to write out the dirty buffers.
8. Run debugfs to take a peek at the backing fs again.
9. Print out some more stats from meminfo.

Here's the script:
And the output generated:
The interesting bits to note here is that writes to a loop back filesystems in Linux are not guaranteed to be on disk until you evict the pagecache and force a sync.

If you are interested in further digging, here's debug output from:

blktrace:
Debugfs and dumpe2fs:

Precedence for IPv4 vs IPv6

You can use /etc/gai.conf to set up your IPV4/IPV6 precedence as documented here and here.


Say we have two hosts www.he.net and www.ripe.net

$ host www.he.net
www.he.net is an alias for he.net.
he.net has address 216.218.186.2
he.net has IPv6 address 2001:470:0:76::2


$ host www.ripe.net
www.ripe.net has address 193.0.6.139
www.ripe.net has IPv6 address 2001:67c:2e8:22::c100:68b

Case 1: Prefer IPV4
Append the following to /etc/gai.conf

precedence ::ffff:0:0/96  100

then we have:

$ telnet www.ripe.net 80
Trying 193.0.6.139...
^C
$ telnet www.he.net 80
Trying 216.218.186.2...

Case 1: Prefer IPV6 for specific hosts
If we append

precedence 2001:470::/32 100

then we have

$ telnet www.ripe.net 80
Trying 193.0.6.139...
^C
$ telnet www.he.net 80
Trying 2001:470:0:76::2...
^C

So we seem to prefer that network for ipv6 and ipv4 everywhere else.


Case 3: Prefer ipv4 for specific hosts
Wondering if we invert the mask the reverse will be true.

Unix Permission modes

This is a long post covering unix permission modes. A file's mode is stored in it's inode structure. Most of this post is inline in the following shell interaction snippet(plain text here). The quirky bits are:

1. Permissions are analysed in increasing generality i.e User, Group, Others. The first match is accepted whether positive or negative
2. Executing a setuid binary is equivalent to executing that binary as the user owning the binary
3. Executing a setgid binary is equivalent to executing the binary as the group owning that binary
4. Binaries require setting the x bit only while shell scripts require r & x bits to be set.
5. If you setgid a directory and create a file in that directory, the file's group will be set to the group of the directory (Group permissions become inherited regardless of whether you were in that group).
6. Sticky bits on a directory restrict deletion to the owner of the directory, the creator of a file and the superuser. That's how /tmp works! It's also known as the restricted deletion flag.

# Lets make a couple of tests
$ test_read () { $(cat  2>/dev/null test.file > /dev/null ); if [ $? -eq 0 ];then echo "Read : True "; else echo "Read : False"; fi; }
$ test_write () { $(echo 2>/dev/null -e '#!/bin/sh\necho hello' > test.file ); if [ $? -eq 0 ];then echo "Write: True "; else echo "Write: False"; fi; }
$ test_exec () { $( ./test.file 2>/dev/null 1>/dev/null); if [ $? -eq 0 ];then echo "Exec : True "; else echo "Exec : False"; fi; }

# Touch a test.file
$ ls -l test.file
-rwx------ 1 lmwangi lmwangi 21 Apr 14 16:31 test.file

# Test mode as user 
$ for mode in $(seq -w 0 100 700); do echo "Mode $mode"; touch test.file; chmod $mode test.file; test_read; test_write; test_exec;echo;  done
Mode 000
Read : False
Write: False
Exec : False

Mode 100
Read : False
Write: False
Exec : False

Mode 200
Read : False
Write: True 
Exec : False

Mode 300
Read : False
Write: True 
Exec : False

Mode 400
Read : True 
Write: False
Exec : False

Mode 500
Read : True 
Write: False
Exec : True 

Mode 600
Read : True 
Write: True 
Exec : False

Mode 700
Read : True 
Write: True 
Exec : True 


# File permissions like firewall acls stop at the first match. If I am the owner of the file, matching stops with the user bits. Groups/others are never consulted... So we chgrp of file to a group you are not in say root and test.
$ sudo chgrp root test.file 
$ ls -l
total 4
----rwx--- 1 lmwangi root 21 Apr 14 16:36 test.file

$ for mode in $(seq -w 0 010 070); do echo "Mode $mode"; touch test.file; chmod $mode test.file; test_read; test_write; test_exec;echo;  done
Mode 000
Read : False
Write: False
Exec : False

Mode 010
Read : False
Write: False
Exec : False

Mode 020
Read : False
Write: False
Exec : False

Mode 030
Read : False
Write: False
Exec : False

Mode 040
Read : False
Write: False
Exec : False

Mode 050
Read : False
Write: False
Exec : False

Mode 060
Read : False
Write: False
Exec : False

Mode 070
Read : False
Write: False
Exec : False

# It get's interesting if I am not the owner of the file but I am in a group that owns the file.
$ id
uid=1000(lmwangi) gid=1000(lmwangi) groups=1000(lmwangi),20(dialout),24(cdrom),25(floppy),29(audio),44(video),46(plugdev),112(netdev),114(fuse),119(libvirt),1003(packetcapture)
$ ls -l
total 4
----rwx--- 1 root lmwangi 21 Apr 14 16:42 test.file

for mode in $(seq -w 0 010 070); do echo "Mode $mode"; sudo chmod $mode test.file; test_read; test_write; test_exec;echo;  done
Mode 000
Read : False
Write: False
Exec : False

Mode 010
Read : False
Write: False
Exec : False

Mode 020
Read : False
Write: True 
Exec : False

Mode 030
Read : False
Write: True 
Exec : False

Mode 040
Read : True 
Write: False
Exec : False

Mode 050
Read : True 
Write: False
Exec : True 

Mode 060
Read : True 
Write: True 
Exec : False

Mode 070
Read : True 
Write: True 
Exec : True 

# The same principle applies. Others mode only applies if I am neither the file owner nor in a group that owns the file.
$ for mode in $(seq -w 0 001 007); do echo "Mode $mode"; sudo chmod $mode test.file; test_read; test_write; test_exec;echo;  done
Mode 000
Read : False
Write: False
Exec : False

Mode 001
Read : False
Write: False
Exec : False

Mode 002
Read : False
Write: True 
Exec : False

Mode 003
Read : False
Write: True 
Exec : False

Mode 004
Read : True 
Write: False
Exec : False

Mode 005
Read : True 
Write: False
Exec : True 

Mode 006
Read : True 
Write: True 
Exec : False

Mode 007
Read : True 
Write: True 
Exec : True 

# It gets better with executables :)
# Let's make a small template executable
$  echo -e '#include <stdio.h>\nint main(){printf("hello\\n"); return 0;}' > hello.c && gcc hello.c -o hello
$ ./hello 
hello
# Now watch mode 100 compare with the a shell script of mode 100. Do the same for 300, 500 & 700
Mode 000
Read : False
Write: False
Exec : False

Mode 100
Read : False
Write: False
Exec : True 
Mode 200
Read : False
Write: True 
Exec : False

Mode 300
Read : False
Write: True 
Exec : True 

Mode 400
Read : True 
Write: False
Exec : False

Mode 500
Read : True 
Write: False
Exec : True 

Mode 600
Read : True 
Write: True 
Exec : False

Mode 700
Read : True 
Write: True 
Exec : True 

# As can be seen. You must have r & x to exec a shell script while your require only x for a binary.
# Recap again
$ for mode in $(seq -w 0 100 700); do echo $mode; chmod $mode test.file; ls -l test.file |grep test.file; ./test.file; done
000
---------- 1 lmwangi lmwangi 6695 Apr 14 17:06 test.file
-bash: ./test.file: Permission denied
100
---x------ 1 lmwangi lmwangi 6695 Apr 14 17:06 test.file
hello
200
--w------- 1 lmwangi lmwangi 6695 Apr 14 17:06 test.file
-bash: ./test.file: Permission denied
300
--wx------ 1 lmwangi lmwangi 6695 Apr 14 17:06 test.file
hello
400
-r-------- 1 lmwangi lmwangi 6695 Apr 14 17:06 test.file
-bash: ./test.file: Permission denied
500
-r-x------ 1 lmwangi lmwangi 6695 Apr 14 17:06 test.file
hello
600
-rw------- 1 lmwangi lmwangi 6695 Apr 14 17:06 test.file
-bash: ./test.file: Permission denied
700
-rwx------ 1 lmwangi lmwangi 6695 Apr 14 17:06 test.file
hello


# Now let's look at the  setuid/setgid/sticky bits. 
1 = Sticky - if set not relevant in a binary in my system [Linux]
2 = SetGID - if set the progran runs with eGID set to the binary group
4 = SetUID - if set the program runs with euid set tot the binary owner.

# Let's make a program that shows us the uid, effective uid, gid and effective gid
$ echo -e '#include <stdio.h>\n#include <unistd.h>\nint main(){printf("hello\\nUID:%d\\nEUID:%d\\nGID:%d\\nEGID:%d\\n",getuid(),geteuid(),getgid(),getegid()); return 0;}' > hello.c && gcc hello.c -o hello                                                      

$ ./hello 
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

$ id
uid=1000(lmwangi) gid=1000(lmwangi) groups=1000(lmwangi),20(dialout),24(cdrom),25(floppy),29(audio),44(video),46(plugdev),112(netdev),114(fuse),119(libvirt),1003(packetcapture)

$ for owner in lmwangi:lmwangi lmwangi:root root:lmwangi root:root; do echo ">>>> $owner <<<<"; for mode in 1777 2777 4777 6777; do echo "mode $mode"; cp hello test.file ; sudo chown $owner test.file; sudo chmod $mode test.file; ls -l test.file; ./test.file;echo; done;  done
>>>> lmwangi:lmwangi <<<<
mode 1777
-rwxrwxrwt 1 lmwangi lmwangi 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

mode 2777
-rwxrwsrwx 1 lmwangi lmwangi 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

mode 4777
-rwsrwxrwx 1 lmwangi lmwangi 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

mode 6777
-rwsrwsrwx 1 lmwangi lmwangi 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

>>>> lmwangi:root <<<<
mode 1777
-rwxrwxrwt 1 lmwangi root 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

mode 2777
-rwxrwsrwx 1 lmwangi root 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:0

mode 4777
-rwsrwxrwx 1 lmwangi root 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

mode 6777
-rwsrwsrwx 1 lmwangi root 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:0

>>>> root:lmwangi <<<<
mode 1777
-rwxrwxrwt 1 root lmwangi 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

mode 2777
-rwxrwsrwx 1 root lmwangi 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

mode 4777
-rwsrwxrwx 1 root lmwangi 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:0
GID:1000
EGID:1000

mode 6777
-rwsrwsrwx 1 root lmwangi 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:0
GID:1000
EGID:1000

>>>> root:root <<<<
mode 1777
-rwxrwxrwt 1 root root 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:1000

mode 2777
-rwxrwsrwx 1 root root 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:1000
GID:1000
EGID:0

mode 4777
-rwsrwxrwx 1 root root 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:0
GID:1000
EGID:1000

mode 6777
-rwsrwsrwx 1 root root 7205 Apr 14 17:31 test.file
hello
UID:1000
EUID:0
GID:1000
EGID:0

# Sticky bit (RESTRICTED DELETION FLAG) on directories 

$ mkdir test; sudo chown root:root test; sudo chmod 1777 test; ls -l |grep test
drwxrwxrwt 2 root    root    4096 Apr 14 17:39 test
-rwsrwsrwx 1 root    root    7205 Apr 14 17:31 test.file

$ echo lmwangi > test/by_lmwangi
$ ls -l test/
total 4
-rw-r--r-- 1 lmwangi lmwangi 8 Apr 14 17:41 by_lmwangi

# No one can remove the file
$ sudo su guest -c "rm test/by_lmwangi"
rm: remove write-protected regular file `test/by_lmwangi'? y
rm: cannot remove `test/by_lmwangi': Operation not permitted

# Root can  override though or the owner of the dir
$ sudo rm test/by_lmwangi 

$ rm -rf test
$ mkdir test
$ chmod 1777 test
$ sudo su guest -c "echo hello > test/by_guest"
$ ls -l test
total 4
-rw-r--r-- 1 guest guest 6 Apr 14 17:38 by_guest
$ ls -l |grep test
drwxrwxrwt 2 lmwangi lmwangi 4096 Apr 14 17:38 test
-rwsrwsrwx 1 root    root    7205 Apr 14 17:31 test.file
$ cat test/by_guest 
hello
$ rm test/by_guest 
rm: remove write-protected regular file `test/by_guest'? y

$ ls -l test
total 0

# And we go out with a bang
# setuid on dir == no change on files                                                                  
$ rm -rf test && mkdir test; sudo chown root:root test; sudo chmod 4777 test; ls -l|grep test
$ cd test/
$ touch aha
$ ls -l
total 0
-rw-r--r-- 1 lmwangi lmwangi 0 Apr 14 17:46 aha
$ cd ..

# But setgid on dir == changes any file created in it to the group owning the dir
$ rm -rf test && mkdir test; sudo chown root:root test; sudo chmod 2777 test; ls -l|grep test
drwxrwsrwx 2 root    root    4096 Apr 14 17:47 test
-rwsrwsrwx 1 root    root    7205 Apr 14 17:31 test.file
$ cd test/
$ touch aha
$ ls -l
total 0
-rw-r--r-- 1 lmwangi root 0 Apr 14 17:47 aha
$ id
uid=1000(lmwangi) gid=1000(lmwangi) groups=1000(lmwangi),20(dialout),24(cdrom),25(floppy),29(audio),44(video),46(plugdev),112(netdev),114(fuse),119(libvirt),1003(packetcapture)

Unix is loads of fun :)

Poweroff or poweron usb devices in Linux

Sometimes, I need my external drive to suspend/spin down when I am suspending my machine.

Use lsusb to replace the vendor/product ids.

In my case

$ lsusb

...

Bus 001 Device 008: ID 1058:0704 Western Digital Technologies, Inc. Passport External HDD

...

~/bin/poweroff-usbdisk.sh

#!/bin/sh
# Poweroff a device identified by the vendor/product id below.
# If you have two similar devices (e.g two western digital drives),
# the script will fail. The fix is easy: enumerate the devices
# and suspend them individually. 

vendorid=1058
productid=0704

# Look for western digital
USBDIR=$(dirname $(find  /sys/bus/usb/devices/usb[1-5]/ -iname "*vendor*"|xargs grep $vendorid|cut -f1 -d: ))

# Verify it's the hdd
grep $productid $USBDIR/idProduct

#Power off 
if [ $? -eq 0 ]; then
sudo umount /mnt/audio/ && echo suspend |sudo tee $USBDIR/power/level;
fi
~/bin/poweron-usbdisk.sh

#!/bin/sh

# Poweron a device identified by the vendor/product id below.

# Look for western digital
USBDIR=$(dirname $(find  /sys/bus/usb/devices/usb[1-5]/ -iname "*vendor*"|xargs grep $vendorid |cut -f1 -d: ))
vendorid=1058
productid=0704


# Verify it's the hdd
grep $productid $USBDIR/idProduct

#Power off 
if [ $? -eq 0 ]; then  echo on |sudo tee $USBDIR/power/level; fi