Hotfix for leaking filedescriptors

I just came across this neat gdb trick. Say you have a badly written program that's leaking file descriptors all over.. and it's in prod (the horror!) and it's imperative that you let it limp along till the patched binary comes around... GDB can close the file descriptors for you!

Here's how:

Leaky Program

	#include <stdio.h>
	#include <unistd.h>
	#include <string.h>
	void open_file(char *filename)
	{
	FILE *fd = fopen(filename, "r");
	}
	int main(void)
	{
	char filename[FILENAME_MAX];
	strcpy(filename, "/etc/passwd");
	int iteration = 0;
	while (1) {
	iteration++;
	printf("Iteration %d\n", iteration);
	open_file(filename);
	sleep(60);
	}
	return 0;
	}

view raw leaky.c hosted with ❤ by GitHub

Lsof tells it all. Take note of the two open descriptors to /etc/passwd

	$ lsof -p $(pgrep leaky)
	COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
	leaky 16095 laban cwd DIR 0,30 4096 10936796 /home/laban/devel/private/learning/c/leak_descriptors
	leaky 16095 laban rtd DIR 252,1 4096 2 /
	leaky 16095 laban txt REG 0,30 11267 10919599 /home/laban/devel/private/learning/c/leak_descriptors/leaky
	leaky 16095 laban mem REG 252,1 1811128 19136595 /lib/x86_64-linux-gnu/libc-2.15.so
	leaky 16095 laban mem REG 252,1 149280 19136932 /lib/x86_64-linux-gnu/ld-2.15.so
	leaky 16095 laban 0u CHR 136,19 0t0 22 /dev/pts/19
	leaky 16095 laban 1u CHR 136,19 0t0 22 /dev/pts/19
	leaky 16095 laban 2u CHR 136,19 0t0 22 /dev/pts/19
	leaky 16095 laban 3r REG 252,1 2667 1575354 /etc/passwd
	leaky 16095 laban 4r REG 252,1 2667 1575354 /etc/passwd

view raw lsof hosted with ❤ by GitHub

Playing around with variables in GDB. Can you guess why we chose the magic value 2147483646? For a hint, look at the last gist in this post

	$ sudo gdb -p $(pgrep leaky)
	GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
	Copyright (C) 2012 Free Software Foundation, Inc.
	License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
	This is free software: you are free to change and redistribute it.
	There is NO WARRANTY, to the extent permitted by law. Type "show copying"
	and "show warranty" for details.
	This GDB was configured as "x86_64-linux-gnu".
	For bug reporting instructions, please see:
	<http://bugs.launchpad.net/gdb-linaro/>.
	Attaching to process 16095
	Reading symbols from /home/laban/devel/private/learning/c/leak_descriptors/leaky...done.
	Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.15.so...done.
	done.
	Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
	Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/lib/x86_64-linux-gnu/ld-2.15.so...done.
	done.
	Loaded symbols for /lib64/ld-linux-x86-64.so.2
	0x00007f253e7cf820 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:82
	82 ../sysdeps/unix/syscall-template.S: No such file or directory.
	(gdb)
	(gdb) bt
	#0 0x00007f253e7cf820 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:82
	#1 0x00007f253e7cf6dc in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138
	#2 0x0000000000400620 in main () at leaky.c:21
	(gdb) up
	#1 0x00007f253e7cf6dc in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138
	138 ../sysdeps/unix/sysv/linux/sleep.c: No such file or directory.
	(gdb) up
	#2 0x0000000000400620 in main () at leaky.c:21
	21 sleep(60);
	(gdb) set filename="/etc/group"
	...
	... Awhile later
	...
	(gdb) set iteration=4
	...
	... Awhile later
	...
	(gdb) set iteration=2147483646
	(gdb) p iteration
	$1 = 2147483646
	(gdb) quit
	A debugging session is active.
	Inferior 1 [process 16095] will be detached.
	Quit anyway? (y or n) y
	Detaching from program: /home/laban/devel/private/learning/c/leak_descriptors/leaky, process 16095

view raw gistfile1.txt hosted with ❤ by GitHub

What does lsof think?

	$ lsof -p $(pgrep leaky)
	COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
	leaky 16095 laban cwd DIR 0,30 4096 10936796 /home/laban/devel/private/learning/c/leak_descriptors
	leaky 16095 laban rtd DIR 252,1 4096 2 /
	leaky 16095 laban txt REG 0,30 11267 10919599 /home/laban/devel/private/learning/c/leak_descriptors/leaky
	leaky 16095 laban mem REG 252,1 1811128 19136595 /lib/x86_64-linux-gnu/libc-2.15.so
	leaky 16095 laban mem REG 252,1 149280 19136932 /lib/x86_64-linux-gnu/ld-2.15.so
	leaky 16095 laban 0u CHR 136,19 0t0 22 /dev/pts/19
	leaky 16095 laban 1u CHR 136,19 0t0 22 /dev/pts/19
	leaky 16095 laban 2u CHR 136,19 0t0 22 /dev/pts/19
	leaky 16095 laban 3r REG 252,1 2667 1575354 /etc/passwd
	leaky 16095 laban 4r REG 252,1 2667 1575354 /etc/passwd
	leaky 16095 laban 5r REG 252,1 1210 1575723 /etc/group
	leaky 16095 laban 6r REG 252,1 1210 1575723 /etc/group
	leaky 16095 laban 7r REG 252,1 1210 1575723 /etc/group
	leaky 16095 laban 8r REG 252,1 1210 1575723 /etc/group

view raw lsof-output2.sh hosted with ❤ by GitHub

Bulk close of fds

	Leaks all over the place
	$ lsof -p $(pgrep leaky)
	COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
	leaky 16095 laban cwd DIR 0,30 4096 10936796 /home/laban/devel/private/learning/c/leak_descriptors
	leaky 16095 laban rtd DIR 252,1 4096 2 /
	leaky 16095 laban txt REG 0,30 11267 10919599 /home/laban/devel/private/learning/c/leak_descriptors/leaky
	leaky 16095 laban mem REG 252,1 1811128 19136595 /lib/x86_64-linux-gnu/libc-2.15.so
	leaky 16095 laban mem REG 252,1 149280 19136932 /lib/x86_64-linux-gnu/ld-2.15.so
	leaky 16095 laban 0u CHR 136,19 0t0 22 /dev/pts/19
	leaky 16095 laban 1u CHR 136,19 0t0 22 /dev/pts/19
	leaky 16095 laban 2u CHR 136,19 0t0 22 /dev/pts/19
	leaky 16095 laban 3r REG 252,1 2667 1575354 /etc/passwd
	leaky 16095 laban 4r REG 252,1 2667 1575354 /etc/passwd
	leaky 16095 laban 5r REG 252,1 1210 1575723 /etc/group
	leaky 16095 laban 6r REG 252,1 1210 1575723 /etc/group
	leaky 16095 laban 7r REG 252,1 1210 1575723 /etc/group
	leaky 16095 laban 8r REG 252,1 1210 1575723 /etc/group
	GDB closing fds
	$ sudo gdb -p $(pgrep leaky)
	GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
	Copyright (C) 2012 Free Software Foundation, Inc.
	License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
	This is free software: you are free to change and redistribute it.
	There is NO WARRANTY, to the extent permitted by law. Type "show copying"
	and "show warranty" for details.
	This GDB was configured as "x86_64-linux-gnu".
	For bug reporting instructions, please see:
	<http://bugs.launchpad.net/gdb-linaro/>.
	Attaching to process 16095
	Reading symbols from /home/laban/devel/private/learning/c/leak_descriptors/leaky...done.
	Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.15.so...done.
	done.
	Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
	Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/lib/x86_64-linux-gnu/ld-2.15.so...done.
	done.
	Loaded symbols for /lib64/ld-linux-x86-64.so.2
	0x00007f253e7cf820 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:82
	82 ../sysdeps/unix/syscall-template.S: No such file or directory.
	(gdb) set $max=8
	(gdb) set $current=3
	(gdb) while ($current < $max)
	> p close($current++)
	>end
	$5 = -1
	$6 = 0
	$7 = 0
	$8 = 0
	$9 = 0
	(gdb) ^Z
	[2]+ Stopped sudo gdb -p $(pgrep leaky)
	What lsof thinks after GDB's done
	$ lsof -p $(pgrep leaky)
	COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
	leaky 16095 laban cwd DIR 0,30 4096 10936796 /home/laban/devel/private/learning/c/leak_descriptors
	leaky 16095 laban rtd DIR 252,1 4096 2 /
	leaky 16095 laban txt REG 0,30 11267 10919599 /home/laban/devel/private/learning/c/leak_descriptors/leaky
	leaky 16095 laban mem REG 252,1 1811128 19136595 /lib/x86_64-linux-gnu/libc-2.15.so
	leaky 16095 laban mem REG 252,1 149280 19136932 /lib/x86_64-linux-gnu/ld-2.15.so
	leaky 16095 laban 0u CHR 136,19 0t0 22 /dev/pts/19
	leaky 16095 laban 1u CHR 136,19 0t0 22 /dev/pts/19
	leaky 16095 laban 2u CHR 136,19 0t0 22 /dev/pts/19
	leaky 16095 laban 8r REG 252,1 1210 1575723 /etc/group

view raw gistfile1.txt hosted with ❤ by GitHub

Shell output from leaky

	$ ./leaky
	Iteration 1
	Iteration 2
	Iteration 5
	Iteration 6
	Iteration 2147483647
	Iteration -2147483648
	Iteration -2147483647

view raw gistfile1.txt hosted with ❤ by GitHub