Monday, April 2, 2012

fgrep if you don't need regexes

I did know that fgrep was faster but I didn't know by how much!

So looking up a set of values (say IP address) from somefilesource in a bunch of logs (300 files with ~ 500k lines)

$ wc -l Some*
 3840 SomeFilesource

$ time fgrep  -hf SomeFilesource *log* > Wantedlogs
real 0m0.952s
user 0m0.890s
sys 0m0.030s


time grep -f  SomeFilesource *log* > Wantedlogs
real 33m45.601s
user 33m39.150s
sys 0m0.350s

That is quite a speed up. From now on it's fgrep by default... Now if I need to see whether the --mmap optimization speeds up subsequent [f]greps.