Monday, February 28, 2011

Importing old data into munin

This python script tries to figure out the original rrdtool create parameters that were used to create a given rrd. It's very basic (handles only RRA:Average afaik) so don't expect magic from it. It expects rrdtool to be your path.

Sample invocation:
lmwangi@jaghpu:~/rrd$ python rrdinfo-parser.py -f all.rrd
rrdtool create all.rrd --start 978300900 --step 300 \
DS:a:COUNTER:600:U:U \
DS:c:DERIVE:600:U:U \
DS:b:GAUGE:600:U:U \
DS:d:ABSOLUTE:600:U:U \
RRA:AVERAGE:0.5:1:10 \

lmwangi@jaghpu:~/rrd$ python rrdinfo-parser.py -f test.rrd
rrdtool create test.rrd --start 920804400 --step 300 \
DS:speed:COUNTER:600:U:U \
RRA:AVERAGE:0.5:1:24 \
RRA:AVERAGE:0.5:6:10 \

So say we have a Munin derived rrd that we need to import old data into. First we run the script to extract the schema, then we edit the --start parameter to an epoch timestamp that predates your data and finally recreate the rrd.
rrdtool create service-logs-total-g.rrd --start 1108598400 --step 300 \                                                   
DS:42:GAUGE:600:U:U \
RRA:AVERAGE:0.5:1:576 \
RRA:MIN:0.5:1:576 \
RRA:MAX:0.5:1:576 \
RRA:AVERAGE:0.5:6:432 \
RRA:MIN:0.5:6:432 \
RRA:MAX:0.5:6:432 \
RRA:AVERAGE:0.5:24:540 \
RRA:MIN:0.5:24:540 \
RRA:MAX:0.5:24:540 \
RRA:AVERAGE:0.5:288:450
Then we import the processed data (in my case, log file summaries) which looks like:
head 2010.data
1262296800:4241:221:173:276
1262297400:3920:197:155:231
1262298000:4171:184:226:208
1262298600:3700:197:159:244
1262299200:3350:195:166:227
where the fields are ts:total:errors:unknowns:etc.
Using a simple awk script we extract field one and two (timestamp and total) and import them into the rrd and when we are done, overwrite the original rrd.
FILE=service-logs-total-g.rrd
for record in $(cat 2010.data|awk -F: '{print $1":"$2}');
do
rrdupdate $FILE $record;
done

No comments:

Post a Comment