Merge multiple RRDs over time

Question

I've got an old RRD file that was only set up to track 1 year of history. I decided more history would be nice. I did rrdtool resize, and the RRD is now bigger. I've got old backups of this RRD file and I'd like to merge the old data in so that the up-to-date RRD also has the historical data.

I've tried the rrd contrib "merged-rrd.py" but it gives:

    $ python merged-rrd.py ../temperature-2010-12-06.rrd ../temperature-2011-05-24.rrd merged1.rrd
    merging old:../temperature-2010-12-06.rrd to new:../temperature-2011-05-24.rrd. creating merged rrd: merged1.rrd
    Traceback (most recent call last):
        File "merged-rrd.py", line 149, in <module>
            mergeRRD(old_path, new_path, mer_path)
        File "merged-rrd.py", line 77, in mergeRRD
            odict = getXmlDict(oxml)
        File "merged-rrd.py", line 52, in getXmlDict
            cf = line.split()[1]
    IndexError: list index out of range

Also tried "rrd_merger.pl":

    $ perl rrd_merger.pl --oldrrd=../temperature-2010-12-06.rrd --newrrd=../temperature-2011-05-24.rrd --mergedrrd=merged1.rrd
    Dumping ../temperature-2010-12-06.rrd to XML: /tmp/temperature-2010-12-06.rrd_old_8615.xml
    Dumping ../temperature-2011-05-24.rrd to XML: /tmp/temperature-2011-05-24.rrd_new_8615.xml
    Parsing ../temperature-2010-12-06.rrd XML......parsing completed
    Parsing ../temperature-2011-05-24.rrd XML...
    Last Update: 1306217100
    Start processing Round Robin DB
    Can't call method "text" on an undefined value at rrd_merger.pl line 61.
     at rrd_merger.pl line 286
     at rrd_merger.pl line 286

Is there a tool to combine or merge RRDs that works?

Answer 1

I ended up putting together a really simple script that works well enough for my case, by examining the existing python script.

http://gist.github.com/2166343

Answer 2

That fixed rrdtool-merge.pl for me:

<       my $xff         = $new_rra->first_child( 'xff' )->text;
---
>       my $xff         = $new_rra->first_child_text( 'xff' );

From XML::Twig documentation:

first_child_text ($optional_condition)
    Return the text of the first child of the element, or the first child
    matching the $optional_condition If there is no first_child then returns ''. This
    avoids getting the child, checking for its existence then getting the text for trivial
    cases.

Answer 3

The rrdmerge.pl utility, included with Routers2 in the /extras directory, can do this. Collect the latest version of Routers2 from http://www.steveshipway.org/software/rrd/

This is a utility I wrote for the purpose of merging multiple archived MRTG RRD files which sounds exactly like the situation you are mentioning.

This is probably too late for the OP but will hopefully be useful to later people who come here. It can merge any RRD files, even with different DS, RRA or intervals, and can generate XML or RRD, and will pick the best available data from the component RRD files to make the output.

Example:

rrdmerge.pl --rrd --libpath $RRDLIBDIR --output /tmp/merge.rrd --rows 12000 $FROMDIR/file.rrd $ARCHIVE/*.rrd

Answer 4

Looking at the XML file generated by rrdtool, there is a simple logic error in the Perl script. The elements AVERAGE and are simple enough but the tag is contained within tag with the text inside.

            <cf> AVERAGE </cf>
            <pdp_per_row> 1 </pdp_per_row> <!-- 300 seconds -->

            <params>
            <xff> 5.0000000000e-01 </xff>
            </params>

The parsing just has to be tweaked a bit and when it is working, the fix fed back here (where it is easy to 'Google') and also to the script's author for a fix.