Skip navigation
9816 Views 6 Replies Latest reply: Dec 5, 2012 1:39 PM by gcollis RSS
spawn!!! Rank: White Belt 36 posts since
Jan 26, 2011
Currently Being Moderated

Feb 3, 2011 8:19 AM

How long does RRD file accumulate the data?

Hi All,

 

I want to know how long RRD files can accumulate the data received in Zenoss? or does it has dumping mechanism which deletes or dump old data in the file after 1 month or certain period of time span?

 

Thanks in advance

  • jskeane Rank: Green Belt 67 posts since
    Sep 15, 2010
    Currently Being Moderated
    1. Feb 3, 2011 12:31 PM (in response to spawn!!!)
    Re: How long does RRD file accumulate the data?

    The answer to your question is somewhat complex.  First, it depends on exactly which RRD file you look at; Zenoss does not create them all with exactly the same structure.  Second, each Zenoss RRD file actually contains several archives (the one I just looked at, the system uptime for a Linux server, had seven, four for averages and three for maximum values - we'll ignore the maximum value ones for this discussion)  Each of these archives spans a different period of time.  The first one just stores the data points as they come in.  That happens every five minutes (specified as 300 seconds for the RRD "step" size).  The database I looked at has 600 records in each of the archives, so the first archive can store data for 50 hours.  After 50 hours, each new data point overwrites the oldest data point in that archive; it's a circular buffer.

     

    But that's not the whole story.  The second archive in that same database file averages six of the input data points to create each one of its data points.  So each data point in that archive gets added every 30 minutes instead of every five minutes.  That archive has 600 records, so it covers 12.5 days.

     

    The next archive averages 24 data points for each of its data points.  5 minutes X 24 X 600 = 14,400 minutes = 240 hours = 50 days.  The last archive averages 288 points for each of its points, so it spans 600 days, or almost two years.

     

    When Zenoss requests data from the RRD, the RRD software looks so see which archives cover the time span being requested.  If there is more than one, it picks the one with the least amount of time between samples.  In general, the greater the time span you request or the farther back in time you go, the further down the list of archives you have to go to find one that will cover it.  The data that is returned will have poorer time resolution than the archives that average fewer data points but cannot cover the requested time span.

     

    So, for the RRD file that I looked at, the answer to your question is 600 days for the coarsest resolution data.  You can look at your own files with "rrdtool info <filename>".  To determine how often input data points are expected, look for something like "step=300" to tell you the base RRD step size in seconds.  Then for the round robin archives, look for "rra[<num>].pdp_per_row" to see how many of those points are averaged (or fed into a min or max function, see "rra[<num>].cf" for the function being used) to create a point for the archive.  Take the biggest pdp_per_row for that data source, multiply that by the step size, and then multiply that by the number of records in the archive (rra[<num>].rows) and you will have the time span in seconds for the longest duration archive before new data starts overwriting old data.

     

    Regards,

     

    jskeane

  • jmp242 ZenossMaster 4,060 posts since
    Mar 7, 2007
    Currently Being Moderated
    2. Feb 3, 2011 4:07 PM (in response to spawn!!!)
    Re: How long does RRD file accumulate the data?

    You'll want to search the forums, but it's available out to 720 days averaged across longer timeframes the further back you go.

     

    --

    James Pulver

    Information Technology Area Supervisor

    LEPP Computer Group

    Cornell University

  • phonegi Rank: Brown Belt 446 posts since
    Apr 15, 2009
    Currently Being Moderated
    3. Feb 3, 2011 11:14 PM (in response to spawn!!!)
    Re: How long does RRD file accumulate the data?

    To fully understand how rrd files store data, take a look here:

    http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html

     

    Also notice that there are two ways to define the contents of an rrd file:

    First, an SNMP data point in a template can be assigned a custom Create Command. This consists of RRA lines (see link above). If you leave this setting blank, the system will use the system-wide default rrd create command which is found at Advanced > Collectors > localhost > Edit > Default RRD Create Command.

  • Rank: Brown Belt 260 posts since
    Mar 30, 2009
    Currently Being Moderated
    5. Jul 22, 2011 1:07 PM (in response to spawn!!!)
    Re: How long does RRD file accumulate the data?

    I want to provide some more detailed information from something that I wrote up. It should make calculating this RRD stuff easier.

     

    Default RRD settings

    RRDTool does data consolidation, not aggregation. You can read about how it does this under the RRA section of http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html.  You can define what RRAs Zenoss creates by setting the "Default RRD  Create Command" under the Edit tab for your performance monitor.

    Here are what we use by default:

    RRA:AVERAGE:0.5:1:600 > average on single data point, stored 600 times, 300s * 600 = 180.000s = 5min average for 50h
    RRA:AVERAGE:0.5:6:600
    > average on 6 data points (30 min), stored 600 times, (300s*6) * 600 = 1.080.000s = 30min average for 12.5d
    RRA:AVERAGE:0.5:24:600
    > average on 24 data points (2 hours), stored 600 times, (300s*24) * 600 = 4.320.000s = 2hour average for 50d
    RRA:AVERAGE:0.5:288:600
    > average on 288 data points (24 hours), stored 600 times, (300s*288) * 600 = 24h average for ~ 2 years
    RRA:MAX:0.5:6:600
    > 50h
    RRA:MAX:0.5:24:600
    > 50d
    RRA:MAX:0.5:288:600
    > ~ 2 years

    File Size for one data point: 35k
    At 100.000 data points performance data will consume 3.3GB

    Of course the more of these archives you create, and the more "rows"  you put in them, the larger each RRD file will be on the disk. This will  in turn use more cache memory to remain up to date, and thus give you  less monitoring capacity per Zenoss collector.


    Example RRD setups

    Here are some other examples of RRD archives one might choose to setup:

    RRA:AVERAGE:0.5:1:8640 > average on a single data point, stored 8640 times = 30d (this is the as collected data)
    RRA:AVERAGE:0.5:6:2880
    > 30min average for 60d
    RRA:AVERAGE:0.5:12:1872
    > 60min average for 90d
    RRA:AVERAGE:0.5:288:600
    > 1 day average for 2 years
    RRA:MAX:0.5:1:8640
    >
    RRA:MAX:0.5:12:1872
    > same as above just max instead of average
    RRA:MAX:0.5:288:600
    >
    File Size: 198k
    At 100.000 data points performance data will consume 18.9GB


    RRA:AVERAGE:0.5:1:25920 > average on a single data point, stored 25920 times = 90d (this is the as collected data)
    RRA:AVERAGE:0.5:6:2880
    > 30min average for 60d
    RRA:AVERAGE:0.5:12:1872
    > 60min average for 90d
    RRA:AVERAGE:0.5:288:600
    > 1 day average for 2 years
    RRA:MAX:0.5:1:25920
    >
    RRA:MAX:0.5:12:1872
    > same as above just max instead of average
    RRA:MAX:0.5:288:600
    >
    File Size: 468k
    At 100.000 data points performance data will consume 38.8GB


    RRA:AVERAGE:0.5:1:4032 > average on a single data point, stored 4032 times = 14d (this is the as collected data)
    RRA:AVERAGE:0.5:12:1440
    > 60min average for 60d
    RRA:AVERAGE:0.5:288:180
    > 1 day average for 6mo
    RRA:AVERAGE:0.5:2016:52
    > 1 week average for 1 year
    RRA:AVERAGE:0.5:8064:60
    > 1 month average for 5 years
    RRA:AVERAGE:0.5:96768:5
    > 1 year average for 5 years
    RRA:MAX:0.5:1:4032
    >
    RRA:MAX:0.5:12:1440
    > same as above just max instead of average
    RRA:MAX:0.5:288:180
    >

    File Size: 91k
    At 100.000 data points performance data will consume 8.7GB

  • gcollis Rank: White Belt 19 posts since
    Nov 5, 2010
    Currently Being Moderated
    6. Dec 5, 2012 1:39 PM (in response to Nick Yeates)
    Re: How long does RRD file accumulate the data?

    How would I go about changing the derault RRD values, using the default polling interval, to something like:

     

    15min average for 60 days

    30min average for for 90 days

    30min average for 1 year

     

    RRA:AVERAGE:0.5:1:8640 5 min average for 30 days

    RRA:AVERAGE:0.5:3:288 > 15min average for 60d

    RRA:AVERAGE:0.5:6:4320 > 30min average for 90d

    RRA:AVERAGE:0.5:6:17532 > 30min average for 1 year

    RRA:MAX:0.5:1:8640 >

    RRA:MAX:0.5:6:4320 > same as above just max instead of average

    RRA:MAX:0.5:6:17532 >

     

    How can I work out what the filesize and data consumption would be at 100.000 data points? I have a 20GB partition to work with.

     

    Is anyone working at this level of RRD granularity? I wonder if I need to alter the memory profile of my server, and how this will affect my zenoss collector(s).

     

    Thanks

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 4 points
  • Helpful Answers - 2 points