RRD Graph stopped graphing
list Michael Beatty
I had two custom RRD graphs setup and running fine on client systems. The script that collects the data was running every minute. After I finished testing, I changed the frequency of the scripts on the clients to 1 hour in the clientlaunch.cfg file. When I did, the RRD graph stopped displaying data. It ran like this for a couple days. I changed clientlaunch.cfg back to 1 minute... and the graphs immediately started graphing again. Is there a minimum time frame that is needed, or something else that must be done? What am I missing? -- Michael Beatty
list Larry Barber
When you change the frequency of data collection you need to update your rrd definitions. The rrd databases are fixed length files, so you are leaving lots of blank entries between data points. Thanks, Larry Barber On Tue, Dec 18, 2012 at 11:42 AM, Michael Beatty <user-4aea7c115850@xymon.invalid
▸
wrote:
I had two custom RRD graphs setup and running fine on client systems. The script that collects the data was running every minute. After I finished testing, I changed the frequency of the scripts on the clients to 1 hour in the clientlaunch.cfg file. When I did, the RRD graph stopped displaying data. It ran like this for a couple days. I changed clientlaunch.cfg back to 1 minute... and the graphs immediately started graphing again. Is there a minimum time frame that is needed, or something else that must be done? What am I missing? -- Michael Beatty ______________________________**
Xymon at xymon.com<
list Michael Beatty
How do I do that? Michael Beatty Sherwin-Williams IT Analyst/Developer user-4aea7c115850@xymon.invalid XXX-XXX-XXXX
▸
On 12/18/2012 02:13 PM, Larry Barber wrote:When you change the frequency of data collection you need to update
your rrd definitions. The rrd databases are fixed length files, so you
are leaving lots of blank entries between data points.
Thanks,
Larry Barber
On Tue, Dec 18, 2012 at 11:42 AM, Michael Beatty
<user-4aea7c115850@xymon.invalid <mailto:user-4aea7c115850@xymon.invalid>> wrote:
I had two custom RRD graphs setup and running fine on client
systems. The script that collects the data was running every
minute. After I finished testing, I changed the frequency of the
scripts on the clients to 1 hour in the clientlaunch.cfg file.
When I did, the RRD graph stopped displaying data. It ran like
this for a couple days. I changed clientlaunch.cfg back to 1
minute... and the graphs immediately started graphing again. Is
there a minimum time frame that is needed, or something else that
must be done? What am I missing?
--
Michael Beatty
list Tim McCloskey
Looks like I forget to reply all, so this did not hit the group. From: Tim McCloskey Sent: Tuesday, December 18, 2012 10:18 AM To: Michael Beatty Subject: RE: [Xymon] RRD Graph stopped graphing You probably need to recreate the rrd files based on your new interval. One easy way would be to stop the client, move the current 'testname.rrd' file to a backup location, then restart the client. You can later use rrdtool --dump (or whatever the syntax is, I don't recall) to look at the internals of the old and new files if you are interested. Regards, Tim
▸
From: xymon-bounces at xymon.com [xymon-bounces at xymon.com] on behalf of Michael Beatty [user-4aea7c115850@xymon.invalid]
Sent: Tuesday, December 18, 2012 9:42 AM
To: xymon at xymon.com
Subject: [Xymon] RRD Graph stopped graphing
I had two custom RRD graphs setup and running fine on client systems.
The script that collects the data was running every minute. After I
finished testing, I changed the frequency of the scripts on the clients
to 1 hour in the clientlaunch.cfg file. When I did, the RRD graph
stopped displaying data. It ran like this for a couple days. I changed
clientlaunch.cfg back to 1 minute... and the graphs immediately started
graphing again. Is there a minimum time frame that is needed, or
something else that must be done? What am I missing?
--
Michael Beatty
list Larry Barber
man rrdtool. Or you can just redo what you did with to create the custom graphs, but just change the time interval. Thanks, Larry Barber On Tue, Dec 18, 2012 at 2:56 PM, Michael Beatty <user-4aea7c115850@xymon.invalid>wrote:
How do I do that? Michael Beatty Sherwin-Williams IT Analyst/user-e9af76ecb6db@xymon.invalidXXX-XXX-XXXX
▸
On 12/18/2012 02:13 PM, Larry Barber wrote: When you change the frequency of data collection you need to update your rrd definitions. The rrd databases are fixed length files, so you are leaving lots of blank entries between data points. Thanks, Larry Barber On Tue, Dec 18, 2012 at 11:42 AM, Michael Beatty < user-4aea7c115850@xymon.invalid> wrote:I had two custom RRD graphs setup and running fine on client systems. The script that collects the data was running every minute. After I finished testing, I changed the frequency of the scripts on the clients to 1 hour in the clientlaunch.cfg file. When I did, the RRD graph stopped displaying data. It ran like this for a couple days. I changed clientlaunch.cfg back to 1 minute... and the graphs immediately started graphing again. Is there a minimum time frame that is needed, or something else that must be done? What am I missing? -- Michael Beatty
list Michael Beatty
It seems that any frequency above 10 minutes doesn't work. From what I understand, this has to do with the step and heartbeat of the rrd. It looks like by default the step is 300 seconds and the heartbeat is 600 seconds (10 minutes). So, as long as it doesn't go above 10 minutes it will accept data, anything past 10 minutes will be tossed which would result in me not getting graphs. So, in reading the man pages, I figured out that I can override the 600 second default by appending it to my ENV tags in xymonserver.cfg NCV_MyAlert="MyAlert:GAUGE:900" At first glance, this worked perfectly. I deleted the rrd file, restarted xymon, did a rrd dump and the XML was now showing a heartbeat of 900 (15 minutes). So, I set my script to run every 15 minutes expecting a giant rainbow to arc over my screen, but, it still doesn't work. My XML looks like this: <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE rrd SYSTEM "http://oss.oetiker.ch/rrdtool/rrdtool.dtd">; <!-- Round Robin Database Dump --><rrd> <version> 0003 </version> <step> 300 </step> <!-- Seconds --> <lastupdate> 1355940819 </lastupdate> <!-- 2012-12-19 13:13:39 EST --> <ds> <name> Shortterm </name> <type> GAUGE </type> <minimal_heartbeat> 900 </minimal_heartbeat> <min> 6.0000000000e+02 </min> <max> NaN </max> <!-- PDP Status --> <last_ds> 1.080055 </last_ds> <value> NaN </value> <unknown_sec> 219 </unknown_sec> </ds> My rrd info:
▸
Michael Beatty
Sherwin-Williams
IT Analyst/Developer
user-4aea7c115850@xymon.invalid
XXX-XXX-XXXX
On 12/18/2012 06:11 PM, Larry Barber wrote:man rrdtool. Or you can just redo what you did with to create the
custom graphs, but just change the time interval.
Thanks,
Larry Barber
On Tue, Dec 18, 2012 at 2:56 PM, Michael Beatty
<user-4aea7c115850@xymon.invalid <mailto:user-4aea7c115850@xymon.invalid>> wrote:
How do I do that?
Michael Beatty
Sherwin-Williams
IT Analyst/Developer
user-4aea7c115850@xymon.invalid <mailto:user-4aea7c115850@xymon.invalid>
XXX-XXX-XXXX <tel:XXX-XXX-XXXX>
▸
On 12/18/2012 02:13 PM, Larry Barber wrote:When you change the frequency of data collection you need to update your rrd definitions. The rrd databases are fixed length files, so you are leaving lots of blank entries between data points. Thanks, Larry Barber On Tue, Dec 18, 2012 at 11:42 AM, Michael Beatty <user-4aea7c115850@xymon.invalid <mailto:user-4aea7c115850@xymon.invalid>> wrote: I had two custom RRD graphs setup and running fine on client systems. The script that collects the data was running every minute. After I finished testing, I changed the frequency of the scripts on the clients to 1 hour in the clientlaunch.cfg file. When I did, the RRD graph stopped displaying data. It ran like this for a couple days. I changed clientlaunch.cfg back to 1 minute... and the graphs immediately started graphing again. Is there a minimum time frame that is needed, or something else that must be done? What am I missing? -- Michael Beatty
list David W David Gore
I haven't done this myself but my coworker has used it to fix graphing on one of my scripts that went from every 5 minutes to every 15 minutes: rrdtool tune your.rrd --heartbeat lambda:1200 # I don't know why he used 1200 since it seems like you would use 900 Regardless try man rrdtool to see the details. ~David
▸
From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Michael Beatty Sent: Wednesday, December 19, 2012 13:38 To: Larry Barber Cc: xymon at xymon.com Subject: Re: [Xymon] RRD Graph stopped graphing It seems that any frequency above 10 minutes doesn't work. From what I understand, this has to do with the step and heartbeat of the rrd. It looks like by default the step is 300 seconds and the heartbeat is 600 seconds (10 minutes). So, as long as it doesn't go above 10 minutes it will accept data, anything past 10 minutes will be tossed which would result in me not getting graphs. So, in reading the man pages, I figured out that I can override the 600 second default by appending it to my ENV tags in xymonserver.cfg NCV_MyAlert="MyAlert:GAUGE:900" At first glance, this worked perfectly. I deleted the rrd file, restarted xymon, did a rrd dump and the XML was now showing a heartbeat of 900 (15 minutes). So, I set my script to run every 15 minutes expecting a giant rainbow to arc over my screen, but, it still doesn't work. My XML looks like this: <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE rrd SYSTEM "http://oss.oetiker.ch/rrdtool/rrdtool.dtd">; <!-- Round Robin Database Dump --><rrd> <version> 0003 </version> <step> 300 </step> <!-- Seconds --> <lastupdate> 1355940819 </lastupdate> <!-- 2012-12-19 13:13:39 EST --> <ds> <name> Shortterm </name> <type> GAUGE </type> <minimal_heartbeat> 900 </minimal_heartbeat> <min> 6.0000000000e+02 </min> <max> NaN </max> <!-- PDP Status --> <last_ds> 1.080055 </last_ds> <value> NaN </value> <unknown_sec> 219 </unknown_sec> </ds> My rrd info: Michael Beatty Sherwin-Williams IT Analyst/Developer user-4aea7c115850@xymon.invalid XXX-XXX-XXXX On 12/18/2012 06:11 PM, Larry Barber wrote: man rrdtool. Or you can just redo what you did with to create the custom graphs, but just change the time interval. Thanks, Larry Barber On Tue, Dec 18, 2012 at 2:56 PM, Michael Beatty <user-4aea7c115850@xymon.invalid> wrote: How do I do that? Michael Beatty Sherwin-Williams IT Analyst/Developer user-4aea7c115850@xymon.invalid XXX-XXX-XXXX On 12/18/2012 02:13 PM, Larry Barber wrote: When you change the frequency of data collection you need to update your rrd definitions. The rrd databases are fixed length files, so you are leaving lots of blank entries between data points. Thanks, Larry Barber On Tue, Dec 18, 2012 at 11:42 AM, Michael Beatty <user-4aea7c115850@xymon.invalid> wrote: I had two custom RRD graphs setup and running fine on client systems. The script that collects the data was running every minute. After I finished testing, I changed the frequency of the scripts on the clients to 1 hour in the clientlaunch.cfg file. When I did, the RRD graph stopped displaying data. It ran like this for a couple days. I changed clientlaunch.cfg back to 1 minute... and the graphs immediately started graphing again. Is there a minimum time frame that is needed, or something else that must be done? What am I missing? -- Michael Beatty
list Michael Beatty
Thank you for your help, but it isn't working. I've been playing around with everything I can think of, but I cannot get the graphs to display if I collect data in frequencies greater than 10 minutes. I was briefly looking through the code and I found in do_rrd.c that the default interval for RRD is 300 seconds (5 minuets). I cannot figure out a way to change this. The closest I have come to getting this to work was to add a heartbeat value to the NCV_ setting in xymonserver.cfg file to: NCV_mytest=dataset:GAUGE:1800. Then I was able to dump the XML using rrdtools, manually update the STEP to 900, and reload the rrd. When I do so, I get a strange behavior that makes me think the 300 value that is hardcoded overrides the changes I made. When the first report comes in, the rrd file is created, the second report grabs the correct data, then the third report the last_ds value is populated with the previously correct data and the current value is set to "NaN" and that is the way it stays for all subsequent tests. Ultimately my RRD looks like:
▸
<?xml version="1.0" encoding="utf-8"?> <!DOCTYPE rrd SYSTEM "http://oss.oetiker.ch/rrdtool/rrdtool.dtd">; <!-- Round Robin Database Dump --><rrd> <version> 0003 </version>
<step> 900 </step> <!-- Seconds -->
<lastupdate> 1356032234 </lastupdate> <!-- 2012-12-20 14:37:14 EST -->
<ds>
<name> Shortterm </name>
<type> GAUGE </type>
<minimal_heartbeat> 1800 </minimal_heartbeat>
▸
<min> 6.0000000000e+02 </min>
<max> NaN </max>
<!-- PDP Status -->
<last_ds> 1.080055 </last_ds>
<value> NaN </value>
<unknown_sec> 434 </unknown_sec>
</ds>
<ds>
<name> Longterm </name>
<type> GAUGE </type>
<minimal_heartbeat> 1800 </minimal_heartbeat>
▸
<min> 6.0000000000e+02 </min>
<max> NaN </max>
<!-- PDP Status -->
<last_ds> 1.080055 </last_ds>
<value> NaN </value>
<unknown_sec> 434 </unknown_sec>
</ds>
▸
Michael Beatty
Sherwin-Williams
IT Analyst/Developer
user-4aea7c115850@xymon.invalid
XXX-XXX-XXXX
On 12/18/2012 06:11 PM, Larry Barber wrote:man rrdtool. Or you can just redo what you did with to create the custom graphs, but just change the time interval. Thanks, Larry Barber On Tue, Dec 18, 2012 at 2:56 PM, Michael Beatty <user-4aea7c115850@xymon.invalid <mailto:user-4aea7c115850@xymon.invalid>> wrote: How do I do that? Michael Beatty Sherwin-Williams IT Analyst/Developer user-4aea7c115850@xymon.invalid <mailto:user-4aea7c115850@xymon.invalid> XXX-XXX-XXXX <tel:XXX-XXX-XXXX> On 12/18/2012 02:13 PM, Larry Barber wrote:When you change the frequency of data collection you need to update your rrd definitions. The rrd databases are fixed length files, so you are leaving lots of blank entries between data points. Thanks, Larry Barber On Tue, Dec 18, 2012 at 11:42 AM, Michael Beatty <user-4aea7c115850@xymon.invalid <mailto:user-4aea7c115850@xymon.invalid>> wrote: I had two custom RRD graphs setup and running fine on client systems. The script that collects the data was running every minute. After I finished testing, I changed the frequency of the scripts on the clients to 1 hour in the clientlaunch.cfg file. When I did, the RRD graph stopped displaying data. It ran like this for a couple days. I changed clientlaunch.cfg back to 1 minute... and the graphs immediately started graphing again. Is there a minimum time frame that is needed, or something else that must be done? What am I missing? -- Michael Beatty
list Michael Beatty
I figured it out. And I believe I found a bug in the process.
Xymon uses a default step of 300 seconds with a heart beat of 600 seconds (10 mintues). If RRD doesn't get a status in under 10 minutes, it'll take it as an unknown. In order to override the defaults, you need set the values in xymonserver.cfg. This builds the the data set part of the rrd. From the documentation:
NCV_slab="inodecache:GAUGE,dentrycache:GAUGE"
I figured out early on that you can add in the heartbeat. ie. NCV_slab="inodecache:GAUGE:1800,dentrycache:GAUGE:1800"
This sets the heartbeat for those two data sets to be 1800 seconds (30 minutes). What I failed to figure out early on is that xymon doesn't like this. For some reason (I would consider this a bug) if you supply your own paramters to override the default, xymon still sticks its default value into the rrdcreate string which ultimately left it to be:
NCV_slab="inodecache:GAUGE:1800:600"
So, when the create tool executed, it took the default heartbeat value (600) and added it to the end of the create string. This set the MINIMUM value, which you can clearly see in the XML I have posted below. I failed to see this. This is where my problem lied. My values were fractions of seconds... rrd needed a value greater than 600, so it was throwing out my data.
In order to fix the issue I had to explicitly provide all of the rrdcreate parameters:
NCV_slab"iinodecache:GAUGE:1800:0:U"
Also, another thing to note to make it work, you also need to put an entry into rrddefinitions for these records because this is where the step parameter is set:
[slab]
-s 900
RRA:AVERAGE:0.5:1:576
If this isn't a bug, it should at least be documented in the "Custom Graphs" section.
Michael Beatty
▸
On 12/19/2012 10:25 PM, Gore, David W (David) wrote:I haven't done this myself but my coworker has used it to fix graphing on one of my scripts that went from every 5 minutes to every 15 minutes: rrdtool tune your.rrd --heartbeat lambda:1200 # I don't know why he used 1200 since it seems like you would use 900 Regardless try man rrdtool to see the details. ~David From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Michael Beatty Sent: Wednesday, December 19, 2012 13:38 To: Larry Barber Cc: xymon at xymon.com Subject: Re: [Xymon] RRD Graph stopped graphing It seems that any frequency above 10 minutes doesn't work. From what I understand, this has to do with the step and heartbeat of the rrd. It looks like by default the step is 300 seconds and the heartbeat is 600 seconds (10 minutes). So, as long as it doesn't go above 10 minutes it will accept data, anything past 10 minutes will be tossed which would result in me not getting graphs. So, in reading the man pages, I figured out that I can override the 600 second default by appending it to my ENV tags in xymonserver.cfg NCV_MyAlert="MyAlert:GAUGE:900" At first glance, this worked perfectly. I deleted the rrd file, restarted xymon, did a rrd dump and the XML was now showing a heartbeat of 900 (15 minutes). So, I set my script to run every 15 minutes expecting a giant rainbow to arc over my screen, but, it still doesn't work. My XML looks like this: <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE rrd SYSTEM "http://oss.oetiker.ch/rrdtool/rrdtool.dtd">; <!-- Round Robin Database Dump --><rrd> <version> 0003 </version> <step> 300 </step> <!-- Seconds --> <lastupdate> 1355940819 </lastupdate> <!-- 2012-12-19 13:13:39 EST --> <ds> <name> Shortterm </name> <type> GAUGE </type> <minimal_heartbeat> 900 </minimal_heartbeat> <min> 6.0000000000e+02 </min> <max> NaN </max> <!-- PDP Status --> <last_ds> 1.080055 </last_ds> <value> NaN </value> <unknown_sec> 219 </unknown_sec> </ds> My rrd info: Michael Beatty On 12/18/2012 06:11 PM, Larry Barber wrote: man rrdtool. Or you can just redo what you did with to create the custom graphs, but just change the time interval. Thanks, Larry Barber On Tue, Dec 18, 2012 at 2:56 PM, Michael Beatty <user-4aea7c115850@xymon.invalid> wrote: How do I do that? When you change the frequency of data collection you need to update your rrd definitions. The rrd databases are fixed length files, so you are leaving lots of blank entries between data points. Thanks, Larry Barber On Tue, Dec 18, 2012 at 11:42 AM, Michael Beatty <user-4aea7c115850@xymon.invalid> wrote: I had two custom RRD graphs setup and running fine on client systems. The script that collects the data was running every minute. After I finished testing, I changed the frequency of the scripts on the clients to 1 hour in the clientlaunch.cfg file. When I did, the RRD graph stopped displaying data. It ran like this for a couple days. I changed clientlaunch.cfg back to 1 minute... and the graphs immediately started graphing again. Is there a minimum time frame that is needed, or something else that must be done? What am I missing?