Graphs are missing data, but it's there!

13 messages in this thread

list Martin Ward · Thu, 24 Jul 2008 12:43:19 +0100 ·

All,

I have a problem with one machine where its data is not being shown in
the graphs even though the data exists.

The machine in question's Hobbit client sends five pieces of numeric
data (email queues) and these are displayed on the web page for this
service:

====
Thursday July 24 11:29:11 UTC 2008

ActiveStatus: green
<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>; ActiveQueue: 106
ActiveTrend: tendency rising with -60 mails.
BounceStatus: green
<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>; BounceQueue: 58
BounceTrend: tendency falling with 3 mails.
DeferStatus: red <http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif>; DeferQueue: 150464
DeferTrend: tendency falling with 95 mails.
CorruptStatus: green
<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>; CorruptQueue: 0
CorruptTrend: amount equal to last measure.
IncomingStatus: red <http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif>; IncomingQueue: 247049
IncomingTrend: amount equal to last measure.

Deferred Queue is too high but is decreasing already.

====
These numbers change over time and the values are accurate.

However, the graph that is displayed below this data is blank. I have
historic data, the files exist, and what is more I have other machines
that are configured identically to this one where the data IS graphed
correctly.

Hobbit graphs are a bit of a black hole to me, can anyone suggest where
I might look?

|\/|artin

*************************************************************************************
The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way.
The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing user-61c7f445d564@xymon.invalid and delete the message and any attachments without retaining any copies.
Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses.
No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party.
Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.

list James Roberts · Thu, 24 Jul 2008 12:47:16 +0100 ·

you need to touch all the rdd.

▸ quoted from Martin Ward

From: Ward, Martin [mailto:user-2d33a6eb6a05@xymon.invalid] Sent: 24 July 2008 12:43
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Graphs are missing data, but it's there!

All, 
I have a problem with one machine where its data is not being shown in
the graphs even though the data exists. 
The machine in question's Hobbit client sends five pieces of numeric
data (email queues) and these are displayed on the web page for this
service:

==== Thursday July 24 11:29:11 UTC 2008 
ActiveStatus:  green
<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>; ActiveQueue: 106
ActiveTrend: tendency rising with -60 mails.
BounceStatus:  green
<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>; BounceQueue: 58
BounceTrend: tendency falling with 3 mails.
DeferStatus:  red <http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif>; DeferQueue: 150464
DeferTrend: tendency falling with 95 mails.
CorruptStatus:  green
<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>; CorruptQueue: 0
CorruptTrend: amount equal to last measure.
IncomingStatus:  red <http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif>; IncomingQueue: 247049
IncomingTrend: amount equal to last measure.

 Deferred Queue is too high but is decreasing already. 
==== These numbers change over time and the values are accurate. 
However, the graph that is displayed below this data is blank. I have
historic data, the files exist, and what is more I have other machines
that are configured identically to this one where the data IS graphed
correctly.

Hobbit graphs are a bit of a black hole to me, can anyone suggest where
I might look? 
|\/|artin 

************************************************************************
*************
The message is intended for the named addressee only and may not be
disclosed to or used by anyone else, nor may it be copied in any way. 
The contents of this message and its attachments are confidential and
may also be subject to legal privilege. If you are not the named
addressee and/or have received this message in error, please advise us
by e-mailing user-61c7f445d564@xymon.invalid and delete the message and any
attachments without retaining any copies. 
Internet communications are not secure and COLT does not accept
responsibility for this message, its contents nor responsibility for any
viruses. 
No contracts can be created or varied on behalf of COLT
Telecommunications, its subsidiaries or affiliates ("COLT") and any
other party by email Communications unless expressly agreed in writing
with such other party. 
Please note that incoming emails will be automatically scanned to
eliminate potential viruses and unsolicited promotional emails. For more
information refer to www.colt.net or contact us on +44(0)20 7390 3900.

list Martin Ward · Thu, 24 Jul 2008 14:20:42 +0100 ·

Thanks for the suggestion but that didn't work (I guess you meant rrd).
 
Any other ideas?

▸ quoted from James Roberts

|\/|

-----Original Message-----
From: Roberts, James [mailto:user-d7df29bbad83@xymon.invalid]
Sent: 24 July 2008 12:47
To: user-ae9b8668bcde@xymon.invalid
Subject: RE: [hobbit] Graphs are missing data, but it's there!

you need to touch all the rdd.

From: Ward, Martin [mailto:user-2d33a6eb6a05@xymon.invalid]
Sent: 24 July 2008 12:43
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Graphs are missing data, but it's there!

All,

I have a problem with one machine where its data is not being
shown in the graphs even though the data exists.

The machine in question's Hobbit client sends five pieces of
numeric data (email queues) and these are displayed on the web page for
this service:

====
Thursday July 24 11:29:11 UTC 2008

ActiveStatus: green
<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>;
ActiveQueue: 106
ActiveTrend: tendency rising with -60 mails.
BounceStatus: green
<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>;
BounceQueue: 58
BounceTrend: tendency falling with 3 mails.
DeferStatus: red
<http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif>;
DeferQueue: 150464
DeferTrend: tendency falling with 95 mails.
CorruptStatus: green
<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>;
CorruptQueue: 0
CorruptTrend: amount equal to last measure.
IncomingStatus: red
<http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif>;
IncomingQueue: 247049
IncomingTrend: amount equal to last measure.

Deferred Queue is too high but is decreasing already.

====
These numbers change over time and the values are accurate.

However, the graph that is displayed below this data is blank. I
have historic data, the files exist, and what is more I have other
machines that are configured identically to this one where the data IS
graphed correctly.

Hobbit graphs are a bit of a black hole to me, can anyone
suggest where I might look?

|\/|artin

The contents of this message and its attachments are
confidential and may also be subject to legal privilege. If you are not
the named addressee and/or have received this message in error, please
advise us by e-mailing user-61c7f445d564@xymon.invalid and delete the message and any
attachments without retaining any copies.

Internet communications are not secure and COLT does not accept
responsibility for this message, its contents nor responsibility for any
viruses.

No contracts can be created or varied on behalf of COLT
Telecommunications, its subsidiaries or affiliates ("COLT") and any
other party by email Communications unless expressly agreed in writing
with such other party.

Please note that incoming emails will be automatically scanned
to eliminate potential viruses and unsolicited promotional emails. For
more information refer to www.colt.net or contact us on +44(0)20 7390
3900.

The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing user-61c7f445d564@xymon.invalid and delete the message and any attachments without retaining any copies.

Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses.

No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party.

Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.

list Greg L Hubbard · Thu, 24 Jul 2008 08:33:48 -0500 ·

You know the data exists because you used the rrd dump tool to display
it?
 
Is the graph simply not shown at all, or is there a "hole" in the Web
page where it normally would go?  ("show page source" might have a
clue).
 
Some ideas/shots in the dark:
 
a) check the logs
 
b) meticulously compare a "working" system to the non-working system,
and make sure that they really are identical.
 
c) look at the trends page for this host to see if the graph is okay
there...
 
Etc.  I am sure you know the drill -- a big pain to look under every
rock, but it has to be done...
 
GLH

▸ quoted from Martin Ward



	From: Ward, Martin [mailto:user-2d33a6eb6a05@xymon.invalid] 
	Sent: Thursday, July 24, 2008 8:21 AM
	To: user-ae9b8668bcde@xymon.invalid
	Subject: RE: [hobbit] Graphs are missing data, but it's there!
	
	
	Thanks for the suggestion but that didn't work (I guess you
meant rrd).
	 
	Any other ideas?
	 
	|\/|

		-----Original Message-----
		From: Roberts, James [mailto:user-d7df29bbad83@xymon.invalid]

		Sent: 24 July 2008 12:47
		To: user-ae9b8668bcde@xymon.invalid
		Subject: RE: [hobbit] Graphs are missing data, but it's
there!
		
		
		you need to touch all the rdd.
		
		
		From: Ward, Martin [mailto:user-2d33a6eb6a05@xymon.invalid] 
		Sent: 24 July 2008 12:43
		To: user-ae9b8668bcde@xymon.invalid
		Subject: [hobbit] Graphs are missing data, but it's
there!
		
		
		All, 

		I have a problem with one machine where its data is not
being shown in the graphs even though the data exists. 

		The machine in question's Hobbit client sends five
pieces of numeric data (email queues) and these are displayed on the web
page for this service:

		==== 
		Thursday July 24 11:29:11 UTC 2008 
		
		ActiveStatus:  green
<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>; 
		ActiveQueue: 106
		ActiveTrend: tendency rising with -60 mails.
		BounceStatus:  green
<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>; 
		BounceQueue: 58
		BounceTrend: tendency falling with 3 mails.
		DeferStatus:  red
<http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif>; 
		DeferQueue: 150464
		DeferTrend: tendency falling with 95 mails.
		CorruptStatus:  green
<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>; 
		CorruptQueue: 0
		CorruptTrend: amount equal to last measure.
		IncomingStatus:  red
<http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif>; 
		IncomingQueue: 247049
		IncomingTrend: amount equal to last measure.
		
		 Deferred Queue is too high but is decreasing already. 

		==== 
		These numbers change over time and the values are
accurate. 

		However, the graph that is displayed below this data is
blank. I have historic data, the files exist, and what is more I have
other machines that are configured identically to this one where the
data IS graphed correctly.

		Hobbit graphs are a bit of a black hole to me, can
anyone suggest where I might look? 

		|\/|artin 

		
************************************************************************
*************
		The message is intended for the named addressee only and
may not be disclosed to or used by anyone else, nor may it be copied in
any way. 
		
		The contents of this message and its attachments are
confidential and may also be subject to legal privilege. If you are not
the named addressee and/or have received this message in error, please
advise us by e-mailing user-61c7f445d564@xymon.invalid and delete the message and any
attachments without retaining any copies. 
		
		Internet communications are not secure and COLT does not
accept responsibility for this message, its contents nor responsibility
for any viruses. 
		
		No contracts can be created or varied on behalf of COLT
Telecommunications, its subsidiaries or affiliates ("COLT") and any
other party by email Communications unless expressly agreed in writing
with such other party. 
		
		Please note that incoming emails will be automatically
scanned to eliminate potential viruses and unsolicited promotional
emails. For more information refer to www.colt.net or contact us on
+44(0)20 7390 3900.

list Phil Wild · Thu, 24 Jul 2008 23:30:44 +0800 ·

Are you saying that you run the same tests on multiple hosts and only one
host in not showing data? Does this mean they all share the same NCV
configuration in hobberserver.cfg and the same graph definition in
hobbitgraph.cfg?

If this is correct, then it really points to something not getting into the
rrd file. As previously suggested, rrd dump is your best bet at finding the
problem here. What if you remove the rrd file and let hobbit create a new
one, does that help?

Cheers

Phil

2008/7/24 Hubbard, Greg L <user-d970b5e56ec9@xymon.invalid>:

▸ quoted from Greg L Hubbard

 You know the data exists because you used the rrd dump tool to display
it?

Is the graph simply not shown at all, or is there a "hole" in the Web page
where it normally would go?  ("show page source" might have a clue).

Some ideas/shots in the dark:

a) check the logs

b) meticulously compare a "working" system to the non-working system, and
make sure that they really are identical.


c) look at the trends page for this host to see if the graph is okay there...

Etc.  I am sure you know the drill -- a big pain to look under every rock,
but it has to be done...

GLH

*From:* Ward, Martin [mailto:user-2d33a6eb6a05@xymon.invalid]
*Sent:* Thursday, July 24, 2008 8:21 AM

*To:* user-ae9b8668bcde@xymon.invalid
*Subject:* RE: [hobbit] Graphs are missing data, but it's there!

 Thanks for the suggestion but that didn't work (I guess you meant rrd).

Any other ideas?

|\/|

 -----Original Message-----
*From:* Roberts, James [mailto:user-d7df29bbad83@xymon.invalid]
*Sent:* 24 July 2008 12:47
*To:* user-ae9b8668bcde@xymon.invalid
*Subject:* RE: [hobbit] Graphs are missing data, but it's there!

you need to touch all the rdd.

*From:* Ward, Martin [mailto:user-2d33a6eb6a05@xymon.invalid]
*Sent:* 24 July 2008 12:43
*To:* user-ae9b8668bcde@xymon.invalid
*Subject:* [hobbit] Graphs are missing data, but it's there!

 All,

I have a problem with one machine where its data is not being shown in the
graphs even though the data exists.

The machine in question's Hobbit client sends five pieces of numeric data
(email queues) and these are displayed on the web page for this service:

====
*Thursday July 24 11:29:11 UTC 2008*


ActiveStatus:  ***green*<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>;
ActiveQueue: 106
ActiveTrend: tendency* rising* with* -60* mails.
BounceStatus:  ***green*<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>;

▸ quoted from Greg L Hubbard

BounceQueue: 58
BounceTrend: tendency* falling* with* 3* mails.
DeferStatus:  ***red* <http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif>;
DeferQueue: 150464
DeferTrend: tendency* falling* with* 95* mails.


CorruptStatus:  ***green*<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>;
CorruptQueue: 0
CorruptTrend: amount equal to last measure.
IncomingStatus:  ***red*<http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif>;

▸ quoted from Greg L Hubbard

IncomingQueue: 247049
IncomingTrend: amount equal to last measure.

 Deferred Queue is too high but is decreasing already.

====
These numbers change over time and the values are accurate.

However, the graph that is displayed below this data is blank. I have
historic data, the files exist, and what is more I have other machines that
are configured identically to this one where the data IS graphed correctly.

Hobbit graphs are a bit of a black hole to me, can anyone suggest where I
might look?

|\/|artin


*************************************************************************************
The message is intended for the named addressee only and may not be
disclosed to or used by anyone else, nor may it be copied in any way.

The contents of this message and its attachments are confidential and may
also be subject to legal privilege. If you are not the named addressee
and/or have received this message in error, please advise us by e-mailing
user-61c7f445d564@xymon.invalid and delete the message and any attachments without
retaining any copies.

Internet communications are not secure and COLT does not accept
responsibility for this message, its contents nor responsibility for any
viruses.

No contracts can be created or varied on behalf of COLT Telecommunications,
its subsidiaries or affiliates ("COLT") and any other party by email
Communications unless expressly agreed in writing with such other party.

Please note that incoming emails will be automatically scanned to eliminate
potential viruses and unsolicited promotional emails. For more information
refer to www.colt.net or contact us on +44(0)20 7390 3900.

--


Tel: XXXX XXX XXX
Fax: XXXX XXX XXX
email: user-e365c1418192@xymon.invalid

list Martin Ward · Thu, 24 Jul 2008 17:02:37 +0100 ·

▸ quoted from Phil Wild

You know the data exists because you used the rrd dump tool to display
it?

Um, no, sorry I didn't explain myself very well. I know the data is
being passed back to the Hobbit server because it is displayed in the
web page (see example in original email below). The issue it that the
graph that is displayed at the bottom of the page is empty.
 
I know that data for the other identical servers is being passed for the
same reason, but on those pages the graphed data IS visible.
 
I have compared the page sources and they look identical, which is why I
think it's the data.
 
Looking at the data files I am getting somewhat confused (or am typing
commands wrong), please confirm my thoughts on this:
 
1. When numeric data is transferred from the Hobbit client to the server
it is stored in an rrd file thanks to the TEST2RRD config option, mine
looks like:
TEST2RRD="cpu=la,disk,inode,qtree,memory,postfixqueue=ncv,postfixdeliver
ies=ncv".
2. The RRD file in 1 takes its name from the test that the client
specifies when it transfers its data.
3. The directory path for where the data is stored is specified in the
hobbitd channel=data using the --rrddir= parameter (/opt/hobbit/data/rrd
for me).
4. The actual path of the RRD data file is the directory specified in 3,
plus the fully qualified host name (mc25.lon.dcn.colt.net) plus the RRD
filename specified in 2, giving me
"/opt/hobbit/data/rrd/mc25.lon.dcn.colt.net/postfixqueue.rrd".
5. When I run "rrdtool --dump" on this file I should get real data, and
it is from this data that ALL the graphs are generated, i.e. this file
can hold data for up to 576 days.
 
Now this is where things go wrong:
 
For both the machine that is failing and for any machine that is working
I get nothing when I run:
$ rrdtool --dump
/opt/hobbit/data/rrd/mc25.lon.dcn.colt.net/postfixqueue.rrd
or
$ rrdtool --dump /opt/hobbit/data/rrd/mc25.lon.dcn.colt.net/memory.rrd
 
yet I can view graphs for the memory usage on this machine going back
for as long as I have had this server up, so I now have no idea what
these RRD files are for.
 
The files are being updated regularly and the have a quantity of data in
the (18KB or so).
 
So, after all of this I am confused about how and where the data is
stored although I can say that the trends show that data was
successfully stored for about two months after the service was first
started, but then it stopped. Being that that was back in February I
have no idea what changes might have happened to the servers.
 
8-(
 
|\/|artin

▸ quoted from Phil Wild

-----Original Message-----
From: Hubbard, Greg L [mailto:user-d970b5e56ec9@xymon.invalid]
Sent: 24 July 2008 14:34
To: user-ae9b8668bcde@xymon.invalid
Subject: RE: [hobbit] Graphs are missing data, but it's there!

You know the data exists because you used the rrd dump tool to
display it?

Is the graph simply not shown at all, or is there a "hole" in
the Web page where it normally would go? ("show page source" might have
a clue).

Some ideas/shots in the dark:

a) check the logs

b) meticulously compare a "working" system to the non-working
system, and make sure that they really are identical.

c) look at the trends page for this host to see if the graph is
okay there...

Etc. I am sure you know the drill -- a big pain to look under
every rock, but it has to be done...

GLH

From: Ward, Martin [mailto:user-2d33a6eb6a05@xymon.invalid]
Sent: Thursday, July 24, 2008 8:21 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: RE: [hobbit] Graphs are missing data, but it's
there!

Thanks for the suggestion but that didn't work (I guess
you meant rrd).

Any other ideas?

|\/|

-----Original Message-----
From: Roberts, James
[mailto:user-d7df29bbad83@xymon.invalid]
Sent: 24 July 2008 12:47
To: user-ae9b8668bcde@xymon.invalid
Subject: RE: [hobbit] Graphs are missing data,
but it's there!

you need to touch all the rdd.

From: Ward, Martin [mailto:user-2d33a6eb6a05@xymon.invalid]

Sent: 24 July 2008 12:43
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Graphs are missing data, but
it's there!

All,

I have a problem with one machine where its data
is not being shown in the graphs even though the data exists.

The machine in question's Hobbit client sends
five pieces of numeric data (email queues) and these are displayed on
the web page for this service:

====
Thursday July 24 11:29:11 UTC 2008

ActiveStatus: green
<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>;
ActiveQueue: 106
ActiveTrend: tendency rising with -60 mails.
BounceStatus: green
<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>;
BounceQueue: 58
BounceTrend: tendency falling with 3 mails.
DeferStatus: red
<http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif>;
DeferQueue: 150464
DeferTrend: tendency falling with 95 mails.
CorruptStatus: green
<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>;
CorruptQueue: 0
CorruptTrend: amount equal to last measure.
IncomingStatus: red
<http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif>;
IncomingQueue: 247049
IncomingTrend: amount equal to last measure.

Deferred Queue is too high but is decreasing
already.

====
These numbers change over time and the values
are accurate.

However, the graph that is displayed below this
data is blank. I have historic data, the files exist, and what is more I
have other machines that are configured identically to this one where
the data IS graphed correctly.

Hobbit graphs are a bit of a black hole to me,
can anyone suggest where I might look?

|\/|artin

The contents of this message and its attachments
are confidential and may also be subject to legal privilege. If you are
not the named addressee and/or have received this message in error,
please advise us by e-mailing user-61c7f445d564@xymon.invalid and delete the message
and any attachments without retaining any copies.

Internet communications are not secure and COLT
does not accept responsibility for this message, its contents nor
responsibility for any viruses.

No contracts can be created or varied on behalf
of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and
any other party by email Communications unless expressly agreed in
writing with such other party.

Please note that incoming emails will be
automatically scanned to eliminate potential viruses and unsolicited
promotional emails. For more information refer to www.colt.net or
contact us on +44(0)20 7390 3900.

The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing user-61c7f445d564@xymon.invalid and delete the message and any attachments without retaining any copies.

Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses.

No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party.

Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.

list Michael Nemeth · Thu, 24 Jul 2008 12:03:34 -0400 ·

And the reverse , having something in the graph def thats not in the rrd 
.  Check the rdd-status.log .
And rrdtool dump, check that the <name> xxxxx </name> s are in the  
graph def  and  nothing  else.

▸ quoted from Phil Wild

Phil Wild wrote:

Are you saying that you run the same tests on multiple hosts and only 
one host in not showing data? Does this mean they all share the same 
NCV configuration in hobberserver.cfg and the same graph definition in 
hobbitgraph.cfg?

If this is correct, then it really points to something not getting 
into the rrd file. As previously suggested, rrd dump is your best bet 
at finding the problem here. What if you remove the rrd file and let 
hobbit create a new one, does that help?

Cheers

Phil

2008/7/24 Hubbard, Greg L <user-d970b5e56ec9@xymon.invalid


<mailto:user-d970b5e56ec9@xymon.invalid>>:

▸ quoted from Martin Ward


    You know the data exists because you used the rrd dump tool
    to display it?
     
    Is the graph simply not shown at all, or is there a "hole" in the
    Web page where it normally would go?  ("show page source" might
    have a clue).
     
    Some ideas/shots in the dark:
     
    a) check the logs
     
    b) meticulously compare a "working" system to the non-working
    system, and make sure that they really are identical.
     
    c) look at the trends page for this host to see if the graph is okay there...
     
    Etc.  I am sure you know the drill -- a big pain to look under
    every rock, but it has to be done...
     
    GLH

        *From:* Ward, Martin [mailto:user-2d33a6eb6a05@xymon.invalid
        <mailto:user-2d33a6eb6a05@xymon.invalid>]
        *Sent:* Thursday, July 24, 2008 8:21 AM


        *To:* user-ae9b8668bcde@xymon.invalid <mailto:user-ae9b8668bcde@xymon.invalid>

▸ quoted from Martin Ward

        *Subject:* RE: [hobbit] Graphs are missing data, but it's there!

        Thanks for the suggestion but that didn't work (I guess you
        meant rrd).
         
        Any other ideas?
         
        |\/|

            -----Original Message-----
            *From:* Roberts, James [mailto:user-d7df29bbad83@xymon.invalid
            <mailto:user-d7df29bbad83@xymon.invalid>]
            *Sent:* 24 July 2008 12:47


            *To:* user-ae9b8668bcde@xymon.invalid <mailto:user-ae9b8668bcde@xymon.invalid>

▸ quoted from Martin Ward

            *Subject:* RE: [hobbit] Graphs are missing data, but it's
            there!

            you need to touch all the rdd.

            *From:* Ward, Martin [mailto:user-2d33a6eb6a05@xymon.invalid
            <mailto:user-2d33a6eb6a05@xymon.invalid>]
            *Sent:* 24 July 2008 12:43


            *To:* user-ae9b8668bcde@xymon.invalid <mailto:user-ae9b8668bcde@xymon.invalid>

▸ quoted from Martin Ward

            *Subject:* [hobbit] Graphs are missing data, but it's there!

            All,

            I have a problem with one machine where its data is not
            being shown in the graphs even though the data exists.

            The machine in question's Hobbit client sends five pieces
            of numeric data (email queues) and these are displayed on
            the web page for this service:

            ====
            *Thursday July 24 11:29:11 UTC 2008*

            ActiveStatus:  _green_
            <http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>;
            ActiveQueue: 106
            ActiveTrend: tendency* rising* with* -60* mails.
            BounceStatus:  _green_
            <http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>;
            BounceQueue: 58
            BounceTrend: tendency* falling* with* 3* mails.
            DeferStatus:  _red_
            <http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif>;
            DeferQueue: 150464
            DeferTrend: tendency* falling* with* 95* mails.
            CorruptStatus:  _green_
            <http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>;
            CorruptQueue: 0
            CorruptTrend: amount equal to last measure.
            IncomingStatus:  _red_
            <http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif>;
            IncomingQueue: 247049
            IncomingTrend: amount equal to last measure.

             Deferred Queue is too high but is decreasing already.

            ====
            These numbers change over time and the values are accurate.

            However, the graph that is displayed below this data is
            blank. I have historic data, the files exist, and what is
            more I have other machines that are configured identically
            to this one where the data IS graphed correctly.

            Hobbit graphs are a bit of a black hole to me, can anyone
            suggest where I might look?

            |\/|artin


            *************************************************************************************
            The message is intended for the named addressee only and
            may not be disclosed to or used by anyone else, nor may it
            be copied in any way.

            The contents of this message and its attachments are
            confidential and may also be subject to legal privilege.
            If you are not the named addressee and/or have received
            this message in error, please advise us by e-mailing


            user-61c7f445d564@xymon.invalid <mailto:user-61c7f445d564@xymon.invalid> and delete

▸ quoted from Martin Ward

            the message and any attachments without retaining any copies.

            Internet communications are not secure and COLT does not
            accept responsibility for this message, its contents nor
            responsibility for any viruses.

            No contracts can be created or varied on behalf of COLT
            Telecommunications, its subsidiaries or affiliates
            ("COLT") and any other party by email Communications
            unless expressly agreed in writing with such other party.

            Please note that incoming emails will be automatically
            scanned to eliminate potential viruses and unsolicited
            promotional emails. For more information refer to


            www.colt.net <http://www.colt.net>; or contact us on
            +44(0)20 7390 3900.


-- 
Tel: XXXX XXX XXX
Fax: XXXX XXX XXX
email: user-e365c1418192@xymon.invalid <http://gmail.com>;

list Martin Ward · Thu, 24 Jul 2008 17:25:46 +0100 ·

▸ quoted from Michael Nemeth

Are you saying that you run the same tests on multiple hosts and only
one host in not showing data?

Yes.

▸ quoted from Michael Nemeth

Does this mean they all share the same NCV configuration in
hobberserver.cfg and the same graph definition in hobbitgraph.cfg?

Yes.

▸ quoted from Michael Nemeth

What if you remove the rrd file and let hobbit create a new one, does
that help?

I did this and as you'd expect initially the web page showed no graph
although it did show data (stored from the previous run I presume).
 
After an interval the file appeared again but running "rrdtool dump" on
it STILL failed to produce any data.
 
I'm starting to wonder about the versions of RRD, but they ought to be
data-compatible; I'm using rrdtool v1.2.15.
 
The histlogs show no errors, the hist/mc25,... data file contains valid
data. I DO get a few RRD errors like this:
 
rrd-status.log:2008-07-21 09:46:19 RRD error updating
/opt/hobbit/data/rrd/mc25.lon.server.colt.net/tcp.smtp.rrd from
10.44.107.48: illegal attempt to update using time 1216633579 when last
update time is 1216633579 (minimum one second step)

which make it look like Hobbit is actually updating the RRD file... I
just can't get any data out!

▸ quoted from Phil Wild

|\/|

-----Original Message-----
From: Phil Wild [mailto:user-e365c1418192@xymon.invalid]
Sent: 24 July 2008 16:31
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Graphs are missing data, but it's there!

Are you saying that you run the same tests on multiple hosts and
only one host in not showing data? Does this mean they all share the
same NCV configuration in hobberserver.cfg and the same graph definition
in hobbitgraph.cfg?

If this is correct, then it really points to something not
getting into the rrd file. As previously suggested, rrd dump is your
best bet at finding the problem here. What if you remove the rrd file
and let hobbit create a new one, does that help?

Cheers

Phil

2008/7/24 Hubbard, Greg L <user-d970b5e56ec9@xymon.invalid>:

You know the data exists because you used the rrd dump
tool to display it?

Is the graph simply not shown at all, or is there a
"hole" in the Web page where it normally would go? ("show page source"
might have a clue).

Some ideas/shots in the dark:

a) check the logs

b) meticulously compare a "working" system to the
non-working system, and make sure that they really are identical.

c) look at the trends page for this host to see if the
graph is okay there...

Etc. I am sure you know the drill -- a big pain to look
under every rock, but it has to be done...

GLH

From: Ward, Martin [mailto:user-2d33a6eb6a05@xymon.invalid]

Sent: Thursday, July 24, 2008 8:21 AM

To: user-ae9b8668bcde@xymon.invalid
Subject: RE: [hobbit] Graphs are missing data,
but it's there!

Thanks for the suggestion but that didn't work
(I guess you meant rrd).

Any other ideas?

|\/|

-----Original Message-----
From: Roberts, James
[mailto:user-d7df29bbad83@xymon.invalid]
Sent: 24 July 2008 12:47
To: user-ae9b8668bcde@xymon.invalid
Subject: RE: [hobbit] Graphs are missing
data, but it's there!

you need to touch all the rdd.

From: Ward, Martin
[mailto:user-2d33a6eb6a05@xymon.invalid]
Sent: 24 July 2008 12:43
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Graphs are missing
data, but it's there!

All,

I have a problem with one machine where
its data is not being shown in the graphs even though the data exists.

The machine in question's Hobbit client
sends five pieces of numeric data (email queues) and these are displayed
on the web page for this service:

====
Thursday July 24 11:29:11 UTC 2008

ActiveStatus: green
<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>;
ActiveQueue: 106
ActiveTrend: tendency rising with -60
mails.
BounceStatus: green
<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>;
BounceQueue: 58
BounceTrend: tendency falling with 3
mails.
DeferStatus: red
<http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif>;
DeferQueue: 150464
DeferTrend: tendency falling with 95
mails.
CorruptStatus: green
<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>;
CorruptQueue: 0
CorruptTrend: amount equal to last
measure.
IncomingStatus: red
<http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif>;
IncomingQueue: 247049
IncomingTrend: amount equal to last
measure.

Deferred Queue is too high but is
decreasing already.

====
These numbers change over time and the
values are accurate.

However, the graph that is displayed
below this data is blank. I have historic data, the files exist, and
what is more I have other machines that are configured identically to
this one where the data IS graphed correctly.

Hobbit graphs are a bit of a black hole
to me, can anyone suggest where I might look?

|\/|artin

The contents of this message and its
attachments are confidential and may also be subject to legal privilege.
If you are not the named addressee and/or have received this message in
error, please advise us by e-mailing user-61c7f445d564@xymon.invalid and delete the
message and any attachments without retaining any copies.

Internet communications are not secure
and COLT does not accept responsibility for this message, its contents
nor responsibility for any viruses.

No contracts can be created or varied on
behalf of COLT Telecommunications, its subsidiaries or affiliates
("COLT") and any other party by email Communications unless expressly
agreed in writing with such other party.

Please note that incoming emails will be
automatically scanned to eliminate potential viruses and unsolicited
promotional emails. For more information refer to www.colt.net or
contact us on +44(0)20 7390 3900.

--
Tel: XXXX XXX XXX
Fax: XXXX XXX XXX
email: user-e365c1418192@xymon.invalid

The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing user-61c7f445d564@xymon.invalid and delete the message and any attachments without retaining any copies.

Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses.

No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party.

Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.

list Phil Wild · Fri, 25 Jul 2008 00:41:55 +0800 ·

The rrd version should be okay, after all it is graphing data from other
hosts with no problem.

It would appear that you ncv and graph configurations are correct as you say
they are working for other hosts. This would indicate it is a problem with
this host's configuration, so where to look...

Just out of interest, can you take an rrd file this test from a host that
works, and copy it into the .../data/rrd/hostname directory of the host that
does not?

I would expect after doing this that you will have a graph for this host.
Can you confirm this works? After doing this and leaving it for 10 minutes,
do you see any new data in the graph?

Can you dump the data from this rrd file?

2008/7/25 Ward, Martin <user-2d33a6eb6a05@xymon.invalid>:

▸ quoted from Martin Ward

Are you saying that you run the same tests on multiple hosts and only
one host in not showing data?

Yes.

Does this mean they all share the same NCV configuration in
hobberserver.cfg and the same graph definition in hobbitgraph.cfg?

Yes.

What if you remove the rrd file and let hobbit create a new one, does
that help?

I did this and as you'd expect initially the web page showed no graph
although it did show data (stored from the previous run I presume).

After an interval the file appeared again but running "rrdtool dump" on it
STILL failed to produce any data.

I'm starting to wonder about the versions of RRD, but they ought to be
data-compatible; I'm using rrdtool v1.2.15.

The histlogs show no errors, the hist/mc25,... data file contains valid
data. I DO get a few RRD errors like this:


rrd-status.log:2008-07-21 09:46:19 RRD error updating /opt/hobbit/data/rrd/
mc25.lon.server.colt.net/tcp.smtp.rrd from 10.44.107.48: illegal attempt

▸ quoted from Martin Ward

to update using time 1216633579 when last update time is 1216633579 (minimum
one second step)
which make it look like Hobbit is actually updating the RRD file... I just
can't get any data out!

|\/|


 -----Original Message-----
*From:* Phil Wild [mailto:user-e365c1418192@xymon.invalid]
*Sent:* 24 July 2008 16:31
*To:* user-ae9b8668bcde@xymon.invalid
*Subject:* Re: [hobbit] Graphs are missing data, but it's there!

Are you saying that you run the same tests on multiple hosts and only one
host in not showing data? Does this mean they all share the same NCV
configuration in hobberserver.cfg and the same graph definition in
hobbitgraph.cfg?

If this is correct, then it really points to something not getting into the
rrd file. As previously suggested, rrd dump is your best bet at finding the
problem here. What if you remove the rrd file and let hobbit create a new
one, does that help?

Cheers

Phil

2008/7/24 Hubbard, Greg L <user-d970b5e56ec9@xymon.invalid>:

 You know the data exists because you used the rrd dump tool to display
it?

Is the graph simply not shown at all, or is there a "hole" in the Web page
where it normally would go?  ("show page source" might have a clue).

Some ideas/shots in the dark:

a) check the logs

b) meticulously compare a "working" system to the non-working system, and
make sure that they really are identical.


c) look at the trends page for this host to see if the graph is okay there...

Etc.  I am sure you know the drill -- a big pain to look under every
rock, but it has to be done...

GLH

 *From:* Ward, Martin [mailto:user-2d33a6eb6a05@xymon.invalid]
*Sent:* Thursday, July 24, 2008 8:21 AM

*To:* user-ae9b8668bcde@xymon.invalid
*Subject:* RE: [hobbit] Graphs are missing data, but it's there!

  Thanks for the suggestion but that didn't work (I guess you meant rrd).

Any other ideas?

|\/|

 -----Original Message-----
*From:* Roberts, James [mailto:user-d7df29bbad83@xymon.invalid]
*Sent:* 24 July 2008 12:47
*To:* user-ae9b8668bcde@xymon.invalid
*Subject:* RE: [hobbit] Graphs are missing data, but it's there!

you need to touch all the rdd.

*From:* Ward, Martin [mailto:user-2d33a6eb6a05@xymon.invalid]
*Sent:* 24 July 2008 12:43
*To:* user-ae9b8668bcde@xymon.invalid
*Subject:* [hobbit] Graphs are missing data, but it's there!

 All,

I have a problem with one machine where its data is not being shown in the
graphs even though the data exists.

The machine in question's Hobbit client sends five pieces of numeric data
(email queues) and these are displayed on the web page for this service:

====
*Thursday July 24 11:29:11 UTC 2008*

ActiveStatus:  ***green*<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>;
ActiveQueue: 106
ActiveTrend: tendency* rising* with* -60* mails.
BounceStatus:  ***green*<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>;
BounceQueue: 58
BounceTrend: tendency* falling* with* 3* mails.
DeferStatus:  ***red* <http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif>;
DeferQueue: 150464
DeferTrend: tendency* falling* with* 95* mails.
CorruptStatus:  ***green*<http://hbt0.lon.oss.colt.net/hobbit/gifs/green.gif>;
CorruptQueue: 0
CorruptTrend: amount equal to last measure.
IncomingStatus:  ***red*<http://hbt0.lon.oss.colt.net/hobbit/gifs/red.gif>;
IncomingQueue: 247049
IncomingTrend: amount equal to last measure.

 Deferred Queue is too high but is decreasing already.

====
These numbers change over time and the values are accurate.

However, the graph that is displayed below this data is blank. I have
historic data, the files exist, and what is more I have other machines that
are configured identically to this one where the data IS graphed correctly.

Hobbit graphs are a bit of a black hole to me, can anyone suggest where I
might look?

|\/|artin


*************************************************************************************
The message is intended for the named addressee only and may not be
disclosed to or used by anyone else, nor may it be copied in any way.

The contents of this message and its attachments are confidential and may
also be subject to legal privilege. If you are not the named addressee
and/or have received this message in error, please advise us by e-mailing
user-61c7f445d564@xymon.invalid and delete the message and any attachments without
retaining any copies.

Internet communications are not secure and COLT does not accept
responsibility for this message, its contents nor responsibility for any
viruses.

No contracts can be created or varied on behalf of COLT
Telecommunications, its subsidiaries or affiliates ("COLT") and any other
party by email Communications unless expressly agreed in writing with such
other party.

Please note that incoming emails will be automatically scanned to
eliminate potential viruses and unsolicited promotional emails. For more
information refer to www.colt.net or contact us on +44(0)20 7390 3900.

--
Tel: XXXX XXX XXX
Fax: XXXX XXX XXX
email: user-e365c1418192@xymon.invalid

-- 
Tel: XXXX XXX XXX
Fax: XXXX XXX XXX
email: user-e365c1418192@xymon.invalid

list Brian Catlin · Thu, 24 Jul 2008 19:38:29 -0400 (Eastern Daylight Time) ·

Check your ownerships and permissions on the files in your data store.  I
have had issues in the past where everyone (Apache, hobbit, rrdtool) could
not play in the sandbox until I reset the permissions and ownerships... 
Check the graph tutorial too on the Shire also for hints.. http://www
trantor.org/theshire/doku.php/tutorials
 user-259d6a9a548a@xymon.invalid -------Original Message-------  From: Phil Wild Date: 7/24/2008 12:46:54 PM To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] Graphs are missing data, but it's there!  The rrd version should be okay, after all it is graphing data from other

▸ quoted from Phil Wild

hosts with no problem.  It would appear that you ncv and graph configurations are correct as you say
they are working for other hosts. This would indicate it is a problem with
this host's configuration, so where to look...  Just out of interest, can you take an rrd file this test from a host that
works, and copy it into the .../data/rrd/hostname directory of the host that
does not?  I would expect after doing this that you will have a graph for this host.
Can you confirm this works? After doing this and leaving it for 10 minutes,
do you see any new data in the graph?  Can you dump the data from this rrd file?   2008/7/25 Ward, Martin <user-2d33a6eb6a05@xymon.invalid>:

Are you saying that you run the same tests on multiple hosts and only one
host in not showing data? Yes.


Does this mean they all share the same NCV configuration in hobberserver
cfg and the same graph definition in hobbitgraph.cfg? Yes.

▸ quoted from Phil Wild

What if you remove the rrd file and let hobbit create a new one, does that
help? I did this and as you'd expect initially the web page showed no graph
although it did show data (stored from the previous run I presume).  After an interval the file appeared again but running "rrdtool dump" on it

STILL failed to produce any data.  I'm starting to wonder about the versions of RRD, but they ought to be
data-compatible; I'm using rrdtool v1.2.15.  The histlogs show no errors, the hist/mc25,... Data file contains valid data
 I DO get a few RRD errors like this:  Rrd-status.log:2008-07-21 09:46:19 RRD error updating


/opt/hobbit/data/rrd/mc25.Lon.server.colt.net/TCP.smtp.rrd from 10.44.107
48: illegal attempt to update using time 1216633579 when last update time is

▸ quoted from Phil Wild

1216633579 (minimum one second step) Which make it look like Hobbit is actually updating the RRD file... I just
can't get any data out! |\/| -----Original Message----- From: Phil Wild [mailto:user-e365c1418192@xymon.invalid] Sent: 24 July 2008 16:31 To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] Graphs are missing data, but it's there! Are you saying that you run the same tests on multiple hosts and only one
host in not showing data? Does this mean they all share the same NCV
configuration in hobberserver.cfg and the same graph definition in
hobbitgraph.cfg? If this is correct, then it really points to something not getting into the
rrd file. As previously suggested, rrd dump is your best bet at finding the
problem here. What if you remove the rrd file and let hobbit create a new
one, does that help? Cheers Phil 2008/7/24 Hubbard, Greg L <user-d970b5e56ec9@xymon.invalid>: You know the data exists because you used the rrd dump tool to display it? Is the graph simply not shown at all, or is there a "hole" in the Web page
where it normally would go? ("show page source" might have a clue). Some ideas/shots in the dark: a) check the logs b) meticulously compare a "working" system to the non-working system, and
make sure that they really are identical. c) look at the trends page for this host to see if the graph is okay there..
Etc. I am sure you know the drill -- a big pain to look under every rock,
but it has to be done... GLH From: Ward, Martin [mailto:user-2d33a6eb6a05@xymon.invalid] Sent: Thursday, July 24, 2008 8:21 AM To: user-ae9b8668bcde@xymon.invalid Subject: RE: [hobbit] Graphs are missing data, but it's there! Thanks for the suggestion but that didn't work (I guess you meant rrd). Any other ideas? |\/| -----Original Message----- From: Roberts, James [mailto:user-d7df29bbad83@xymon.invalid] Sent: 24 July 2008 12:47 To: user-ae9b8668bcde@xymon.invalid Subject: RE: [hobbit] Graphs are missing data, but it's there! You need to touch all the rdd. From: Ward, Martin [mailto:user-2d33a6eb6a05@xymon.invalid] Sent: 24 July 2008 12:43 To: user-ae9b8668bcde@xymon.invalid Subject: [hobbit] Graphs are missing data, but it's there! All, I have a problem with one machine where its data is not being shown in the
graphs even though the data exists. The machine in question's Hobbit client sends five pieces of numeric data
(email queues) and these are displayed on the web page for this service: ==== Thursday July 24 11:29:11 UTC 2008 ActiveStatus: green ActiveQueue: 106 ActiveTrend: tendency rising with -60 mails. BounceStatus: green BounceQueue: 58 BounceTrend: tendency falling with 3 mails. DeferStatus: red DeferQueue: 150464 DeferTrend: tendency falling with 95 mails. CorruptStatus: green CorruptQueue: 0 CorruptTrend: amount equal to last measure. IncomingStatus: red IncomingQueue: 247049 IncomingTrend: amount equal to last measure. Deferred Queue is too high but is decreasing already. ==== These numbers change over time and the values are accurate. However, the graph that is displayed below this data is blank. I have
historic data, the files exist, and what is more I have other machines that
are configured identically to this one where the data IS graphed correctly. Hobbit graphs are a bit of a black hole to me, can anyone suggest where I
might look? |\/|artin *****************************************************************************
******* The message is intended for the named addressee only and may not be
disclosed to or used by anyone else, nor may it be copied in any way. The contents of this message and its attachments are confidential and may
also be subject to legal privilege. If you are not the named addressee
and/or have received this message in error, please advise us by e-mailing
user-61c7f445d564@xymon.invalid and delete the message and any attachments without
retaining any copies. Internet communications are not secure and COLT does not accept
responsibility for this message, its contents nor responsibility for any
viruses. No contracts can be created or varied on behalf of COLT Telecommunications,
its subsidiaries or affiliates ("COLT") and any other party by email
Communications unless expressly agreed in writing with such other party. Please note that incoming emails will be automatically scanned to eliminate
potential viruses and unsolicited promotional emails. For more information
refer to www.colt.net or contact us on +44(0)20 7390 3900. --
Tel: XXXX XXX XXX Fax: XXXX XXX XXX email: user-e365c1418192@xymon.invalid --
Tel: XXXX XXX XXX Fax: XXXX XXX XXX email: user-e365c1418192@xymon.invalid

list Martin Ward · Fri, 25 Jul 2008 14:54:36 +0100 ·

OK, with everyone's help I have made progress. After trying all the
different suggestions it came down to: Why can't I get an "rrdtool dump"
output? The reason was that sometime in the past someone (probably me)
managed to replace the rrdtool binary with an empty file (stop
sniggering at the back please).
 
Having done this before I know how it happens... you type your command
line:
/opt/rrdtool/bin/rrdtool dump postfixqueue.rrd
 
but when using bash command line editing you manage to put a > at the
start, making the command line:

/opt/rrdtool/bin/rrdtool dump postfixqueue.rrd

 
The file still keeps its execute permission, but executing an empty file
returns nothing...
 
So, having got a real, working copy of the rrdtool program and running
it on the dodgy data file I can see that data is indeed being stored
there, and a vast number of lines look like this:
<!-- 2008-07-25 12:45:00 UTC / 1216989900 --> <row><v> NaN </v><v> NaN
</v><v> NaN </v><v> NaN </v><v> NaN </v></row>
 
a copy from one of the working ones shows:
<!-- 2008-07-25 13:00:00 UTC / 1216990800 --> <row><v> 5.5427200000e+03
</v><v> 2.1861333333e+02 </v><v> 1.4601324333e+05 </v><v>
0.0000000000e+00 </v><v> 4.0939317667e+05 </v></row>
 
So it seems to be a problem with translating the output from the client
program into data that RRD can understand.
 
Now, here is the contents of the hostlogs file of the working server,
this should tie up with the data entry above:
----
red Friday July 25 12:59:31 UTC 2008
 
<br><br>
 
<pre>
 
ActiveStatus: &red
ActiveQueue: 5494
ActiveTrend: tendency <b>rising</b> with <b>-81</b> mails.
BounceStatus: &green
BounceQueue: 219
BounceTrend: tendency <b>rising</b> <b>-2</b> mails.
DeferStatus: &green
DeferQueue: 145971
DeferTrend: amount equal to last measure.

▸ quoted from Brian Catlin

CorruptStatus: &green
CorruptQueue: 0
CorruptTrend: amount equal to last measure.
IncomingStatus: &red


IncomingQueue: 409494
IncomingTrend: tendency <b>falling</b> with <b>858</b> mails.
 
 
</pre>
Status unchanged in 0.00 minutes
Message received from 10.44.107.107
Client data ID 1216990657
----
and here are the contents of the non-working one:
----
red Friday July 25 12:45:04 UTC 2008
 
<br><br>
 
<pre>
 
ActiveStatus: &green
ActiveQueue: 39
ActiveTrend: tendency <b>falling</b> with <b>973</b> mails.
BounceStatus: &green
BounceQueue: 58
BounceTrend: amount equal to last measure.
DeferStatus: &red
DeferQueue: 154348
DeferTrend: tendency <b>falling</b> with <b>865</b> mails.

▸ quoted from Brian Catlin

CorruptStatus: &green
CorruptQueue: 0
CorruptTrend: amount equal to last measure.
IncomingStatus: &red


IncomingQueue: 206927
IncomingTrend: tendency <b>rising</b> with <b>-206926</b> mails.
 
 Deferred Queue is too high but is decreasing already.<br>
 
</pre>
Status unchanged in 0.00 minutes
Message received from 10.44.107.105
Client data ID 1216989837
----
 
As mentioned previously all these servers use the same scripts to send
the data to the server and the same scripts to process it once it
arrives, indeed as you can see above the two different entries look
identical in format. I checked the scripts on the remote servers to see
if there were any differences between them and found a few minor
differences but nothing huge. Still, just to be sure I copied the
postfixqueue.sh script from a working server to the broken one and
waited for it to run. Alas, although the script transmits sensible data
back to the Hobbit server:
----
ActiveStatus: &green
ActiveQueue: 448
ActiveTrend: tendency falling with 9 mails.
BounceStatus: &green
BounceQueue: 59
BounceTrend: tendency rising -1 mails.
DeferStatus: &green
DeferQueue: 149697
DeferTrend: amount equal to last measure.

▸ quoted from Brian Catlin

CorruptStatus: &green
CorruptQueue: 0
CorruptTrend: amount equal to last measure.
IncomingStatus: &red


IncomingQueue: 213848
IncomingTrend: amount equal to last measure.
----
The rrd file STILL contains:
<!-- 2008-07-25 13:45:00 UTC / 1216993500 --> <row><v> NaN </v><v> NaN
</v><v> NaN </v><v> NaN </v><v> NaN </v></row>
 
Any RRD experts got any ideas?
 
|\/|artin

▸ quoted from Phil Wild

 

	-----Original Message-----
	From: Phil Wild [mailto:user-e365c1418192@xymon.invalid] 
	Sent: 24 July 2008 17:42
	To: user-ae9b8668bcde@xymon.invalid
	Subject: Re: [hobbit] Graphs are missing data, but it's there!
	
	
	The rrd version should be okay, after all it is graphing data
from other hosts with no problem.
	
	It would appear that you ncv and graph configurations are
correct as you say they are working for other hosts. This would indicate
it is a problem with this host's configuration, so where to look...
	
	Just out of interest, can you take an rrd file this test from a
host that works, and copy it into the .../data/rrd/hostname directory of
the host that does not?
	
	I would expect after doing this that you will have a graph for
this host. Can you confirm this works? After doing this and leaving it
for 10 minutes, do you see any new data in the graph?
	
	Can you dump the data from this rrd file?
	
	
	2008/7/25 Ward, Martin <user-2d33a6eb6a05@xymon.invalid>:

Are you saying that you run the same tests on multiple
hosts and only one host in not showing data?

		Yes.

Does this mean they all share the same NCV
configuration in hobberserver.cfg and the same graph definition in
hobbitgraph.cfg?

		Yes.

What if you remove the rrd file and let hobbit create
a new one, does that help?

I did this and as you'd expect initially the web page
showed no graph although it did show data (stored from the previous run
I presume).

After an interval the file appeared again but running
"rrdtool dump" on it STILL failed to produce any data.

I'm starting to wonder about the versions of RRD, but
they ought to be data-compatible; I'm using rrdtool v1.2.15.

The histlogs show no errors, the hist/mc25,... data file
contains valid data. I DO get a few RRD errors like this:

rrd-status.log:2008-07-21 09:46:19 RRD error updating
/opt/hobbit/data/rrd/mc25.lon.server.colt.net/tcp.smtp.rrd from
10.44.107.48: illegal attempt to update using time 1216633579 when last
update time is 1216633579 (minimum one second step)

which make it look like Hobbit is actually updating the
RRD file... I just can't get any data out!

|\/|

-----Original Message-----
From: Phil Wild [mailto:user-e365c1418192@xymon.invalid]
Sent: 24 July 2008 16:31
To: user-ae9b8668bcde@xymon.invalid

Subject: Re: [hobbit] Graphs are missing data,
but it's there!

Are you saying that you run the same tests on
multiple hosts and only one host in not showing data? Does this mean
they all share the same NCV configuration in hobberserver.cfg and the
same graph definition in hobbitgraph.cfg?

If this is correct, then it really points to
something not getting into the rrd file. As previously suggested, rrd
dump is your best bet at finding the problem here. What if you remove
the rrd file and let hobbit create a new one, does that help?

Cheers

Phil

2008/7/24 Hubbard, Greg L
<user-d970b5e56ec9@xymon.invalid>:

You know the data exists because you
used the rrd dump tool to display it?

Is the graph simply not shown at all, or
is there a "hole" in the Web page where it normally would go? ("show
page source" might have a clue).

Some ideas/shots in the dark:

a) check the logs

b) meticulously compare a "working"
system to the non-working system, and make sure that they really are
identical.

c) look at the trends page for this host
to see if the graph is okay there...

Etc. I am sure you know the drill -- a
big pain to look under every rock, but it has to be done...

GLH

From: Ward, Martin
[mailto:user-2d33a6eb6a05@xymon.invalid]

Sent: Thursday, July 24, 2008 8:21 AM

To: user-ae9b8668bcde@xymon.invalid
Subject: RE: [hobbit] Graphs are missing
data, but it's there!

Thanks for the suggestion but that
didn't work (I guess you meant rrd).