errors with custom graphs 🔗 link

Hello,

I set up some custom graphs to monitor my Memcached instances. However, I appear to have run into an issue. I set up my xymonserver.cfg to collect all of my data into one NCV file, as evidenced by my TEST2RRD and my configuration below. I also believe I defined the datatypes appropriately. The issue that I am seeing is in the RRD file itself, when I use the RRDTOOL to dump the file. All of the values for the various datatypes are showing as NaN. Would someone be able to point me in the direction of what I could be doing wrong?

Thank you,
Sheel

TEST2RRD="<other stuff>,memcache-test=ncv,MC-TFH-8K01-11221=ncv,MC-TFH-8K01-11222=ncv,MC-TFH-8K02-11221=ncv,MC-TFH-8K02-11222=ncv"

NCV_memcache-test="Hits:GAUGE,Misses:GAUGE,reqs:GAUGE,Gets:GAUGE,Sets:GAUGE,storage:GAUGE,items:GAUGE,hitratio:GAUGE,*:NONE"
NCV_MC-TFH-8K01-11111="Hits:DERIVE,Misses:DERIVE,reqs:DERIVE,Gets:DERIVE,Sets:DERIVE,storage:DERIVE,items:DERIVE,hitratio:DERIVE,*:NONE"

On Tue, January 27, 2015 11:45 am, Sheel Shah wrote:

Sheel,

Many of our tests are using SPLITNCV, but that seems like it should be
valid syntax.

Have you been doing development on this system? RRDtool will expect the
template to match exactly and if the rrd file has a different set of DS's
in it from its initial definition the values might not be being inserted
properly.

You might want to try blowing away any existing RRD files for this type
and letting xymond_rrd re-create them.

Also, can you try running xymond_rrd in --debug mode to see if any unusual
errors are thrown on insertion?


HTH,

-jc

Hi J.C.

Thanks for your reply. Unfortunately, switching it to a SPLITNCV didn't have any effect. When I used the RRDTOOL to dump the contents of the RRD files, the data values are still showing as NaN. I confirmed the script I wrote is displaying the data in the new column on the Xymon homepage (column name is MC-TFH-8K01-11111). This is what is showing when I click the green icon to view the detail:

Hits : <number value>
Misses : <number value>
reqs : <number value>
Gets : <number value>
Gets per min : <number value>
Sets per min : <number value>
Sets : <number value>
storage : <number value>
items : <number value>
hitratio : <percentage value> Used Connection : <percentage value>

Are the datatypes required to be uniquely named? In another column on the same host, I have similar data being displayed, with the same names. The graphing functions correctly in that column.

I apologize, but I am somewhat new to writing scripts in Xymon. I'm not familiar with the Xymon_rrd tool. Can you explain how to have it generate the rrd files and to run in debug mode?

Thanks,
Sheel

-----Original Message-----
From: J.C. Cleaver [mailto:user-87556346d4af@xymon.invalid] Sent: Tuesday, January 27, 2015 9:19 PM
To: Sheel Shah
Cc: xymon at xymon.com
Subject: Re: [Xymon] errors with custom graphs

On Tue, January 27, 2015 11:45 am, Sheel Shah wrote:

Sheel,

Many of our tests are using SPLITNCV, but that seems like it should be valid syntax.

Have you been doing development on this system? RRDtool will expect the template to match exactly and if the rrd file has a different set of DS's in it from its initial definition the values might not be being inserted properly.

You might want to try blowing away any existing RRD files for this type and letting xymond_rrd re-create them.

Also, can you try running xymond_rrd in --debug mode to see if any unusual errors are thrown on insertion?


HTH,

-jc

Sheel

Firstly let me suggest you delete the RRD file and confirm that it gets
re-created, then wait 15 minutes for 3 samples to arrive, and dump it once
more.  If it's still not working, then more investigation is needed, but
sometimes this fixes some problems, and helps to identify others.

Secondly, check the rrd-status.log file for indications of problems.

On 30 January 2015 at 08:34, Sheel Shah <user-c23c3c636596@xymon.invalid> wrote:

Run the command "rrdtool info <filename.rrd> | grep last".  This should
give you some useful data.

Firstly, the timestamp of the last update should be shown.  This is
seconds-since-epoch so convert it to your current localtime with something
like "perl -e 'print scalar localtime 1422583073'".  Or run "date +%s" to
show the current time, and calculate the difference.  We'd expect the
last_update time to update when you get more data.

Secondly, check that the "ds" names match what you're putting out, and that
you have no extras or missing ones from your data.  You should see, for
example:

ds[Hits].last_ds = NaN

You might not get the NaN.  Just make sure that everything else looks OK.

Are the datatypes required to be uniquely named?

Only per RRD file.  If you have the same DS names for another test going to
an RRD file with a different name, that's fine.

Should be fine.

No apology necessary.  With the awesome flexibility of Xymon comes a degree
of complexity, and using the extra features requires more than basic
knowledge of how Xymon works.

The xymond_rrd process is run by the xymond_channel process, which in turn
is run by xymonlaunch process - the master process that runs everything.
You should know that there are two xymond_rrd processes, and two
xymond_channel processes that run them, chained together.  One chain is for
data messages, and the other is for status messages.  You are doing data
collection from within test status messages (TEST2RRD), that that's the one
relevant to your situation.

On the Xymon server, the configuration file that determines what runs and
how it runs is tasks.cfg.  Have a look in tasks.cfg and find the
[rrdstatus] section.  (There's a similar [rrddata] section, for the other
channel, as discussed above.)  This is what causes xymond_rrd to run for
the status channel.  You can edit tasks.cfg and add extra parameters to the
command line, and what we want is to add the option to enable debugging.
For example, you could update the [rrdstatus] section to look like this:

[rrdstatus]

	ENVFILE /usr/lib/xymon/server/etc/xymonserver.cfg
	NEEDS xymond
	CMD xymond_channel --channel=status
--log=$XYMONSERVERLOGS/rrd-status.log xymond_rrd
--rrddir=$XYMONVAR/rrd --debug

Notice how I added "--debug" to the end of the last line.  After making the
edit, simply kill xymond_rrd (eg "pkill xymond_rrd") and it will re-start
with the new parameter.  Use "pgrep -f -l ^xymond_rrd" to show the
parameters being used by the two processes (one for data and one for
status).  If one now has "--debug" then you're OK.

The extra debug details will go into the log rrd-status.log.  Have a look
there for interesting details that might indicate a problem.

Cheers
Jeremy