All good questions. Hunting for the answers helped me to see some
patterns I'd missed before.
The xymon server hostname and IP seem to be consistent, but that's about
all that is consistent.
There is a separate column for 'disks' on the main webpage and it
correctly shows the output from
a 'df' command. The script running on the clients' sides is called
"raid.sh", the comments at the top
of the script indicate it is over a decade old; bb-mdstat.h based on
bb-raid.sh. There's a link from
/home/xymon-client/ext to /usr/share/xymon-client/ext on most systems.
The directory and the
scripts in it are owned by either root or xymon. Changing location,
ownership, and perms to match
one of the working systems hasn't helped.
The broken raid reports are all from Linux boxes. The working reports
look like this:
* Mon Aug 31 09:38:49 AKDT 2015 RAID ALL devices OK*
* green md0 Status OK*
* green md1 Status OK*
* green md2 Status OK*
* ============================ /proc/mdstat
===========================*
* Personalities : [raid1] *
* md0 : active raid1 sdc1[1] sda1[0]*
* 511988 blocks super 1.0 [2/2] [UU]*
* md2 : active raid1 sdd[3] sdb[2]*
* 536869888 blocks super 1.2 [2/2] [UU]*
* md1 : active raid1 sdc2[1] sda2[2]*
* 41428924 blocks super 1.1 [2/2] [UU]*
* bitmap: 1/1 pages [4KB], 65536KB chunk*
* unused devices: *
* Run /sbin/mdadm -D /dev/md* for more info*
The non-working systems either show nothing at all (that's better than
purple) OR show the same
three green md[0-2] devices (whether it has three raid devices or not)
on a blue disabled background.
So, I'm almost positive someone copied a working system incorrectly to
other clients without cleaning
up the foreign logs. The working systems overwrote or just aged out the
incorrect information while the
non-working ones just keep reporting it. I have found logs but none for
this raid information. Perhaps the
logs are compressed or otherwise rendered humanly unreadable.
So, I copied the /usr/share/xymon-client/ext scripts from a working
system to several that were reporting
nothing and restarted xymon-client. Most did nothing, one is showing a
"no data" indicator. The raid out-
put looks normal except the device is md127 - perhaps the high number is
confusing the script. But the
wbinfo.sh script I copied at the same time to/from the same directory is
now showing green. Argh!
I don't even know where the xymon-client scripts running here came from
so I'm reluctant (but motivated)
to just rip them all out by the roots and start over from a known
baseline.
WLR
==================================================================================
Phil Crooker <user-e8e31cd73303@xymon.invalid>
3:57 PM (17 hours ago)
Is the hostname wrong somewhere? I'm thinking maybe the scipt is sending
the wrong hostname, somehow....
==================================================================================
Jeremy Laidman <user-71895fb2e44c@xymon.invalid>
7:07 PM (14 hours ago)
On 30 August 2015 at 14:22, Walter Rutherford <user-6b85327bb8bb@xymon.invalid>
wrote:
This is probably an old issue but I didn't see a way to search the
archives.
https://www.google.com/?q=site:lists.xymon.com+purple+raid
Our xymon server is showing purple indicators for two of our custom
scripts
but only on a handful of systems.
The scripts are running client-side and/or server-side? Can you
describe how the scripts work? Are they locally-written scripts or did you
get them from somewhere online?
RAID checks are not standard for most Xymon clients. I've never used or
seen RAID checks. A quick look at the source code indicates built-in
support for only Linux, where "md" devices are identified in /proc/mdstat.
At the bottom of the incorrect raid report page there is a
link to "client data". If I follow the link I get a full report
including the correct,
current raid information!
How is the RAID information getting into the client data? This might
not be used by your custom scripts, and so might be a red herring. More
detail is required about the raid scripts. Or whether you're using the
built-in support for Linux RAID meta-devices reporting with client data in
the [mdstat] section. If the latter, perhaps you could show the [mdstat]
section of client data?
Cheers
====================================================================================
---------- Forwarded message ----------
From: Walter Rutherford <user-6b85327bb8bb@xymon.invalid>
Date: Sat, Aug 29, 2015 at 8:22 PM
Subject: purple problems
To: Xymon at xymon.com
Hey all,
This is probably an old issue but I didn't see a way to search the
archives.
Our xymon server is showing purple indicators for two of our custom
scripts
but only on a handful of systems. I've found differences in file
location, file
ownership, UID, GID, etc.. but so far none of that seems to be the
problem.
The custom script checks raids. Strangely, all of the stagnant hosts show
the same three disks entries from mid-July no matter how many disks they
really have. Unfortunately I don't know what may've happened in July;
that
was before I started working here. I suspect the xymon-client software
was
copied from a live system, including the old status reports, but in so
doing
something wasn't re-configured correctly for the new systems.
Even stranger, at my urging the Lead SA undisabled the purple
notifications.
I was expecting the page to go purple but it remains green even though
the
page isn't updating. At the bottom of the incorrect raid report page
there is a
link to "client data". If I follow the link I get a full report *including
the correct,*
*current raid information*!
I think this means that the client is capturing the correct data and
sending
it to the server, the server is actually receiving the report, but after
that the
raid report isn't being handled correctly. Other systems display as
expected.
So far I haven't found anywhere on the server that the purple systems
are
configured or handled differently.
I doubt we're the first to experience this problem. Does this sound
familiar?
Thanks in advance for any hints you can provide for where to look next.
WLR