Hi Greg,
The flapping warning is what tips it off. Flap-detection in xymond
functions by looking at alternating alert states (eg, red/green) happening
within a certain period and "pegging" it at the higher status while it's
going back and forth. This prevents spurious recovery messages and
untimely pager death.
The thing is, though, for a normal functioning 'msgs' test it's almost
impossible for it to actually flap out of the box. The logfetch program
which controls the raw data sent to xymond_client for evaluation actually
walks back 6 "periods" (run cycles) in the log file and sends all
subsequent data up to xymond. This helps with mitigating any lost messages
by turning the 'msgs' test into a "Recent Errors in the Log" test instead
of a direct reflection of the event, since xymon is a state-based
monitoring system rather than a single-fire-and-forget (trap) based
system.
Out of the box, a single red event will cause the msgs test to remain red
for a solid 30m -- far too long for flapping to get triggered in most
cases.
Is there any chance you have multiple servers reporting in with the name
'mgmtconsole'? Especially if you're not using FQDN (which it doesn't seem
like you are), that seems like something that might cause this: Two
different servers with the same name, each sending their own red/green
states every few minutes.
HTH,
-jc
On Thu, February 11, 2016 2:15 pm, Greg Earle wrote:
I'm running Xymon 4.3.12-2 server (yeah, I know ...) on my management
system.
(RHEL 6.5 currently)
A couple of days ago I migrated our central syslog server over to the
Xymon server, so now "/var/log/messages" is getting a ton of stuff in
it that it never had before since all my systems are now reporting in
to it.
Ever since then I've seen something weird - every hour (for about 17+
hours)
I was getting RED alerts for the management console's own "msgs" status,
but
the actual e-mail notifications don't show anything marked red in them!
It's either yellow or, as in the forwarded message below, green. I have
no
idea why I was getting RED alerts for this file if it thinks it's yellow
or
green - any ideas?
The only other thing I can add is that when I go to the Web page for
mgmtconsole:msgs, it says "WARNING: Flapping status" at the top.
Is that a clue?
(Update: interestingly, it looks like the status has finally changed to
green a few minutes ago - after having been red for nearly 17 1/2 hours.
Still seeing "WARNING: Flapping status" on the svcstatus Web page,
though.)
Thanks,
- Greg
Begin forwarded message:
From: xymon Monitor <user-c84c5ca2f00e@xymon.invalid>
Subject: Xymon [750466] mgmtconsole:msgs CRITICAL (RED)
Date: February 11, 2016 at 11:57:25 AM PST
To: user-9179ff85409c@xymon.invalid
green Thu Feb 11 11:57:24 PST 2016 - Log files ok
<pre>
</pre>
No entries in <a
href="/xymon-cgi/svcstatus.sh?CLIENT=mgmtconsole&SECTION=msgs:/var/log/messages">/var/log/messages</a>
Full log <a
href="/xymon-cgi/svcstatus.sh?CLIENT=mgmtconsole&SECTION=msgs:/var/log/messages">/var/log/messages</a>
<...SKIPPED...>
Feb 11 11:57:17 host7 nrpe[20194]: [ID 927837 daemon.info] connect from
mtfuji
[... rest elided ... ]
See
http://mgmtconsole/xymon-cgi/svcstatus.sh?HOST=mgmtconsole&SERVICE=msgs