On Mon, August 24, 2015 10:15 am, Sean MacGuire wrote:
OK, I'll chime in... I wrote Big Brother so know something about
purple alerts.
The purple alert was something no other monitoring system did, and
it took care of the problem of a BB (and Xymon) client dropping dead
and/or machines being in a zombie state (i.e. responding to pings but
otherwise hung).
They're useful as a indication of something being wrong, vs a red or
yellow alert which provide a clear and actionable problem ststus.
So purple alerts are awesome, unless your server has lost contact
with the clients reporting in and everyone goes purple at the same
time resulting in the "Massive Purple Explosion".
Background explanation - the idea was to timestamp reports into the
future, and if a client doesn't report in by then, the validity of
the last report is in question - it's works the exact same way as
the expiration date on a carton of milk - the milk might not have
gone bad, yet, but you might want to check it before drinking it.
Matt Vander Werf wrote:
This is primarily for Henrik and J.C., but anyone else is free to chime
in their thoughts on this as well!
*Background:
We have a Xymon server (the latest Terabithia RPM on RHEL 7) in
production that monitors around 1950 hosts (and consistently growing).
About a week ago, we experienced some pretty bad purple alert storms in
the middle of the night that were all false-positive alerts (over 300
alerts one night). For most of the tests that went purple, they went
back to green at the next update interval. At this point, we've been
unable to figure out a root cause behind this issue, but it hasn't
happened again since early last week (all the easy, understandable
possible causes has been ruled out: network load/bandwidth, CPU load of
Xymon server and of Xymon clients affected, etc.).
We have been using purple alerts for some time now, find them fairly
reliable for the most part, and think they are useful, as machines hang
or something similar (causing the Xymon client on the machine to stop
being able to report to the Xymon server) and we don't get any red or
yellow alerts for any other tests (sometimes a machine can hang but
still have a network connection that can be successfully pinged by
Xymon, we have found). We haven't had any major issues with
false-positive purple alerts (for the most part), or any purple alert
storms, since we started using them consistently for all our machines a
couple years ago.
I understand that when Xymon was first forked from Big Brother a long
while back, it may have been noted that one big change from Big Brother
was that you didn't need to do purple alerts (or something like that)
and that it was discouraged to use purple alerts, as they were seen as
widely unreliable. (I'm hearing this from a coworker of mine, who set up
our original Xymon server some 5 years ago, but have been unable to find
what he's referring to.) But from what I can see from the current
documentation and the mailing list archives, I'm not seeing any place
where the use of purple alerts is discouraged due to them being
unreliable.
*Question(s):
So, I wanted to see what the current thinking/view regarding purple
alerts and the use of purple alerts was by both the original main
maintainer, Henrik, and the more current main maintainer, J.C. (at least
of the current release). Are purple alerts still considered wholly
unreliable, or even somewhat unreliable (or were they ever)? Are they
discouraged in any way or fashion from being used? Have they caused
issues for any of you on this list? Or vice versa: Have they worked well
for you? I'm fully aware that this purple alert storm issue we had is
just a one-off occurrence and we could have not more additional issues
in the future with purple alerts.
I understand that purple alerts are different than other alerts, like
red and yellow alerts, in that it is an indication that the Xymon client
has stopped working/reporting (on a per-test basis) to the Xymon server
for some reason, rather than an issue from a specific test (e.g. with
the CPU load, memory, etc.).
*(Possible) Feature Request:
In addition, I'd be interested if there was a way that you could only
get one alert for a machine if say all the tests for that machine go
purple, instead of an alert for each purple test. I don't believe this
is possible currently, correct? Is this something that could possibly be
implemented in the future? I understand if it's not or if it wouldn't be
very easy.
I appreciate your time in answering my questions and look forward to
your input! (And apologies for the long-winded e-mail!)
Thanks very much in advance!!
--
Matt Vander Werf
Matt,
Generally speaking, a purple alert should be seen first and foremost as an
indication of a failure in the monitoring *system*... where "system"
includes the client pushing data up from the various servers you're paying
attention to.
By having a calculation made on each message's receipt of how long that
message is good for (receipt time + [default, or specified]), we have a
"fail safe" for an unknown issue occurring that requires attention. The
proximate cause of the purple is the failure to receive a message. Whether
that's caused by a hang or death of the usual originator, a bug in a
xymonproxy, a cut network cable, or xymond being unable to handle all of
the traffic sent to it before it times out, is left somewhat as an
exercise for the administrator.
Because purple alerts are generated from xymond's own view of its internal
state (calculated once a minute) and are never sent IN to xymond, purple
alerts should be a reliable indicator that... some other type of
unreliability is going on :)
Because of the wide possibility of different configurations, it's a little
dangerous to create a one-size-fits-all strategy for purples. In a typical
xymon installation with xymonnet and xymond_client running locally on the
same machine, with no proxies or network segments in the middle, and with
clients reporting directly in as well, you really shouldn't see any purple
alerts outside of clients dying... And if the client is dying because the
box is dying, by default you'll only get the 'conn' test red alert instead
of the various xymond_client and xymonnet-generated ones (unless you're
using the 'noclear' line in hosts.cfg).
Your suggestion to have only a single 'purple' come through would
*typically* work, but you'd have to ask yourself which test would be the
representative one. In our case, we found it easiest to nominate a
specific xymond_client test -- "memory" -- and only send purple
notifications for that out to our alert team. That takes care of
xymond_client, while leaving esoteric situations caused by the failure of
different sharded xymonnet's, xymonproxy's, or custom independent tests
free to fail in their own way.
Again, it's the non-typical cases where it gets tricky. What about custom
tests that aren't being generated by xymond_client that are still
functioning? Perhaps you have xymonnet running on a different machine
that's reporting back to xymond (or to a xymonproxy that's reporting back
to xymond!) that has failed in some way. And of course, it could be that
xymond is under heavy load and is unable to keep up with incoming messages
generally (something we experienced in both the TCP and BFQ configs as we
were scaling out).
Sort of along these lines, however, I'd been considering having a more
"host-wide" way of defining certain failure states directly within xymond,
which would allow some of this override logic to happen centrally (and
more reliably). Imagine a 'conn' being red optionally causing *all* tests
to fail-to-clear, removing the need for this calculation from the
remainder of xymonnet tests. Or a true host-wide "disable" that gets
applied to all tests, even new ones, as a xymond flag. A host-wide
"purple-state" could be conceptualized as well.
That's just a thought, though, and it kind of depends on whether people
would find such a feature useful.
Anyway, I hope that's answered some of your questions!
Regards,
-jc