recovery messages
list Larry Barber
I've noticed that if I have a situation where I have red alert that turns yellow before finally returning to green that no recovery messages are being sent. Alert notifications are only being sent on reds, not yellow. I suspect that when the red turns to a yellow that Hobbit is "forgetting" that it sent out the notification when it turned red since it is not sending them out on yellows. Does anybody out there know how to fix this? Thanks, Larry Barber
list Josh Luthman
What are your alert rules? I believe the same step that "turns it red" also goes through alerting.
▸
On 9/4/09, Larry Barber <user-6ef9c2864140@xymon.invalid> wrote:I've noticed that if I have a situation where I have red alert that turns yellow before finally returning to green that no recovery messages are being sent. Alert notifications are only being sent on reds, not yellow. I suspect that when the red turns to a yellow that Hobbit is "forgetting" that it sent out the notification when it turned red since it is not sending them out on yellows. Does anybody out there know how to fix this? Thanks, Larry Barber
--
Josh Luthman
Office: XXX-XXX-XXXX
Direct: XXX-XXX-XXXX
XXXX Wayne St
Suite XXXX
Troy, OH XXXXX
"When you have eliminated the impossible, that which remains, however
improbable, must be the truth."
--- Sir Arthur Conan Doyle
list Johan Sjöberg
We had the same problem. We "solved" it by changing yellow to an "OK color" in hobbitserver.cfg, so now it sends out recovered when it goes from red to yellow. But if you have alerts on yellow, you probably don't want to do that. /Johan
▸
From: Larry Barber [mailto:user-6ef9c2864140@xymon.invalid]
Sent: den 4 september 2009 15:59
To: hobbit
Subject: [hobbit] recovery messages
I've noticed that if I have a situation where I have red alert that turns yellow before finally returning to green that no recovery messages are being sent. Alert notifications are only being sent on reds, not yellow. I suspect that when the red turns to a yellow that Hobbit is "forgetting" that it sent out the notification when it turned red since it is not sending them out on yellows. Does anybody out there know how to fix this?
Thanks,
Larry Barber
list Alan Sparks
This is a known problem with the alert mechanism. I posted an analysis of why (I believe) it happens in this mailing list message: http://www.hswn.dk/hobbiton/2008/07/msg00132.html Unfortunately, there appears to be no way in current state to work around the problem. -Alan
▸
Johan Sjöberg wrote:We had the same problem. We “solved” it by changing yellow to an “OK color” in hobbitserver.cfg, so now it sends out recovered when it goes from red to yellow. But if you have alerts on yellow, you probably don’t want to do that. /Johan *From:* Larry Barber [mailto:user-6ef9c2864140@xymon.invalid] *Sent:* den 4 september 2009 15:59 *To:* hobbit *Subject:* [hobbit] recovery messages I've noticed that if I have a situation where I have red alert that turns yellow before finally returning to green that no recovery messages are being sent. Alert notifications are only being sent on reds, not yellow. I suspect that when the red turns to a yellow that Hobbit is "forgetting" that it sent out the notification when it turned red since it is not sending them out on yellows. Does anybody out there know how to fix this? Thanks, Larry Barber
list Larry Barber
The alert rules are very simple:
PAGE=%^APHIS$
SCRIPT /usr/local/hobbit/server/ext/SaveAlert.py DUMMY COLORS=red
DURATION>0 REPEAT=30d RECOVERED
... and so on for each page of the Hobbit installation. The script is the
start of a process that helps our NOC to manage alerts and notify
responsible parties, not being able to automatically remove things that go
red->yellow->green will cause problems.
Thanks,
Larry Barber
▸
On 9/4/09, Josh Luthman <user-4c45a83f15cb@xymon.invalid> wrote:What are your alert rules? I believe the same step that "turns it red" also goes through alerting. On 9/4/09, Larry Barber <user-6ef9c2864140@xymon.invalid> wrote:I've noticed that if I have a situation where I have red alert that turns yellow before finally returning to green that no recovery messages are being sent. Alert notifications are only being sent on reds, not yellow. I suspect that when the red turns to a yellow that Hobbit is "forgetting" that it sent out the notification when it turned red since it is not sending them out on yellows. Does anybody out there know how to fix this? Thanks, Larry Barber-- Josh Luthman Office: XXX-XXX-XXXX Direct: XXX-XXX-XXXX XXXX Wayne St Suite XXXX Troy, OH XXXXX "When you have eliminated the impossible, that which remains, however improbable, must be the truth." --- Sir Arthur Conan Doyle
list Larry Barber
Actually, that will work for me since I have a separate Hobbit instance handling my alerting from the one that runs the script. Thanks, Larry Barber
▸
On 9/4/09, Johan Sjöberg <user-74c177c1220d@xymon.invalid> wrote:We had the same problem. We “solved” it by changing yellow to an “OK color” in hobbitserver.cfg, so now it sends out recovered when it goes from red to yellow. But if you have alerts on yellow, you probably don’t want to do that. /Johan *From:* Larry Barber [mailto:user-6ef9c2864140@xymon.invalid] *Sent:* den 4 september 2009 15:59 *To:* hobbit *Subject:* [hobbit] recovery messages I've noticed that if I have a situation where I have red alert that turns yellow before finally returning to green that no recovery messages are being sent. Alert notifications are only being sent on reds, not yellow. I suspect that when the red turns to a yellow that Hobbit is "forgetting" that it sent out the notification when it turned red since it is not sending them out on yellows. Does anybody out there know how to fix this? Thanks, Larry Barber
list Alan Sparks
Given the state of the alerts module for this type of case (reference my earlier reply), I've had to live with it. Only other solution beside a rewrite of the alert module I've imagined, has been to just use a single rule to send all alerts via script to an external script/process, that does a more complete management of sending alerts. My belief is that the recovery event does happen - there's just no way to write the matching recovery rule. -Alan
▸
Larry Barber wrote:The alert rules are very simple: PAGE=%^APHIS$ SCRIPT /usr/local/hobbit/server/ext/SaveAlert.py DUMMY COLORS=red DURATION>0 REPEAT=30d RECOVERED ... and so on for each page of the Hobbit installation. The script is the start of a process that helps our NOC to manage alerts and notify responsible parties, not being able to automatically remove things that go red->yellow->green will cause problems. Thanks, Larry Barber On 9/4/09, *Josh Luthman* <user-4c45a83f15cb@xymon.invalid <mailto:user-4c45a83f15cb@xymon.invalid>> wrote: What are your alert rules? I believe the same step that "turns it red" also goes through alerting. On 9/4/09, Larry Barber <user-6ef9c2864140@xymon.invalid <mailto:user-6ef9c2864140@xymon.invalid>> wrote:I've noticed that if I have a situation where I have red alert that turns yellow before finally returning to green that no recovery messages are being sent. Alert notifications are only being sent on reds, not yellow. I suspect that when the red turns to a yellow that Hobbit is "forgetting" that it sent out the notification when it turned red since it is not sending them out on yellows. Does anybody out there know how to fix this? Thanks, Larry Barber-- Josh Luthman Office: XXX-XXX-XXXX Direct: XXX-XXX-XXXX XXXX Wayne St Suite XXXX Troy, OH XXXXX "When you have eliminated the impossible, that which remains, however improbable, must be the truth." --- Sir Arthur Conan Doyle