delete queued alerts 🔗 link

We had add an 'a-ha' moment the other day when our Hobbit server lost network connectivity, and once rebooted it blasted out basically every alert configured.  If this were to happen again, is there a way to silence or delete queued alerts before rebooting?

I'm also curious how this community handles a case like that.

In <user-ed8172b74598@xymon.invalid> "user-747f2264746a@xymon.invalid" <user-747f2264746a@xymon.invalid> writes:

We had add an 'a-ha' moment the other day when our Hobbit server lost netwo=
rk connectivity, and once rebooted it blasted out basically every alert con=
figured.=A0 If this were to happen again, is there a way to silence or dele=
te queued alerts before rebooting?=0A=0AI'm also curious how this community=
handles a case like=A0that.=0A=0A=0A      

I suppose your alerts go out via e-mail ? In that case, the issue resolves to
stopping your mailer daemon and clearing the mail queue of the pending alerts.


Regards,
Henrik

We use email for most and scripts for critical alerts.  It appears that both were held up 'in queue' while the network connection was hosed.  I was curious if Hobbit actually had a queue where this stuff ended up, since the alerts sent after the reboot.

----- Original Message ----
From: Henrik Stoerner <user-ce4a2c883f75@xymon.invalid>
To: user-ae9b8668bcde@xymon.invalid
Sent: Tuesday, October 14, 2008 3:42:07 PM
Subject: Re: [hobbit] delete queued alerts

In <user-ed8172b74598@xymon.invalid> "user-747f2264746a@xymon.invalid" <user-747f2264746a@xymon.invalid> writes:

I suppose your alerts go out via e-mail ? In that case, the issue resolves to
stopping your mailer daemon and clearing the mail queue of the pending alerts.


Regards,
Henrik

Although I've been using Hobbit and Big Brother for a few years, I don't
consider myself a "guru".  But I think that if your Hobbit server was
not receiving messages from the clients, and, of course, the ping tests
were failing, the alerts were generating and being sent out via email.
But since your email program couldn't send them out, the alert emails
just kept stacking-up in the mail queue.  As soon as network
connectivity was restored, all those emails got sent...BOOM!

 
I doubt if you would have had time to do anything to stop those messages
from going out.  It probably only took a few seconds for them to get
sent.

 
Perhaps it would be possible to write a script that would shut down the
Hobbit server application if the email program couldn't send messages.
Other than that, I can't immediately come up with a clean solution for
this issue.

 
Later!

 
Jim Smith

SVHS

Little Rock

 
From: user-747f2264746a@xymon.invalid [mailto:user-747f2264746a@xymon.invalid] 
Sent: Tuesday, October 14, 2008 4:36 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] delete queued alerts

 
We use email for most and scripts for critical alerts.  It appears that
both were held up 'in queue' while the network connection was hosed.  I
was curious if Hobbit actually had a queue where this stuff ended up,
since the alerts sent after the reboot.

 
----- Original Message ----
From: Henrik Stoerner <user-ce4a2c883f75@xymon.invalid>
To: user-ae9b8668bcde@xymon.invalid
Sent: Tuesday, October 14, 2008 3:42:07 PM
Subject: Re: [hobbit] delete queued alerts

In <user-ed8172b74598@xymon.invalid> "user-747f2264746a@xymon.invalid"
<user-747f2264746a@xymon.invalid> writes:

I suppose your alerts go out via e-mail ? In that case, the issue
resolves to
stopping your mailer daemon and clearing the mail queue of the pending
alerts.


Regards,
Henrik

When Ohio was torn apart by the storm a few weeks ago I did as Henrik just
suggested.  I let it boot normally, then went in and stopped sendmail.  I
dropped the mail spool and a few hours (or a day) later when customers
started to gain power I brought it back with a clean queue.  I had several
alerts but as they were all email I didn't much mind.

The alerts I had that were for our major backhauls go to my mobile via SMS.
Those can cause an issue and really the only way I can imagine getting
around this is to create route: for the hop previous to it.

One suggestion I would love to see in Hobbit is a summary email of alerts in
the case of a certain number of red hosts.

Josh Luthman
Office: XXX-XXX-XXXX
Direct: XXX-XXX-XXXX
XXXX Wayne St
Suite XXXX
Troy, OH XXXXX

Those who don't understand UNIX are condemned to reinvent it, poorly.
--- Henry Spencer

On Tue, Oct 14, 2008 at 4:42 PM, Henrik Stoerner <user-ce4a2c883f75@xymon.invalid> wrote: