This problem continues and I can’t seem to get anybody’s attention.
Has anyone seen this problem before?????????
Thanks,
John
Upcoming PTO:
John Rothlisberger
IT Strategy, Infrastructure & Security - Technology Growth Platform
TGP for Business Process Outsourcing
Accenture
XXX.XXX.XXXX <(312)%20693-3136> office
*From:* Rothlisberger, John R.
*Sent:* Friday, April 28, 2017 1:40 PM
*To:* 'xymon >> xymon at xymon.com' <xymon at xymon.com>
*Subject:* RE: Alert transition yellow -> red with repeat problem
This problem contributed to an outage last night – something is wrong.
Last night we had a disk that was in the warning state – warning email
sent.
That disk then went into an alert state and an alert email was triggered
(right away as the DURATION value took into account the time it was yellow)
and then again 15 minutes later as designed.
THEN – another disk on that server went yellow. It did NOT trigger any
emails (as expected we should continue to focus on the alerts) but it
somehow interfered with the REPEAT time of the alerts and those STOPPED.
There is a bug somewhere.
Thanks,
John
Upcoming PTO:
John Rothlisberger
IT Strategy, Infrastructure & Security - Technology Growth Platform
TGP for Business Process Outsourcing
Accenture
XXX.XXX.XXXX <(312)%20693-3136> office
*From:* Rothlisberger, John R.
*Sent:* Tuesday, April 11, 2017 7:49 AM
*To:* 'xymon >> xymon at xymon.com' <xymon at xymon.com>
*Subject:* Alert transition yellow -> red with repeat problem
This is a problem I have seen for a long long time and have actually
brought it up on the list before.
Xymon 4.3.21
Ubuntu 14.04LTS
The problem I have (in this instance) a warning which is to be repeated
daily suddenly goes red and triggers a single alert but doesn’t repeat
again until the repeat time of the warning has passed.
From the notification.log:
Tue Mar 28 05:31:28 2017 ServerA.disk (IP) disk_warn 1490697082 100 <-
warning sets a repeat time of 1 day
Tue Mar 28 05:53:32 2017 ServerA.disk (IP) disk_alert 1490698405 100 <-
minutes later it goes red (red repeat time is 15 minutes but no further
alerts are generated)
Next alert comes out 1 day after the above warning:
Wed Mar 29 05:31:31 2017 ServerA.disk (IP) disk_alert 1490783486 100 <- 1
day after previous warning. This should have been repeated every 15
minutes.
Wed Mar 29 05:46:40 2017 ServerA.disk (IP) disk_alert 1490784394 100 <-
now, the repeat time is 15 minutes
… <- and continues every 15 minutes.
This alert went a full 24 hours with only a single notification. L
I have seen this before (not always) where the repeat time in a warning
overrides a follow up alert until the warning repeat time has expired.
Alert rules:
SCRIPT /home/xymon/server/ext/pg/exwarn_SCRIPT disk_warn DURATION>30
REPEAT=1d COLOR=yellow SERVICE=disk FORMAT=TEXT UNMATCHED
SCRIPT /home/xymon/server/ext/pg/exalert_SCRIPT disk_alert DURATION>20
REPEAT=15 COLOR=red SERVICE=disk FORMAT=TEXT UNMATCHED
Ideas/thoughts?
Thanks,
John
Upcoming PTO: 4/3
John Rothlisberger
IT Strategy, Infrastructure & Security - Technology Growth Platform
TGP for Business Process Outsourcing
Accenture
XXX.XXX.XXXX <(312)%20693-3136> office
This message is for the designated recipient only and may contain
privileged, proprietary, or otherwise confidential information. If you have
received it in error, please notify the sender immediately and delete the
original. Any other use of the e-mail by you is prohibited. Where allowed
by local law, electronic communications with Accenture and its affiliates,
including e-mail and instant messaging (including content), may be scanned
by our systems for the purposes of information security and assessment of
internal compliance with Accenture policy.
www.accenture.com