Double, triple notifications 🔗 link

For some tests and hosts I receive several notification copies (identical) 
- sometimes two, sometimes three.
I had reviewed the configuration, but can not detect where is the problem.

1) How can I debug which rule worked out for each copy?

2) Feature request - I suppose it should be quite easy to avoid redundant 
notification copies before sending,
for example  using 
cat   (rows-list of recepients) | sort | uniq 

P.S. XyMon version 4.3.17 on CentOS (64-bit) if this makes sense.

Best regards,

Andrey Chervonets
SIA CoMinder
http://www.cominder.eu/

Two places to look to troubleshoot are the notifications.log, to check to see whether Xymon actually sent multiples or not, (could be your mail server, theoretically) and the "info" test on the server in question to see what the notifications settings parsed to. You can also look at the config report for that kind of info.

--
____ *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences*
 || \\UTGERS      |---------------------*O*---------------------
 ||_// Biomedical | Ryan Novosielski - Senior Technologist
 || \\ and Health | user-46c89e614701@xymon.invalid - 973/972.0922 (2x0922)
 ||  \\  Sciences | OIRT/High Perf & Res Comp - MSB C630, Newark

      `'
From: Xymon [xymon-bounces at xymon.com] On Behalf Of Andrey Chervonets [user-e7fb5c02322c@xymon.invalid]
Sent: Wednesday, October 01, 2014 8:35 AM
To: xymon at xymon.com
Subject: [Xymon] Double, triple notifications

For some tests and hosts I receive several notification copies (identical) - sometimes two, sometimes three.
I had reviewed the configuration, but can not detect where is the problem.

1) How can I debug which rule worked out for each copy?

2) Feature request - I suppose it should be quite easy to avoid redundant notification copies before sending,
for example  using
cat   (rows-list of recepients) | sort | uniq

P.S. XyMon version 4.3.17 on CentOS (64-bit) if this makes sense.

Best regards,

Andrey Chervonets
SIA CoMinder
http://www.cominder.eu/

On 2014-10-01 9:02 am, Novosielski, Ryan wrote:

Ryan makes some good points, but, as is the case in my setup, it's more 
likely that you have multiple rules in your alerts.cfg that send emails 
for the same alarm, covering multiple sets of servers.

-- 
Mike Burger
http://www.bubbanfriends.org

"It's always suicide-mission this, save-the-planet that. No one ever 
just stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1

Andrey,

I'm running on RHEL 6.5 xymon version 4.3.17.

I'm having a similar problem but only on recovery notices.  Alert picks up the correct rule and stops.  Recover hits that rule then continues on down the list and will then duplicate on my catch all default rule.  This isn't on all alert recoveries.  Appears to only happen on "server" names that have an underscore or dash in it.  I'm just starting to dig in and experiment with it.

One thing that is really helpful in diagnosing alerts.cfg is adding the --cfid option to tasks.cfg xymond_channel.

[alert]
        ENVFILE /data/xymon/server/etc/xymonserver.cfg
        NEEDS xymond
        CMD xymond_channel      --channel=page  \
                                --log=$XYMONSERVERLOGS/alert.log \
                                xymond_alert \
                                --checkpoint-file=$XYMONTMP/alert.chk \
                                --checkpoint-interval=600 \
                                --debug \
                                --cfid

That will put the alerts.cfg line number at the end of the alert email subject line.  "cat -n alerts.cfg" gives you a nice display of all line numbers.

Single alert notice.

Xymon [6578] PHX_FIKE:trouble CRITICAL (RED) [cfid:243]

Duplicate recovery notices.

Xymon PHX_FIKE:trouble recovered [cfid:243]
Xymon PHX_FIKE:trouble recovered [cfid:432]

From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Andrey Chervonets
Sent: Wednesday, October 01, 2014 7:36 AM
To: xymon at xymon.com
Subject: [Xymon] Double, triple notifications

For some tests and hosts I receive several notification copies (identical) - sometimes two, sometimes three.
I had reviewed the configuration, but can not detect where is the problem.

1) How can I debug which rule worked out for each copy?

2) Feature request - I suppose it should be quite easy to avoid redundant notification copies before sending,
for example  using
cat   (rows-list of recepients) | sort | uniq

P.S. XyMon version 4.3.17 on CentOS (64-bit) if this makes sense.

Best regards,

Andrey Chervonets
SIA CoMinder
http://www.cominder.eu/

CONFIDENTIALITY NOTICE:
This electronic mail message is intended exclusively for
recipient to which it is addressed. The contents of this message
and any attachments may contain confidential and privileged
information. Any unauthorized review, use, print, storage, copy,
disclosure or distribution is strictly prohibited. If you have
received this message in error, please advise the sender
immediately by replying to the message's sender and delete all
copies of this message and its attachments without disclosing
the contents to anyone, or using the contents for any purpose.

Turning on the cfid option in xymond_alert might be useful as well.

-jc


--cfid If  this  option  is present, alert messages will include a line
with "cfid:N" where N is the linenumber in the alerts.cfg file that caused
this message to be sent. This can be useful to track down problems with
duplicate alerts.

On Wed, October 1, 2014 6:02 am, Novosielski, Ryan wrote:

Thanks a lot! Options: --debug  --cfid was very useful.   I had found at least one place, which most probably caused the problem.

I would like to explain a bit:

In alerts config. I had used some variables like:

$HOSTS_PROD=host1,host2,host3
$HOSTS_TEST=thost1,thost2
$HOSTS_DEV=dhost1

And some rules, including for  $HOSTS_DEV (some separate as well as some together with $HOSTS_TEST)
and merged variables too:
$HOSTS_ALL=$HOSTS_PROD,$HOSTS_TEST,$HOSTS_DEV

This worked fine until we switched of dhost1 host (reason does not matter) and I had commended variabled definition:
# $HOSTS_DEV=dhost1

But rules remained. As result  there was rules for empty element 
# development
HOST=$HOSTS_DEV       SERVICES=$SVC_DB_BUSN COLOR=red,yellow,purple
  MAIL=$CM_SUPPORT_DBA    DURATION>15m REPEAT=60m RECOVERED  DURATION<180m FORMAT=PLAIN

this rule really resulted to:
HOST=   SERVICES=dbinvobj COLOR=red,yellow,purple
  MAIL=user-67221db774f9@xymon.invalid    DURATION>15m REPEAT=60m RECOVERED DURATION<180m FORMAT=PLAIN

This rule most probably worked out in some cases together with correct one.


Best regards,

Andrey Chervonets
SIA CoMinder
http://www.cominder.eu/