Intelligent alerting across multiple servers? 🔗 link

Date: Thu, 30 Jul 2009 11:15:14 +0100
From: user-c15424b7e83a@xymon.invalid
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Intelligent alerting across multiple servers?

Hello everyone,

We have 3 Xymon servers communicating together and I have noticed they are clever enough to communicate enable/disables across the board. For example if I was to disable a check on the gui from one xymon server it will communicate this with the other 2 and blue them out automatically.

On Thu, Jul 30, 2009 at 8:32 AM, T.J. Yang<user-8e841282cda5@xymon.invalid> wrote:

Date: Thu, 30 Jul 2009 11:15:14 +0100
From: user-c15424b7e83a@xymon.invalid
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Intelligent alerting across multiple servers?

Hello everyone,

We have 3 Xymon servers communicating together and I have noticed they are
clever enough to communicate enable/disables across the board. For example
if I was to disable a check on the gui from one xymon server it will
communicate this with the other 2 and blue them out automatically.

I have a two servers setting for redunancy.
This is a new feature to me as a hobbit 4.2.0 user, which version are you
using ?

for blue(maintenance mode) records cloning from one server to the other, we
need to dump the blue records on primary and rsync the blue records to
secondary and finally reimport blue records into secondary.

I was wondering if there is something similar for the alerts? We have 3
xymons for redundancy however as far as I can tell if I set up the alerts
config on all 3 then we are going to get 3x identical alerts. Is there any
way they can talk to each other and only send out one?

To avoid alert duplication from secondary servers with hobbit 4.2.0 ,  we
need to write a perl script to check primary from secondary. if primary has
outage of network connection or hobbit server is down then we enable alert
module of secondary hobbbit server. at any given time only one hobbit server
is allowed to send out alerts.

So are you configuring all your hobbit clients to send message to all three
servers ?
This not a concern for a LAN setting but it may become a problem if the
deployment is at the scale of thousands of clients and sending message
across WAN the has smaller wan pipe.


tj

Many thanks
James

Bing™ brings you maps, menus, and reviews organized in one place. Try it
now.

---- Patrick Nixon <user-1f2406f832af@xymon.invalid> wrote: 

Wouldn't it be better on the secondary server(s) to put a conditional
statement in the alert rules that if the primary server is online not
to alert?

I'm pretty sure the rules can handle that.

On Thu, Jul 30, 2009 at 8:32 AM, T.J. Yang<user-8e841282cda5@xymon.invalid> wrote:

Date: Thu, 30 Jul 2009 11:15:14 +0100
From: user-c15424b7e83a@xymon.invalid
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Intelligent alerting across multiple servers?

Hello everyone,

We have 3 Xymon servers communicating together and I have noticed they are
clever enough to communicate enable/disables across the board. For example
if I was to disable a check on the gui from one xymon server it will
communicate this with the other 2 and blue them out automatically.

I have a two servers setting for redunancy.
This is a new feature to me as a hobbit 4.2.0 user, which version are you
using ?

for blue(maintenance mode) records cloning from one server to the other, we
need to dump the blue records on primary and rsync the blue records to
secondary and finally reimport blue records into secondary.

I was wondering if there is something similar for the alerts? We have 3
xymons for redundancy however as far as I can tell if I set up the alerts
config on all 3 then we are going to get 3x identical alerts. Is there any
way they can talk to each other and only send out one?

To avoid alert duplication from secondary servers with hobbit 4.2.0 ,  we
need to write a perl script to check primary from secondary. if primary has
outage of network connection or hobbit server is down then we enable alert
module of secondary hobbbit server. at any given time only one hobbit server
is allowed to send out alerts.

So are you configuring all your hobbit clients to send message to all three
servers ?
This not a concern for a LAN setting but it may become a problem if the
deployment is at the scale of thousands of clients and sending message
across WAN the has smaller wan pipe.


tj

Many thanks
James

Bing™ brings you maps, menus, and reviews organized in one place. Try it
now.

Date: Thu, 30 Jul 2009 14:57:06 +0100
From: user-c15424b7e83a@xymon.invalid
To: user-ae9b8668bcde@xymon.invalid
CC: user-1f2406f832af@xymon.invalid
Subject: Re: [hobbit] Intelligent alerting across multiple servers?

TJ, we have 3 data centres and each data centre contains a xymon server. All clients in a data centre only report to their local xymon server. However the xymon servers can communicate with each other using BBDISPLAYS (its a little more complicated than that as we utilise a bbproxy in each DC to take the messages and spray them to all 3 xymons).

---- Patrick Nixon  wrote:

Wouldn't it be better on the secondary server(s) to put a conditional
statement in the alert rules that if the primary server is online not
to alert?

I'm pretty sure the rules can handle that.

On Thu, Jul 30, 2009 at 8:32 AM, T.J. Yang wrote:

Date: Thu, 30 Jul 2009 11:15:14 +0100
From: user-c15424b7e83a@xymon.invalid
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Intelligent alerting across multiple servers?

Hello everyone,

We have 3 Xymon servers communicating together and I have noticed they are
clever enough to communicate enable/disables across the board. For example
if I was to disable a check on the gui from one xymon server it will
communicate this with the other 2 and blue them out automatically.

I have a two servers setting for redunancy.
This is a new feature to me as a hobbit 4.2.0 user, which version are you
using ?

for blue(maintenance mode) records cloning from one server to the other, we
need to dump the blue records on primary and rsync the blue records to
secondary and finally reimport blue records into secondary.

I was wondering if there is something similar for the alerts? We have 3
xymons for redundancy however as far as I can tell if I set up the alerts
config on all 3 then we are going to get 3x identical alerts. Is there any
way they can talk to each other and only send out one?

To avoid alert duplication from secondary servers with hobbit 4.2.0 ,  we
need to write a perl script to check primary from secondary. if primary has
outage of network connection or hobbit server is down then we enable alert
module of secondary hobbbit server. at any given time only one hobbit server
is allowed to send out alerts.

So are you configuring all your hobbit clients to send message to all three
servers ?
This not a concern for a LAN setting but it may become a problem if the
deployment is at the scale of thousands of clients and sending message
across WAN the has smaller wan pipe.


tj

Many thanks
James

Bing™ brings you maps, menus, and reviews organized in one place. Try it
now.

Date: Thu, 30 Jul 2009 09:02:45 -0400
From: user-1f2406f832af@xymon.invalid
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Intelligent alerting across multiple servers?

Wouldn't it be better on the secondary server(s) to put a conditional
statement in the alert rules that if the primary server is online not
to alert?

I'm pretty sure the rules can handle that.

On Thu, Jul 30, 2009 at 8:32 AM, T.J. Yang wrote:

Date: Thu, 30 Jul 2009 11:15:14 +0100
From: user-c15424b7e83a@xymon.invalid
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Intelligent alerting across multiple servers?

Hello everyone,

We have 3 Xymon servers communicating together and I have noticed they are
clever enough to communicate enable/disables across the board. For example
if I was to disable a check on the gui from one xymon server it will
communicate this with the other 2 and blue them out automatically.

I have a two servers setting for redunancy.
This is a new feature to me as a hobbit 4.2.0 user, which version are you
using ?

for blue(maintenance mode) records cloning from one server to the other, we
need to dump the blue records on primary and rsync the blue records to
secondary and finally reimport blue records into secondary.

I was wondering if there is something similar for the alerts? We have 3
xymons for redundancy however as far as I can tell if I set up the alerts
config on all 3 then we are going to get 3x identical alerts. Is there any
way they can talk to each other and only send out one?

To avoid alert duplication from secondary servers with hobbit 4.2.0 ,  we
need to write a perl script to check primary from secondary. if primary has
outage of network connection or hobbit server is down then we enable alert
module of secondary hobbbit server. at any given time only one hobbit server
is allowed to send out alerts.

So are you configuring all your hobbit clients to send message to all three
servers ?
This not a concern for a LAN setting but it may become a problem if the
deployment is at the scale of thousands of clients and sending message
across WAN the has smaller wan pipe.


tj

Many thanks
James

Bing™ brings you maps, menus, and reviews organized in one place. Try it
now.

NEW mobile Hotmail. Optimized for YOUR phone.  Click here.
http://windowslive.com/Mobile?ocid=TXT_TAGLM_WL_CS_MB_new_hotmail_072009

On Thu, Jul 30, 2009 at 12:48 PM, T.J. Yang<user-8e841282cda5@xymon.invalid> wrote:

Date: Thu, 30 Jul 2009 09:02:45 -0400
From: user-1f2406f832af@xymon.invalid
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Intelligent alerting across multiple servers?

Wouldn't it be better on the secondary server(s) to put a conditional
statement in the alert rules that if the primary server is online not
to alert?

I'm pretty sure the rules can handle that.

This is new to me, would you mind to elaborate more on how to do this kind of alert configuration file in hobbit-alerts.cfg ?

tj

On Thu, Jul 30, 2009 at 8:32 AM, T.J. Yang wrote:

Date: Thu, 30 Jul 2009 11:15:14 +0100
From: user-c15424b7e83a@xymon.invalid
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Intelligent alerting across multiple servers?

Hello everyone,

We have 3 Xymon servers communicating together and I have noticed they are
clever enough to communicate enable/disables across the board. For example
if I was to disable a check on the gui from one xymon server it will
communicate this with the other 2 and blue them out automatically.

I have a two servers setting for redunancy.
This is a new feature to me as a hobbit 4.2.0 user, which version are you
using ?

for blue(maintenance mode) records cloning from one server to the other, we
need to dump the blue records on primary and rsync the blue records to
secondary and finally reimport blue records into secondary.

I was wondering if there is something similar for the alerts? We have 3
xymons for redundancy however as far as I can tell if I set up the alerts
config on all 3 then we are going to get 3x identical alerts. Is there any
way they can talk to each other and only send out one?

To avoid alert duplication from secondary servers with hobbit 4.2.0 ,  we
need to write a perl script to check primary from secondary. if primary has
outage of network connection or hobbit server is down then we enable alert
module of secondary hobbbit server. at any given time only one hobbit server
is allowed to send out alerts.

So are you configuring all your hobbit clients to send message to all three
servers ?
This not a concern for a LAN setting but it may become a problem if the
deployment is at the scale of thousands of clients and sending message
across WAN the has smaller wan pipe.


tj

Many thanks
James

Bing™ brings you maps, menus, and reviews organized in one place. Try it
now.

NEW mobile Hotmail. Optimized for YOUR phone.  Click here.
http://windowslive.com/Mobile?ocid=TXT_TAGLM_WL_CS_MB_new_hotmail_072009

Date: Thu, 30 Jul 2009 21:14:19 -0400
From: user-1f2406f832af@xymon.invalid
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Intelligent alerting across multiple servers?

Oh wow, I must have been in space when I thought about that.

In the standard hobbit-alerts, you are not able to do what I just mentioned.

However, using an external paging script, you can use the 'bb query
HOSTNAME.TESTNAME' command, get the status from secondary server2 for
primary server1 and send the alert to it's destination if server1 is
down. Otherwise discard the alert.

 
tj

On Thu, Jul 30, 2009 at 12:48 PM, T.J. Yang<user-8e841282cda5@xymon.invalid> wrote:

Date: Thu, 30 Jul 2009 09:02:45 -0400
From: user-1f2406f832af@xymon.invalid
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Intelligent alerting across multiple servers?

Wouldn't it be better on the secondary server(s) to put a conditional
statement in the alert rules that if the primary server is online not
to alert?

I'm pretty sure the rules can handle that.

This is new to me, would you mind to elaborate more on how to do this kind of alert configuration file in hobbit-alerts.cfg ?

tj

On Thu, Jul 30, 2009 at 8:32 AM, T.J. Yang wrote:

Date: Thu, 30 Jul 2009 11:15:14 +0100
From: user-c15424b7e83a@xymon.invalid
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Intelligent alerting across multiple servers?

Hello everyone,

We have 3 Xymon servers communicating together and I have noticed they are
clever enough to communicate enable/disables across the board. For example
if I was to disable a check on the gui from one xymon server it will
communicate this with the other 2 and blue them out automatically.

I have a two servers setting for redunancy.
This is a new feature to me as a hobbit 4.2.0 user, which version are you
using ?

for blue(maintenance mode) records cloning from one server to the other, we
need to dump the blue records on primary and rsync the blue records to
secondary and finally reimport blue records into secondary.

I was wondering if there is something similar for the alerts? We have 3
xymons for redundancy however as far as I can tell if I set up the alerts
config on all 3 then we are going to get 3x identical alerts. Is there any
way they can talk to each other and only send out one?

To avoid alert duplication from secondary servers with hobbit 4.2.0 ,  we
need to write a perl script to check primary from secondary. if primary has
outage of network connection or hobbit server is down then we enable alert
module of secondary hobbbit server. at any given time only one hobbit server
is allowed to send out alerts.

So are you configuring all your hobbit clients to send message to all three
servers ?
This not a concern for a LAN setting but it may become a problem if the
deployment is at the scale of thousands of clients and sending message
across WAN the has smaller wan pipe.


tj

Many thanks
James

Bing™ brings you maps, menus, and reviews organized in one place. Try it
now.

NEW mobile Hotmail. Optimized for YOUR phone.  Click here.
http://windowslive.com/Mobile?ocid=TXT_TAGLM_WL_CS_MB_new_hotmail_072009