grouping methods

It would be nice to have an elegant solution, but I don't worry about it
that much because 1) linux servers go down so infrequently, and 2) it
would be pretty trivial to set up your own redundancy between hobbit
servers.  I can think of half a dozen ways to do it off the top of my
head.  For example:

Main Hobbit Server (MHS) does its thing normally.  Backup Hobbit Server
(BHS) syncs/mirrors the drive of the MHS server via rsync or whatever,
and runs a copy of hobbit which monitors only one other system: the MHS.
If the BHS detects that the MHS is down, the alert triggers a script
that brings up its mirror copy of the server.

Doug

-----Original Message-----
From: Josh Luthman [mailto:user-4c45a83f15cb@xymon.invalid] Sent: Monday, June 16, 2008 1:58 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] grouping methods

This is quite obviously a well found problem and sought after feature
- getting redundant Hobbit servers.

Please help us, code monkeys =)

Josh

On Mon, Jun 16, 2008 at 1:45 PM, Sloan <user-b1d2c84d244b@xymon.invalid> wrote:

Josh Luthman wrote:

Not sure what the real reasoning is behind this but if you have 1000 >> servers monitored behind 3 hobbit servers each, figure one Hobbit >> server goes down you lost 1000/3000 being monitored.  If you have >> 3000 servers being monitored behind 1 hobbit server, that one point >> of failure leaves you blind of all 3000 servers.

We do it with redundancy. Each server in our various data centers is > monitored by two bb servers, with one of the two set up to send > notifications, but in all other aspects the monitoring is > active/active, and we get only one notification for alerts, rather > than a pair of redundant notifications.

We've not had a bb server go down in all the years we've been using > it, but sometimes wan connectivity goes away due to circumstances > beyond our control, and a bb server in Arizona can't talk to the > corresponding bb server in California, so the normally passive > monitoring server goes into failover mode, and begins sending > notification for alerts, since it can't verify that the other bb server is alive.

Thus, we always receive notifications for all alerts, and in the worst > case we may get redundant notifications in the case of a split brain > situation, which is the lesser of the evils.

Once this notification failover capability makes it into hobbit, we > can finally switch from bb to hobbit.

Joe

--
Josh Luthman
Office: XXX-XXX-XXXX
Direct: XXX-XXX-XXXX
XXXX Wayne St
Suite XXXX
Troy, OH XXXXX

Those who don't understand UNIX are condemned to reinvent it, poorly.
--- Henry Spencer