Redundant Hobbit Servers 🔗 link

I'm interested in knowing what everyone is doing for redundancy.  I
currently have 2 hobbit servers and each one performs the same tests as
the other. 

I find this to be a bit wasteful and am looking for the active/passive
type of configuration that has been talked about on this list before.

This configuration works fine, except for alerting.  In order to not be
overwhelmed by messages, I only have one hobbit server performing the
alerting task, the other only sends alerts on the other hobbit host.

I have considered using the "depends=" test, but then I have to maintain
two separate bb-hosts files, rather than copying and pasting from one to
the other.

Is there a better way than what I'm doing?  My goal is to have both
servers configured the same, as much as possible, and have only one of
them at a time send me alerts.  

I'd love to know what you guys are doing to get around these limits.  I
could easily have missed something in the newest update that solves this
problem, but I haven't seen it yet.

Thanks for your thoughts on this subject.

Michael

 
We are moving to running all our monitoring services in (VMWare) virtual
machines.  The vm's are backed up each night to a NAS share.

If the physical server dies, no big deal I can just start the vm on
another host connected to the same SAN.
If the SAN dies (unlikely), I can restore the vm from the backup on some
other host.

We also run Sitescope, which alerts us if BigBrother/Hobbit dies (and
vice versa).

-- 
Scott Kelley, Disney Enterprise IT Platform Services


-----Original Message-----
From: hobbit-return-2800-Scott.Kelley=user-67dd37a217ed@xymon.invalid
[mailto:hobbit-return-2800-Scott.Kelley=user-67dd37a217ed@xymon.invalid] On Behalf Of
Lowery, Michael
Sent: Tuesday, June 21, 2005 9:13 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Redundant Hobbit Servers

I'm interested in knowing what everyone is doing for redundancy.  I
currently have 2 hobbit servers and each one performs the same tests as
the other. 

I find this to be a bit wasteful and am looking for the active/passive
type of configuration that has been talked about on this list before.

This configuration works fine, except for alerting.  In order to not be
overwhelmed by messages, I only have one hobbit server performing the
alerting task, the other only sends alerts on the other hobbit host.

I have considered using the "depends=" test, but then I have to maintain
two separate bb-hosts files, rather than copying and pasting from one to
the other.

Is there a better way than what I'm doing?  My goal is to have both
servers configured the same, as much as possible, and have only one of
them at a time send me alerts.  

I'd love to know what you guys are doing to get around these limits.  I
could easily have missed something in the newest update that solves this
problem, but I haven't seen it yet.

Thanks for your thoughts on this subject.

Michael

On Tue, Jun 21, 2005 at 11:13:01AM -0500, Lowery, Michael wrote:

The simplest solution is an active/passive cluster using a bit of 
shared storage (SAN, NAS or whatever). Put the Hobbit configuration-
and data-files on the SAN, and setup two hosts in an active/passive
cluster. When one host dies, make the cluster fail-over and start up 
Hobbit on the other.

There's a description of setting up a Linux cluster like this in the
"contrib/" directory in Hobbit 4.0.3+.

If you dont worry about preserving full history and trend graphs
in a fail-over situation, you can skip the shared storage requirement
and just setup the two hosts as a cluster. I'd then recommend using
rsync regularly to synchronize the configuration files and also the
checkpoint-files in the server/tmp/ directory - the checkpoint files
contain a snapshot of the Hobbit server memory so if those are copied
over to the failover server, it will start up with a fairly recent view 
of your system, including all of the info about what hosts have been
disabled, which alerts have been acknowledged etc. Synchronizing these
every 10-15 minutes will get you very close to a completely transparent
failover.


Regards,
Henrik

Michael,

I have not yet put 2 or more Hobbit servers in a redundant configuration. Although I have build a simular setting with 2 BB servers. Here are a few hints:

For (configuration) file syncing don't count on Hobbit, use other tools like rsync or even better cvs with pserver. Another option here is using clustering tools, have a look at http://tentakel.biskalar.de/similar for a list of handy distribution scripts or try tentakel self.
This is a nice way to push bb-hosts files to your clients too.

I'm sure you have allready made both servers a BBdisplay and BBnet in your bb-hosts file, and configured all your clients this way also so they report to both servers.
Make both of them BBpager too, just one with a filled hobbit-alert.cfg (master) and one with an empty hobbit-alert.cfg (slave). In the master hobbit-alert.cfg make a rule which runs a script when bbd is red or purple on your slave hobbitserver. This script has to login through ssh and copy the master hobbit-alert.cfg over your slave hobbit-alert.cfg.
Now your second hobbitserver is master (next to your first hobbitserver).

Good luck!
Arnoud Post

Lowery, Michael wrote: