Some thoughts on clustered hobbit 🔗 link

On Mon, May 09, 2005 at 04:19:43PM -0500, Kauffman, Tom wrote:

Henrik -- check me on this: it's my impression we no longer need a
'BBPAGER' entry on the client-side bb-hosts because the hobbit server
passes all potentially alertable statuses to hobbit-alert and it decides
if an alert is really required.

Brian -- no offense, but I would rather categorise your configuration as
"active/inactive". I'm looking at doing an "active/passive" cluster when
time frees up -- about a month from now. The difference? I'm running two
hobbit/apache instances all the time -- but the 'passive' (fallover)
side is not doing alerting or network tests. It does build displays
(it's my technical documentation server as well) and it does keep both
history and rrd data updated. Both hosts show up on the client side as
'BBDISPLAY'. On failover it will take over the IP address for the hobbit
display and re-launch hobbit with network testing and alerting enabled.

-----Original Message-----
From: Kauffman, Tom [mailto:user-3feba9e60a8b@xymon.invalid] 
Sent: Monday, May 09, 2005 5:20 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Some thoughts on clustered hobbit

First, let me express my thanks to Brian for putting this document
together and allowing Henrik to distribute it! I've a lot of experience
with IBM's HACMP for AIX, and getting a clustered configuration working
as desired is not a trivial procedure.

Henrik -- check me on this: it's my impression we no longer need a
'BBPAGER' entry on the client-side bb-hosts because the hobbit server
passes all potentially alertable statuses to hobbit-alert and it decides
if an alert is really required.

Brian -- no offense, but I would rather categorise your configuration as
"active/inactive". I'm looking at doing an "active/passive" cluster when
time frees up -- about a month from now. The difference? I'm running two
hobbit/apache instances all the time -- but the 'passive' (fallover)
side is not doing alerting or network tests. It does build displays
(it's my technical documentation server as well) and it does keep both
history and rrd data updated. Both hosts show up on the client side as
'BBDISPLAY'. On failover it will take over the IP address for the hobbit
display and re-launch hobbit with network testing and alerting enabled.

Depending on host count and test count, this might be a bad idea -- but
we've only got about 300 entries in bb-hosts. 

So -- thanks ever so much, again, for providing this -- it will make my
life ever so much easier next month when I get the time to automate the
failover environment.

Tom

Tom Kauffman
NIBCO, Inc

Hi All,

I am adding a custom graph to track Oracle tablespace usage.  I currently
have an "oracle" test which shows the overall green/yellow/red status of
tablespaces based on percent full thresholds.

I want to append graphs showing each tablespace's size/used stats for a
better view of historical growth, but I don't want the raw data showing
in
the HTML page.

I seem to recall using a <data> type tag in the message text but can't
find
a good example.  I'm comfortable with post-processing the message to
generate the actual RRDs and graphs but would appreciate some help on the
message side!

TIA,
Andy.

TIA,
Andy.

#####################################################################################

This email is intended for the person to whom it is addressed
only. If you are not the intended recipient, do not read, copy
or use the contents in any way. The opinions expressed may not
necessarily reflect those of ZESPRI Group of Companies ('ZESPRI').

While every effort has been made to verify the information
contained herein, ZESPRI does not make any representations 
as to the accuracy of the information or to the performance
of any data, information or the products mentioned herein.
ZESPRI will not accept liability for any losses, damage or
consequence, however, resulting directly or indirectly from
the use of this e-mail/attachments.
#####################################################################################

On Mon, May 09, 2005 at 08:41:37PM -0400, Jeffcoat, Al wrote:

Did I miss something?  I don't see anything regarding a clustered hobbit
setup in the last couple of days, but would be greatly interested in
clustering 2 AIX servers together across locations to handle hobbit.

On 5/9/05, Kauffman, Tom <user-3feba9e60a8b@xymon.invalid> wrote:

First, let me express my thanks to Brian for putting this document
together and allowing Henrik to distribute it! I've a lot of experience
with IBM's HACMP for AIX, and getting a clustered configuration working
as desired is not a trivial procedure.

Henrik -- check me on this: it's my impression we no longer need a
'BBPAGER' entry on the client-side bb-hosts because the hobbit server
passes all potentially alertable statuses to hobbit-alert and it decides
if an alert is really required.

Brian -- no offense, but I would rather categorise your configuration as
"active/inactive". I'm looking at doing an "active/passive" cluster when
time frees up -- about a month from now. The difference? I'm running two
hobbit/apache instances all the time -- but the 'passive' (fallover)
side is not doing alerting or network tests. It does build displays
(it's my technical documentation server as well) and it does keep both
history and rrd data updated. Both hosts show up on the client side as
'BBDISPLAY'. On failover it will take over the IP address for the hobbit
display and re-launch hobbit with network testing and alerting enabled.

Depending on host count and test count, this might be a bad idea -- but

we've only got about 300 entries in bb-hosts.

So -- thanks ever so much, again, for providing this -- it will make my
life ever so much easier next month when I get the time to automate the
failover environment.

Tom

Tom Kauffman
NIBCO, Inc

On May 10, 2005, at 3:21 AM, Brian Lynch wrote:

On 5/9/05, Kauffman, Tom <user-3feba9e60a8b@xymon.invalid> wrote:
First, let me express my thanks to Brian for putting this document
together and allowing Henrik to distribute it! I've a lot of  experience
with IBM's HACMP for AIX, and getting a clustered configuration  working
as desired is not a trivial procedure.

Henrik -- check me on this: it's my impression we no longer need a
'BBPAGER' entry on the client-side bb-hosts because the hobbit server
passes all potentially alertable statuses to hobbit-alert and it  decides
if an alert is really required.

Brian -- no offense, but I would rather categorise your  configuration as
"active/inactive". I'm looking at doing an "active/passive" cluster  when
time frees up -- about a month from now. The difference? I'm  running two
hobbit/apache instances all the time -- but the 'passive' (fallover)
side is not doing alerting or network tests. It does build displays
(it's my technical documentation server as well) and it does keep both
history and rrd data updated. Both hosts show up on the client side as
'BBDISPLAY'. On failover it will take over the IP address for the  hobbit
display and re-launch hobbit with network testing and alerting  enabled.

I agree with your assessment, but chose the model for a few reasons
(note that I'm basing my experience on about 2 1/2 years running a  dual
big brother failover setup):

1. There is always one repository for both configuration and data  that are
kept reasonably identical on both systems (within the synch delay).
2. There is only one ip address accepting BB reports cutting down on
both network traffic and firewall rules (for hosts in locked down  vlans).
3. The other system can be dedicated to another purpose (it currently
hosts our documentation site that fails over in the opposite  direction).
4. No redundant work is done. Indeed, no load is being 'shared' across
the systems unless you host the web server on the other box.
 There is a risk to this based on the possibility of complete  machine failure
 in between synchronizations.  Hence, Hobbit may come up without all
 the updates for hosts or alerts.  Based on my current model, I  will lose
about a day of historical data.  These synch rates can be changed and
a gigabit crossover between machines cuts down on any traffic imposed
by multiple synch's.

Note that you could very easily turn off the hobbit alerts with the  same

Not sure how to disable the network tests, so that may require some
custom coding... Once complete, you could use the same cluster
resource sw to accomplish a 'hot' standby.

From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid] 
Sent: Tuesday, May 10, 2005 1:22 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Some thoughts on clustered hobbit

Brian Lynch wrote up a description of how to cluster two Hobbit
servers - it's in the "contrib" directory in the hobbit-4.0.3rc2
sources.

On Mon, May 09, 2005 at 08:41:37PM -0400, Jeffcoat, Al wrote:

Did I miss something?  I don't see anything regarding a clustered
hobbit
setup in the last couple of days, but would be greatly interested in
clustering 2 AIX servers together across locations to handle hobbit.

Henrik


This e-mail message and any attached files are confidential and are intended solely for the use of the addressee(s) named above. If you are not the intended recipient, any review, use, or distribution of this e-mail message and any attached files is strictly prohibited. This communication may contain material protected by Federal privacy regulations, attorney-client work product, or other privileges. If you have received this confidential communication in error, please notify the sender immediately by reply e-mail message and permanently delete the original message.  To reply to our email administrator directly, send an email to:  user-ecde3bbc361d@xymon.invalid .  If this e-mail message concerns a contract matter, be advised that no employee or agent is authorized to conclude any binding agreement on behalf of Orlando Regional Healthcare by e-mail without express written confirmation by an officer of the corporation. Any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Orlando Regional Healthcare.

 
-----Original Message-----
From: Brian Lynch [mailto:user-0420823115a8@xymon.invalid] 
Sent: Tuesday, May 10, 2005 2:22 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Some thoughts on clustered hobbit

 
I agree with your assessment, but chose the model for a few reasons
(note that I'm basing my experience on about 2 1/2 years running a dual
big brother failover setup):
 
1. There is always one repository for both configuration and data that
are
kept reasonably identical on both systems (within the synch delay). 
2. There is only one ip address accepting BB reports cutting down on 
both network traffic and firewall rules (for hosts in locked down
vlans). 
3. The other system can be dedicated to another purpose (it currently
hosts our documentation site that fails over in the opposite direction).
4. No redundant work is done. Indeed, no load is being 'shared' across
the systems unless you host the web server on the other box.
 There is a risk to this based on the possibility of complete machine
failure
 in between synchronizations.  Hence, Hobbit may come up without all
 the updates for hosts or alerts.  Based on my current model, I will
lose 
about a day of historical data.  These synch rates can be changed and 
a gigabit crossover between machines cuts down on any traffic imposed 
by multiple synch's. 

Note that you could very easily turn off the hobbit alerts with the same
clustering software by truncating and restoring the hobbit-alerts.cfg
file. 
Not sure how to disable the network tests, so that may require some
custom coding... Once complete, you could use the same cluster
resource sw to accomplish a 'hot' standby.