Remote hobbit server's ports & procs flapping Green/Clear on secondary BBDISPLAY 🔗 link

Per my email to the list last night to Neil entitled "Re: [hobbit] Securing
Xymon Over Internet"  I have some clients with their own local hobbit server
which I have configured to also send to my xymon server as a secondary BBDISPLAY.

I have noticed that two tests on one remote server continually go green and
then clear at the same time.   That is, just now at 08:03:08 procs shows that
it when from CLEAR to GREEN, then at the same time when BACK from GREEN to
CLEAR. This continues with every 5  minute cycle.

--[snip from the Current non-green Systems screen]--

Tue Feb 17 08:03:08 2009 	dns.myclient.org	ports	 G  C
Tue Feb 17 08:03:08 2009 	dns.myclient.org	procs	 G  C	
Tue Feb 17 08:03:08 2009 	dns.myclient.org	ports	 C  G 	
Tue Feb 17 08:03:08 2009 	dns.myclient.org	procs	 C  G

--[snip from the Current non-green Systems screen]--


At the client site, their hobbit server currently reports "No events received
in the last 240 minutes."

These two tests, ports and procs are actually ON my clients hobbit server. All
other ports/procs tests on their other machines report back to me fine and are
not flapping Green/Clear.

Where can I look to begin to determine what is causing this green/clear flapping?

THANKS!


--
Bill Arlofski
Reverse Polarity, LLC

On Tue, Feb 17, 2009 at 08:20:32AM -0500, Bill Arlofski wrote:

Check the BBDISP and BBDISPLAY settings in your hobbitserver.cfg. It
sounds as if you have BBDISP=0.0.0.0 and BBDISPLAYS="ip1 ip2" - that
causes hobbitd_client (the module analyzing client data) to send the
results to both of your Xymon servers, causing double updates for each
status. The reason the status is flapping between clear and green is
that your hobbit-clients.cfg is different on the two servers.

What happens is:

 - Client sends the client data to server1 and server2.
 - server1 passes the client data to the local hobbitd_client task.
 - server1's hobbitd_client analyses the data using the
   hobbit-clients.cfg found on server1. Then it generates status
   updates; these are sent to the servers listed in BBDISPLAYS,
   i.e. both server1 and server2.
 - server2 does exactly the same, and sends another set of
   status updates to both server1 and server2.


The solution is to make sure you have identical hobbit-clients.cfg
files on both servers, and to avoid updating everything twice you
should only list the local server in hobbitserver.cfg's BBDISP setting.
(But keep both servers in the hobbitclient.cfg's BBDISP / BBDISPLAYS 
setting).


Regards,
Henrik

Henrik Størner wrote:

Hi Henrik... Thanks for the fast and detailed reply.


Ahhh... OK.. I have now copied the hobbit-clients.cfg from my client's hobbit
server to my central xymon server's hobbit-clients.cfg file and the flapping
stopped.

However.. I still have a question based on your description:

At my client's site, their hobbit server monitors several other local servers
with the hobbit client installed (cpu, disk, procs etc) and other network
equipment (ping telnet, ssh etc)

Their server has my server and their server in the
~/server/etc/hobbitserver.cfg's BBDISPLAYS (with just 0.0.0.0 in BBDISP)

On my server I only have itself in the BBDISP, and BBDISPLAYS=""

I have the BBLOCATION set to their site name in their server and in front of
all the tests in bb-hosts I have # NET="TheirSiteName"   On my server, I have
# NET:"MySiteName" in front of all of my local tests.

Their server also runs some local client scripts to gather NUT (Network UPS
tools) data from their UPSes. So I found that I needed to make the same
changes to BBDISP and BBDISPLAYS their ~/client/etc/hobbit-clients.cfg file
otherwise MY Xymon server didn't get their UPS report data.

I think that this is what fixed it.

If I do as you explained then I get the UPS data from the client scripts on my
client's hobbit server, but all the other tests for my client's site go purple
on my xymon system...

So in the end, I found that I needed to make sure that BBDISP=0.0.0.0  and
BBDISPLAY="my.xymon.ip their.hobbit.ip"  on their hobbit server's
~/server/etc/hobbitserver.cfg and ~/client/etc/hobbitclient.cfg files

So currently, with things configured as described it does not appear that any
of their data is being sent twice to my xymon server, nothing is flapping, and
I am seeing the UPS data from their client scripts too.

So I think ALL is well now.  Thanks again for the quick reply!

--
Bill Arlofski
Reverse Polarity, LLC