Xymon Mailing List Archive search

All network tests now purple

11 messages in this thread

list Rich Smrcina · Thu, 26 Oct 2006 11:16:06 -0500 ·
We had a very strange problem happen this morning.  About two hours ago, all of the tests performed by the Hobbit server turned purple.  The clients are still reporting and they are (mostly) green.

I applied the all-in-one patch and still there is no joy.

I checked the archive for similar problems and don't see any messages in bb-network.log.  Also, there are no core files (in the directory where Hobbit starts, or /).

hobbitlaunch.log shows:

2006-10-26 10:54:17 hobbitlaunch starting
2006-10-26 10:54:17 Loading tasklist configuration from /home/hobbit/server/etc/hobbitlaunch.cfg
2006-10-26 10:54:17 Loading hostnames
2006-10-26 10:54:17 Loading saved state
2006-10-26 10:54:17 Setting up network listener on 0.0.0.0:1984
2006-10-26 10:54:17 Setting up signal handlers
2006-10-26 10:54:17 Setting up hobbitd channels
2006-10-26 10:54:17 Setting up logfiles

-- 
Rich Smrcina
VM Assist, Inc.
Phone: XXX-XXX-XXXX
Ans Service:  XXX-XXX-XXXX
user-61add9955ef9@xymon.invalid

Catch the WAVV!  http://www.wavv.org
WAVV 2007 - Green Bay, WI - May 18-22, 2007
list Rich Smrcina · Fri, 27 Oct 2006 11:17:01 -0500 ·
I setup bbtest-net with the --debug option and I see in the bbnetwork.log file the following:

2006-10-27 07:49:54 Got DNS result for host gogen01 : 172.17.1.246
2006-10-27 07:49:54 DNS lookup failed for netwvrdev - status Domain name not found (4)
2006-10-27 07:49:54 DNS lookup failed for tmpl - status Domain name not found (4)
2006-10-27 07:49:54 DNS lookup failed for gots02 - status Domain name not found (4)
2006-10-27 07:50:24 Got DNS result for host people3 : 172.17.1.243

All of the DNS 'Add...' messages and 'Got...' messages are issued with in 1 second (07:49:53-07:49:54) for all hosts, except for the one called people3 which comes back 30 seconds later.  The hosts with the 'DNS lookup failed...' messages are all configured as dialup.

After this there are no additional messages in the file.
quoted from Rich Smrcina
-- 
Rich Smrcina
VM Assist, Inc.
Phone: XXX-XXX-XXXX
Ans Service:  XXX-XXX-XXXX
user-61add9955ef9@xymon.invalid

Catch the WAVV!  http://www.wavv.org
WAVV 2007 - Green Bay, WI - May 18-22, 2007
list Greg L Hubbard · Fri, 27 Oct 2006 11:30:00 -0500 ·
Rich,

I had some similar problems some time back -- it appears that that the
network test module can stall out if you have DNS problems, even though
it is not SUPPOSED to do so.  You can verify by adding the names that
are not being found to the host table on the network test host and see
if the problem clears up.

GLH 
quoted from Rich Smrcina

-----Original Message-----
From: Rich Smrcina [mailto:user-cf452ff334e0@xymon.invalid] 
Sent: Friday, October 27, 2006 11:17 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] All network tests now purple

I setup bbtest-net with the --debug option and I see in the
bbnetwork.log file the following:

2006-10-27 07:49:54 Got DNS result for host gogen01 : 172.17.1.246
2006-10-27 07:49:54 DNS lookup failed for netwvrdev - status Domain name
not found (4)
2006-10-27 07:49:54 DNS lookup failed for tmpl - status Domain name not
found (4)
2006-10-27 07:49:54 DNS lookup failed for gots02 - status Domain name
not found (4)
2006-10-27 07:50:24 Got DNS result for host people3 : 172.17.1.243

All of the DNS 'Add...' messages and 'Got...' messages are issued with
in 1 second (07:49:53-07:49:54) for all hosts, except for the one called
people3 which comes back 30 seconds later.  The hosts with the 'DNS
lookup failed...' messages are all configured as dialup.

After this there are no additional messages in the file.
--
Rich Smrcina
VM Assist, Inc.
Phone: XXX-XXX-XXXX
Ans Service:  XXX-XXX-XXXX
user-61add9955ef9@xymon.invalid

Catch the WAVV!  http://www.wavv.org
WAVV 2007 - Green Bay, WI - May 18-22, 2007
list Rich Smrcina · Fri, 27 Oct 2006 12:03:10 -0500 ·
I commented out the three hosts and restarted Hobbit.  Still no network tests.  The debug log appears to stop in the same place.

I tried to run a couple of them manually using:

~/server/bin/bbcmd bbtest-net --debug people3

This host is good and the network tests for that host come out OK and appear on the web page.  So it appears that the process works (at least a host at a time).

When I run the command with one of the hosts that could not be found, I get this (a small subset of the actual messages produced, but they seem relevant):

2006-10-27 11:48:02 Opening file /home/hobbit/server/etc/bb-hosts
2006-10-27 11:48:02 Adding hostname 'tmpl' to resolver queue
2006-10-27 11:48:02 Processing 1 DNS lookups with ARES
2006-10-27 11:48:02 DNS lookup failed for tmpl - status Domain name not found (4)
2006-10-27 11:48:02 About to do 0 TCP tests running 246 in parallel
2006-10-27 11:48:02 TCP tests completed normally
2006-10-27 11:48:20 Sending results for service conn
2006-10-27 11:48:20 Adding to combo msg: status tmpl.conn clear <!-- [flags:orDAstLe] --> Fri Oct 27 11:48:02 2006 conn ok : Disabled (dialup host)
2006-10-27 11:48:20 Flushing combo message
       status tmpl.conn clear <!-- [flags:orDAstLe] --> Fri Oct 27 11:48:02 2006 conn ok : Disabled (dialup host)

There's an 18 second pause after it indicates that it couldn't find the host and after the tests have completed.  It does send the correct status, it is a dialup host that could not be contacted and send a clear.
quoted from Greg L Hubbard

Hubbard, Greg L wrote:
Rich,

I had some similar problems some time back -- it appears that that the
network test module can stall out if you have DNS problems, even though
it is not SUPPOSED to do so.  You can verify by adding the names that
are not being found to the host table on the network test host and see
if the problem clears up.

GLH 
-----Original Message-----
From: Rich Smrcina [mailto:user-cf452ff334e0@xymon.invalid] Sent: Friday, October 27, 2006 11:17 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] All network tests now purple

I setup bbtest-net with the --debug option and I see in the
bbnetwork.log file the following:

2006-10-27 07:49:54 Got DNS result for host gogen01 : 172.17.1.246
2006-10-27 07:49:54 DNS lookup failed for netwvrdev - status Domain name
not found (4)
2006-10-27 07:49:54 DNS lookup failed for tmpl - status Domain name not
found (4)
2006-10-27 07:49:54 DNS lookup failed for gots02 - status Domain name
not found (4)
2006-10-27 07:50:24 Got DNS result for host people3 : 172.17.1.243

All of the DNS 'Add...' messages and 'Got...' messages are issued with
in 1 second (07:49:53-07:49:54) for all hosts, except for the one called
people3 which comes back 30 seconds later.  The hosts with the 'DNS
lookup failed...' messages are all configured as dialup.

After this there are no additional messages in the file.
--
Rich Smrcina
VM Assist, Inc.
Phone: XXX-XXX-XXXX
Ans Service:  XXX-XXX-XXXX
user-61add9955ef9@xymon.invalid

Catch the WAVV!  http://www.wavv.org
WAVV 2007 - Green Bay, WI - May 18-22, 2007

-- 
Rich Smrcina
VM Assist, Inc.
Phone: XXX-XXX-XXXX
Ans Service:  XXX-XXX-XXXX
user-61add9955ef9@xymon.invalid

Catch the WAVV!  http://www.wavv.org
WAVV 2007 - Green Bay, WI - May 18-22, 2007
list T.J. Yang · Fri, 27 Oct 2006 15:08:29 -0500 ·
BSD is more free than GPL (you don't have to give it back if you don't want to).
BSD Hobbit will attract more resources in  companies to participate the Hobbit project.


T.J. Yang

Stay in touch with old friends and meet new ones with Windows Live Spaces http://clk.atdmt.com/MSN/go/msnnkwsp0070000001msn/direct/01/?href=http://spaces.live.com/spacesapi.aspx?wx_action=create&wx_url=/friends.aspx&mkt=en-us
list Henrik Størner · Fri, 27 Oct 2006 22:35:52 +0200 ·
quoted from T.J. Yang
On Fri, Oct 27, 2006 at 03:08:29PM -0500, T.J. Yang wrote:
BSD is more free than GPL (you don't have to give it back if you don't want 
to).
Which is precisely why Hobbit uses GPL instead of a BSD license.


Regards,
Henrik
list Henrik Størner · Fri, 27 Oct 2006 22:42:34 +0200 ·
quoted from Rich Smrcina
On Fri, Oct 27, 2006 at 12:03:10PM -0500, Rich Smrcina wrote:
I commented out the three hosts and restarted Hobbit.  Still no network tests.  The debug log appears to stop in the same place.

I tried to run a couple of them manually using:

~/server/bin/bbcmd bbtest-net --debug people3

This host is good and the network tests for that host come out OK and appear on the web page.  So it appears that the process works (at least a host at a time).
Good.
quoted from Rich Smrcina
2006-10-27 11:48:02 Adding hostname 'tmpl' to resolver queue
2006-10-27 11:48:02 Processing 1 DNS lookups with ARES
2006-10-27 11:48:02 DNS lookup failed for tmpl - status Domain name not found (4)
2006-10-27 11:48:02 About to do 0 TCP tests running 246 in parallel
2006-10-27 11:48:02 TCP tests completed normally
2006-10-27 11:48:20 Sending results for service conn
2006-10-27 11:48:20 Adding to combo msg: status tmpl.conn clear <!-- [flags:orDAstLe] --> Fri Oct 27 11:48:02 2006 conn ok : Disabled (dialup host)
2006-10-27 11:48:20 Flushing combo message
      status tmpl.conn clear <!-- [flags:orDAstLe] --> Fri Oct 27 11:48:02 2006 conn ok : Disabled (dialup host)

There's an 18 second pause after it indicates that it couldn't find the host and after the tests have completed.  
That is while it waits for the ping tests to complete (hobbit runs the
hobbitping or fping tool in parallel with doing the other network tests).


Henrik
list Larry Barber · Fri, 27 Oct 2006 17:41:09 -0500 ·
With the GPL you don't have to "give back" changes unless you transfer the
program to another company or individual. The GPL just requires that the
code be available wherever the binaries are. You are perfectly free, under
the GPL, to change anything you want for your own use, and not give anybody
else the changes.

Thanks,
Larry Barber
quoted from Henrik Størner

On 10/27/06, Henrik Stoerner <user-ce4a2c883f75@xymon.invalid> wrote:
On Fri, Oct 27, 2006 at 03:08:29PM -0500, T.J. Yang wrote:
BSD is more free than GPL (you don't have to give it back if you don't
want
to).
Which is precisely why Hobbit uses GPL instead of a BSD license.


Regards,
Henrik

list Henrik Størner · Sat, 28 Oct 2006 09:36:11 +0200 ·
quoted from Larry Barber
On Fri, Oct 27, 2006 at 05:41:09PM -0500, Larry Barber wrote:
With the GPL you don't have to "give back" changes unless you transfer the
program to another company or individual. The GPL just requires that the
code be available wherever the binaries are. You are perfectly free, under
the GPL, to change anything you want for your own use, and not give anybody
else the changes.
You're right, of course - but I don't see that as a problem. What you do 
internally for your own benefit is not something I need to be involved in.
(Of course, if you do something really neat and generally useful, I would 
appreciate it if you would share it with the rest of us).

The problem with the BSD license from my point of view is that it is
unfair - you can take my code and benefit from it (eg sell it), but 
I cannot get your improvements and use them to benefit the whole
project, or even just my own use of Hobbit. It's an anti-social 
license, and I really don't see any reason why it would be beneficial
to change license.

(The "more companies would participate if it were BSD licensed" argument
is rubbish. There are lots of examples of companies contributing code to
GPL-licensed projects, simply because it is in their best interest to
have an active community working on the software that they use - and
even rely upon for their business).


Regards,
Henrik
list Jason Altrincham Jones · Mon, 30 Oct 2006 09:40:38 -0000 ·
I can second that, the only reason why my company started with hobbit at
all was the community, the fact that it was GPL vs. BSD never entered
the argument and now we have 9 sites monitored and soon will have a
further 5 in western Europe alone along with x in Russia, America, Japan
etc. etc. all in the hope of eventually getting one server that hosts
them all through bb proxy (but that is still a ways off yet :) ).
Although I do admit that I have not really contributed much myself :(
mainly because I am still learning but am getting better...I think so
anyway :)

But if there is anything I feel worth sharing do I just put it in the
/incoming directory?

Jason.
quoted from Henrik Størner
-----Original Message-----
From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid] 
Sent: 28 October 2006 08:36
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Just a suggestion : License change from GPL to BSD

On Fri, Oct 27, 2006 at 05:41:09PM -0500, Larry Barber wrote:
With the GPL you don't have to "give back" changes unless you transfer
the
program to another company or individual. The GPL just requires that
the
code be available wherever the binaries are. You are perfectly free,
under
the GPL, to change anything you want for your own use, and not give
anybody
else the changes.
You're right, of course - but I don't see that as a problem. What you do

internally for your own benefit is not something I need to be involved
in.
(Of course, if you do something really neat and generally useful, I
would 
appreciate it if you would share it with the rest of us).

The problem with the BSD license from my point of view is that it is
unfair - you can take my code and benefit from it (eg sell it), but 
I cannot get your improvements and use them to benefit the whole
project, or even just my own use of Hobbit. It's an anti-social 
license, and I really don't see any reason why it would be beneficial
to change license.

(The "more companies would participate if it were BSD licensed" argument
is rubbish. There are lots of examples of companies contributing code to
GPL-licensed projects, simply because it is in their best interest to
have an active community working on the software that they use - and
even rely upon for their business).


Regards,
Henrik
list Henrik Størner · Mon, 30 Oct 2006 11:06:57 +0100 ·
quoted from Jason Altrincham Jones
On Mon, Oct 30, 2006 at 09:40:38AM -0000, Jones, Jason (Altrincham) wrote:
But if there is anything I feel worth sharing do I just put it in the
/incoming directory?
It's easier for me if you could put it into "The Shire" project on
Sourceforge. 

Galen Johnson runs "The Shire" - Galen, could you point us to a brief
description of how to go about adding stuff ?


Regards,
Henrik