Josh,
Thanks for help, AGAIN.... One step closer...
I have one host down, and I have the trace option on all of my hosts listed in bb-hosts. When I comment out that downed host, the errors clear up in bb-test. Take a look...
Mon Dec 31 12:22:16 2007
bbtest-net version 4.2.0
SSL library : OpenSSL 0.9.7m 23 Feb 2007
LDAP library: OpenLDAP 20213
Statistics:
Hosts total : 310
Hosts with no tests : 7
Total test count : 307
Status messages : 308
Alert status msgs : 0
Transmissions : 5
DNS statistics:
# hostnames resolved : 303
# succesful : 303
# failed : 0
# calls to dnsresolve : 307
TCP test statistics:
# TCP tests total : 2
# HTTP tests : 1
# Simple TCP tests : 1
# Connection attempts : 2
# bytes written : 135
# bytes read : 553
TIME SPENT
Event Starttime Duration
bbtest-net startup 1199103736.384784 -
Service definitions loaded 1199103736.385887 0.001103 Tests loaded 1199103736.768919 0.383032 DNS lookups completed 1199103736.768928 0.000009 Test engine setup completed 1199103736.772261 0.003333 TCP tests completed 1199103736.773300 0.001039 PING test completed (303 hosts) 1199103755.089536 18.316236 PING test results sent 1199103755.091233 0.001697 Test result collection completed 1199103755.091241 0.000008 LDAP test engine setup completed 1199103755.091245 0.000004 LDAP tests executed 1199103755.091249 0.000004 LDAP tests result collection completed 1199103755.091252 0.000003 NSLOOKUP tests executed 1199103755.095923 0.004671 Test results transmitted 1199103755.098103 0.002180 bbtest-net completed 1199103755.099180 0.001077 TIME TOTAL 18.714396
But once I uncomment out the host and the hobbit server tries to do a traceroute to it, the errors come back again. Even if I disable the alerting of that host. Take a look....
Mon Dec 31 12:32:24 2007
bbtest-net version 4.2.0
SSL library : OpenSSL 0.9.7m 23 Feb 2007
LDAP library: OpenLDAP 20213
Statistics:
Hosts total : 311
Hosts with no tests : 7
Total test count : 308
Status messages : 309
Alert status msgs : 0
Transmissions : 5
DNS statistics:
# hostnames resolved : 304
# succesful : 304
# failed : 0
# calls to dnsresolve : 308
TCP test statistics:
# TCP tests total : 2
# HTTP tests : 1
# Simple TCP tests : 1
# Connection attempts : 2
# bytes written : 135
# bytes read : 553
Error output:
Timeout waiting for data from child, killing it
Timeout waiting for data from child, killing it
Child process terminated with signal 15
TIME SPENT
Event Starttime Duration
bbtest-net startup 1199104344.425092 -
Service definitions loaded 1199104344.426152 0.001060 Tests loaded 1199104344.543955 0.117803 DNS lookups completed 1199104344.543964 0.000009 Test engine setup completed 1199104344.547454 0.003490 TCP tests completed 1199104344.548434 0.000980 PING test completed (304 hosts) 1199104369.082520 24.534086 PING test results sent 1199104399.089988 30.007468 Test result collection completed 1199104399.090003 0.000015 LDAP test engine setup completed 1199104399.090007 0.000004 LDAP tests executed 1199104399.090011 0.000004 LDAP tests result collection completed 1199104399.090015 0.000004 NSLOOKUP tests executed 1199104399.095563 0.005548 Test results transmitted 1199104399.097862 0.002299 bbtest-net completed 1199104399.098975 0.001113 TIME TOTAL 54.673883
Any ideas of why its doing it??? Or how to resolve it???
Thanks, michael
From: Josh Luthman [mailto:user-4c45a83f15cb@xymon.invalid] Sent: Friday, December 28, 2007 5:30 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] bbtest - errors
Try Henrik's fping command at the bottom of this page:
http://www.hswn.dk/hobbiton/2007/11/msg00069.html
and stick a time in front to see how long it takes.
On 12/28/07, Michael A. Price <user-d7d653acf808@xymon.invalid> wrote: Josh,
Thanks for getting back to me so quickly, I updated my /etc/hosts file to have every single one of my monitored hosts, just as a test. I now have 'failed hosts' in my DNS statistic's, but my 'PING test results sent' are still off the charts. I still cant figure out the problem...
bbtest-net version 4.2.0
SSL library : OpenSSL 0.9.7m 23 Feb 2007
LDAP library: OpenLDAP 20213
Statistics:
Hosts total : 311
Hosts with no tests : 7
Total test count : 308
Status messages : 309
Alert status msgs : 0
Transmissions : 5
DNS statistics:
# hostnames resolved : 304
# succesful : 304
# failed : 0
# calls to dnsresolve : 308
TCP test statistics:
# TCP tests total : 2
# HTTP tests : 1
# Simple TCP tests : 1
# Connection attempts : 2
# bytes written : 135
# bytes read : 553
Error output:
Timeout waiting for data from child, killing it
Timeout waiting for data from child, killing it
Child process terminated with signal 15
TIME SPENT
Event Starttime Duration
bbtest-net startup 1198875012.330887 -
Service definitions loaded 1198875012.331984 0.001097 Tests loaded 1198875012.405015 0.073031 DNS lookups completed 1198875012.405024 0.000009
Test engine setup completed 1198875012.408543 0.003519 TCP tests completed 1198875012.409325
0.000782 PING test completed (304 hosts) 1198875037.083126 24.673801
PING test results sent 1198875067.092719 30.009593 Test result collection completed 1198875067.092733 0.000014 LDAP test engine setup completed 1198875067.092737 0.000004 LDAP tests executed 1198875067.092741 0.000004
LDAP tests result collection completed 1198875067.092745 0.000004 NSLOOKUP tests executed 1198875067.096007
0.003262 Test results transmitted 1198875067.098247 0.002240
bbtest-net completed 1198875067.099155 0.000908 TIME TOTAL 54.768268
Thanks, michael
From: Josh Luthman [mailto:user-4c45a83f15cb@xymon.invalid] Sent: Thursday, December 27, 2007 11:15 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] bbtest - errors
Michael,
Try adding "testip" after the comment in as many hosts as possible, IE:
10.0.0.250 myftp.server.com # testip
Josh On 12/27/07, Michael A. Price <user-d7d653acf808@xymon.invalid> wrote:
I just modified the /etc/nsswitch.conf file to remove DNS.
I find it interesting that no matter if the hobbit server uses DNS servers or local host files to look up the hosts the 'PING Test Results Sent' number is still off the charts.
Thanks so much for getting back to me
Thanks, michael
From: Josh Luthman [mailto: user-4c45a83f15cb@xymon.invalid] Sent: Wednesday, December 26, 2007 6:00 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] bbtest - errors
Your calls to dnsresolve went up one, how in the world did you "[update] the hobbit server to not use the DNS servers"?
It looks like it is still doing the exact same stuff concerning DNS to me... On 12/26/07, Michael A. Price <user-d7d653acf808@xymon.invalid> wrote:
Thanks for getting back to me on this.
I updated the hobbit server to not use the DNS servers and all that does is cause it to go from 100 failed hosts to 299 failed hosts.
I think it's the large "PING test results sent" number, what else could be the problem???
Here is another printout...
Thanks, michael
bbtest-net version 4.2.0
SSL library : OpenSSL 0.9.7m 23 Feb 2007
LDAP library: OpenLDAP 20213
Statistics:
Hosts total : 311
Hosts with no tests : 7
Total test count : 308
Status messages : 309
Alert status msgs : 0
Transmissions : 5
DNS statistics:
# hostnames resolved : 304
# succesful : 203
# failed : 101
# calls to dnsresolve : 308
TCP test statistics:
# TCP tests total : 2
# HTTP tests : 1
# Simple TCP tests : 1
# Connection attempts : 2
# bytes written : 135
# bytes read : 553
Error output:
Timeout waiting for data from child, killing it
Timeout waiting for data from child, killing it
Child process terminated with signal 15
TIME SPENT
Event Starttime Duration
bbtest-net startup
1198691205.281738 -
Service definitions loaded 1198691205.282850
0.001112
Tests loaded 1198691205.316420 0.033570
DNS lookups completed 1198691215.446830
10.130410 Test engine setup completed
1198691215.450594 0.003764 TCP tests completed 1198691215.451393 0.000799 PING test completed (304 hosts) 1198691240.081987 24.630594
PING test results sent 1198691270.090627 30.008640
Test result collection completed 1198691270.090642
0.000015
LDAP test engine setup completed 1198691270.090656 0.000014
LDAP tests executed 1198691270.090660 0.000004
LDAP tests result collection completed
1198691270.090663 0.000003
NSLOOKUP tests executed 1198691270.146990 0.056327
Test results transmitted 1198691270.149410 0.002420
bbtest-net completed 1198691270.150271 0.000861
TIME TOTAL
64.868533
From: Josh Luthman [mailto: user-4c45a83f15cb@xymon.invalid] Sent: Thursday, December 20, 2007 11:04 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] bbtest - errors
If that was the only change you made recently try switching the DNS servers back to see if the problem disappears.
On 12/20/07, Michael A. Price < user-d7d653acf808@xymon.invalid> wrote:
Thanks...
Actually, I updated my DNS servers and went from 300 failed lookups to 100. So I thought I was going to improve....
But it got worse!!!! Any other ideas???
Thanks, michael
From: Josh Luthman [mailto: user-4c45a83f15cb@xymon.invalid] Sent: Thursday, December 20, 2007 8:10 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] bbtest - errors
# failed : 100 <--- may be the cause, lots of failed DNS queries
On 12/19/07, Michael A. Price < user-d7d653acf808@xymon.invalid> wrote:
My bbtest time went from 10 seconds to 89.0 ....
Has anyone seen this before???
Wed Dec 19 19:15:55 2007
bbtest-net version 4.2.0
SSL library : OpenSSL 0.9.7m 23 Feb 2007
LDAP library: OpenLDAP 20213
Statistics:
Hosts total : 310 Hosts with no tests : 7
Total test count : 307
Status messages : 308
Alert status msgs : 0
Transmissions : 5
DNS statistics:
# hostnames resolved : 303 # succesful : 203
# failed : 100
# calls to dnsresolve : 307
TCP test statistics:
# TCP tests total : 2
# HTTP tests : 1
# Simple TCP tests : 1 # Connection attempts : 2
# bytes written : 135
# bytes read : 553
Error output:
Timeout waiting for data from child, killing it
Timeout waiting for data from child, killing it Child process terminated with signal 15
Timeout waiting for data from child, killing it
Timeout waiting for data from child, killing it
Child process terminated with signal 15
TIME SPENT
Event Starttime Duration
bbtest-net startup 1198091755.294810
• Service definitions loaded 1198091755.297812
0.003002
Tests loaded 1198091755.346908
0.049096 DNS lookups completed 1198091765.439050
10.092142
Test engine setup completed 1198091765.442685
0.003635
TCP tests completed 1198091765.443457
0.000772 PING test completed (303 hosts) 1198091790.084027
24.640570
PING test results sent 1198091850.102236
60.018209
Test result collection completed 1198091850.102455
0.000219 LDAP test engine setup completed 1198091850.102472
0.000017
LDAP tests executed 1198091850.102475
0.000003
LDAP tests result collection completed 1198091850.102482
0.000007 NSLOOKUP tests executed 1198091850.111523
0.009041
Test results transmitted 1198091850.118622
0.007099
bbtest-net completed 1198091850.120484
0.001862 TIME TOTAL
94.825674
Thanks, michael
--
Josh Luthman
Office: XXX-XXX-XXXX
Direct: XXX-XXX-XXXX
XXXX Wayne St
Suite XXXX
Troy, OH XXXXX
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer
--
Josh Luthman
Office: XXX-XXX-XXXX
Direct: XXX-XXX-XXXX
XXXX Wayne St
Suite XXXX
Troy, OH XXXXX
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer
--
Josh Luthman
Office: XXX-XXX-XXXX
Direct: XXX-XXX-XXXX
XXXX Wayne St
Suite XXXX
Troy, OH XXXXX
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer
--
Josh Luthman
Office: XXX-XXX-XXXX
Direct: XXX-XXX-XXXX
XXXX Wayne St
Suite XXXX
Troy, OH XXXXX
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer
--
Josh Luthman
Office: XXX-XXX-XXXX
Direct: XXX-XXX-XXXX
XXXX Wayne St
Suite XXXX
Troy, OH XXXXX
Those who don't understand UNIX are condemned to reinvent it, poorly. --- Henry Spencer