bbtest-net output & poll time
list Eric E *hs Schwimmer
Hi all, got another few posers: I have a bb-hosts file with 1229 hosts in it, but no TCP (http/ssh/ftp) enabled. In my hobbitlaunch.cfg, I have bbnet configured as follows: CMD bbtest-net --report --ping --checkresponse --dns=ip --concurrency=1500 --dns-timeout=5 --timeout=5 The bbtest output for my hobbit server shows the following snippets: <snip> DNS statistics: # hostnames resolved : 1220 # succesful : 25 # failed : 975 # calls to dnsresolve : 0 </snip> <snip> TIME SPENT Event Starttime Duration bbtest-net startup 1112644638.542264 - Service definitions loaded 1112644638.545180 0.002916 Tests loaded 1112644659.079652 20.534472 DNS lookups completed 1112644659.080039 0.000387 Test engine setup completed 1112644659.089031 0.008992 TCP tests completed 1112644659.089883 0.000852 PING test completed (1229 hosts) 1112644712.816387 53.726504 PING test results sent 1112644713.047301 0.230914 Test result collection completed 1112644713.047324 0.000023 LDAP test engine setup completed 1112644713.047339 0.000015 LDAP tests executed 1112644713.047369 0.000030 LDAP tests result collection completed 1112644713.047384 0.000015 Test results transmitted 1112644713.101506 0.054122 bbtest-net completed 1112644713.104883 0.003377 TIME TOTAL 74.562619 </snip> I thought that the --dns=ip option in the hobbitlaunch.cfg precluded any DNS resolution, but the bbtest output above seems to indicate the opposite. Does anybody know what those DNS tests are the result of? Also, I'm a bit curious as to the chronlogical breakdown of the bbtest events; specifically the "Tests Loaded" section. Does 20 seconds seem reasonable for a bb-hosts file our our size? (I'm trying desperately to get below the 60 second mark :) TIA, -Eric Network Engineer University of Virginia HSCS
list Henrik Størner
▸
On Mon, Apr 04, 2005 at 04:09:34PM -0400, Schwimmer, Eric E *HS wrote:
I have a bb-hosts file with 1229 hosts in it, but no TCP (http/ssh/ftp) enabled. In my hobbitlaunch.cfg, I have bbnet configured as follows: CMD bbtest-net --report --ping --checkresponse --dns=ip --concurrency=1500 --dns-timeout=5 --timeout=5 The bbtest output for my hobbit server shows the following snippets: DNS statistics: # hostnames resolved : 1220 # succesful : 25 # failed : 975 # calls to dnsresolve : 0
I thought that the --dns=ip option in the hobbitlaunch.cfg precluded any DNS resolution, but the bbtest output above seems to indicate the opposite. Does anybody know what those DNS tests are the result of?
The heading for that bit of statistics is slightly misleading. It really is just a summary of how many hostnames were converted into IP-adresses; this can happen via DNS, but when you use --dns=ip it is done entirely by using the IP's in the bb-hosts file. (The clue here is that the "# calls to dnsresolve" is 0). As your timing statistics show:
DNS lookups completed 1112644659.080039 0.000387
those 1000 IP's were found in less than one millisecond - that would be quite a feat if any DNS was involved.
▸
Also, I'm a bit curious as to the chronlogical breakdown of the bbtest events;
Service definitions loaded 1112644638.545180 0.002916 Tests loaded 1112644659.079652 20.534472
specifically the "Tests Loaded" section. Does 20 seconds seem reasonable for a bb-hosts file our our size?
No, it seems a bit much. My main server has about 1500 hosts in it, and spends 4 seconds loading that file (actually, it is split into about 30 files). If you look at the "bbgen" status, what's the time reported for the "Load bbhosts done" line ? I've looked over the code, but can't really spot anything that would explain why it takes so long.
(I'm trying desperately to get below the 60 second mark :)
Well, in that case perhaps you should lower the timeout on your ping-tests - they account for 70% of the total time:
PING test completed (1229 hosts) 1112644712.816387 53.726504 TIME TOTAL 74.562619
The ping tests are performed using "fping", so you may want to try and play with fping options to control the timeout and # of retries it does. You can add those to the FPING setting in hobbitserver.cfg. Any particular reason you want to get below 60 seconds ? Are you aware that Hobbit has a "re-test" script that performs more frequent tests on hosts that go down ? When a network test begins to fail, Hobbit puts that host on "frequent-test" list meaning that for up to 30 minutes that host will be tested once a minute rather than once every 5 minutes; so recoveries should be picked up faster. Regards, Henrik
list Eric E *hs Schwimmer
▸
those 1000 IP's were found in less than one millisecond - that would be quite a feat if any DNS was involved.
Thanks. I was seeing two conflicting data points and didn't know which to believe.
▸
specifically the "Tests Loaded" section. Does 20 seconds seem reasonable for a bb-hosts file our our size?No, it seems a bit much. My main server has about 1500 hosts in it, and spends 4 seconds loading that file (actually, it is split into about 30 files). If you look at the "bbgen" status, what's the time reported for the "Load bbhosts done" line ?
From bbgen: Load bbhosts done 1112648694.989501 0.178303 I'll try and fiddle with the format of our bb-hosts file and see if I can get anything to float to the surface.
▸
Any particular reason you want to get below 60 seconds ? Are you aware that Hobbit has a "re-test" script that performs more frequent tests on hosts that go down ? When a network test begins to fail, Hobbit puts that host on "frequent-test" list meaning that for up to 30 minutes that host will be tested once a minute rather than once every 5 minutes; so recoveries should be picked up faster.
We'll, the opinion from "on high" here is that the sooner we know something is having problems, the sooner we can fix it. Thats the main reason. I guess the second reason would be bragging rights ;) Thanks! -Eric
list Henrik Størner
▸
On Mon, Apr 04, 2005 at 05:09:47PM -0400, Schwimmer, Eric E *HS wrote:
specifically the "Tests Loaded" section. Does 20 seconds seem reasonable for a bb-hosts file our our size?No, it seems a bit much. My main server has about 1500 hosts in it, and spends 4 seconds loading that file (actually, it is split into about 30 files). If you look at the "bbgen" status, what's the time reported for the "Load bbhosts done" line ?From bbgen:Load bbhosts done 1112648694.989501 0.178303
OK, that's interesting - because it means that the code that does the basic parsing of the file is pretty fast, and the code that handles the network test setup is awfullly slow. I hadn't noticed that before, but I looked at my own servers now and there is a difference between the time for loading the bb-hosts file for just bbgen and for bbtest-net (1 second vs. 4 seconds in my case). I wonder why ... it something I'll need to look at one day. Henrik
list Olivier Beau
Hi, my test hobbit server has 1500 entry in bbhost (only doing network test for now) and bbtest-net's report says for me : Tests loaded 1112649412.475584 56.090490 that seems awfully high compared to henrik's 4 seconds.. what do you guys have on your servers ? Olivier
▸
Also, I'm a bit curious as to the chronlogical breakdown of the bbtest events;Service definitions loaded 1112644638.545180 0.002916 > Tests loaded 1112644659.079652 20.534472 specifically the "Tests Loaded" section. Does 20 seconds seem reasonable for a bb-hosts file our our size?No, it seems a bit much. My main server has about 1500 hosts in it, and spends 4 seconds loading that file (actually, it is split into about 30 files). If you look at the "bbgen" status, what's the time reported for the "Load bbhosts done" line ? I've looked over the code, but can't really spot anything that would explain why it takes so long.
list Olivier Beau
oops.. havent seen that on my hobbit bbgen report : Load bbhosts done 1112650273.999850 0.659456 versus 56s in bbtest's report.. -- olivier Selon Henrik Stoerner <user-ce4a2c883f75@xymon.invalid>:
▸
On Mon, Apr 04, 2005 at 05:09:47PM -0400, Schwimmer, Eric E *HS wrote:specifically the "Tests Loaded" section. Does 20 seconds seem reasonable for a bb-hosts file our our size?No, it seems a bit much. My main server has about 1500 hosts in it, and spends 4 seconds loading that file (actually, it is split into about 30 files). If you look at the "bbgen" status, what's the time reported for the "Load bbhosts done" line ?From bbgen:Load bbhosts done 1112648694.989501 0.178303OK, that's interesting - because it means that the code that does the basic parsing of the file is pretty fast, and the code that handles the network test setup is awfullly slow. I hadn't noticed that before, but I looked at my own servers now and there is a difference between the time for loading the bb-hosts file for just bbgen and for bbtest-net (1 second vs. 4 seconds in my
case). I wlist Daniel J McDonald
▸
On Mon, 2005-04-04 at 23:29 +0200, user-fe6e0e6a0d05@xymon.invalid wrote:
Hi, my test hobbit server has 1500 entry in bbhost (only doing network test for now) and bbtest-net's report says for me : Tests loaded 1112649412.475584 56.090490 that seems awfully high compared to henrik's 4 seconds.. what do you guys have on your servers ?
311 devices, about 1900 entries, most from bbmrtg.pl... TIME SPENT Event Starttime Duration bbtest-net startup 1112650367.265647 - Service definitions loaded 1112650367.267585 0.001938 Tests loaded 1112650367.290754 0.023169 DNS lookups completed 1112650372.316388 5.025634 Test engine setup completed 1112650372.320199 0.003811 TCP tests completed 1112650382.998951 10.678752 PING test completed (295 hosts) 1112650384.326074 1.327123 PING test results sent 1112650384.340618 0.014544 Test result collection completed 1112650384.340642 0.000024 LDAP test engine setup completed 1112650384.340648 0.000006 LDAP tests executed 1112650384.340663 0.000015 LDAP tests result collection completed 1112650384.340666 0.000003 NTP tests executed 1112650403.296982 18.956316 DIG tests executed 1112650403.304854 0.007872 Test results transmitted 1112650403.316908 0.012054 bbtest-net completed 1112650403.324258 0.007350 TIME TOTAL 36.058611 Oh, and hobbit 4.0.1 compiles and runs just fine on Mandrake 2005LE. RPM built nicely, et alia. The only thing I had to do was change the umask in ~hobbit/.bash_profile to 022, since running Mandrake in Paranoid MSEC mode sets the umask to 077, which means that apache can't read the bb.html page... -- Daniel J McDonald, CCIE # 2495, CNX Austin Energy user-290ce4e24e19@xymon.invalid
list Eric E *hs Schwimmer
Running hobbit on a dedicated server: Dual Xeon 1GHz Compaq ML350, 1.2GB RAM, ultra3 scsi raid5. This is just a temp setup; once we have all the kinks iron out, we'll replace BB on our real test server, which is a little more beefy. I just now tried editing our bb-hosts file to be completely flat (no pages/subpages) and removed everything after the hostnames, but it didn't affect the load time at all. Changing the FPING options in my hobbitserver.cfg to "-i15 -r2 -b12" seems to have helped my ping time quit a bit, though (thanks again, Henrik!) -Eric -----Original Message----- From: user-fe6e0e6a0d05@xymon.invalid [mailto:user-fe6e0e6a0d05@xymon.invalid] Sent: Mon 4/4/2005 5:29 PM To: user-ae9b8668bcde@xymon.invalid Cc: Subject: Re: [hobbit] bbtest-net output & poll time
▸
Hi,
my test hobbit server has 1500 entry in bbhost (only doing network test for now)
and bbtest-net's report says for me :
Tests loaded 1112649412.475584 56.090490
that seems awfully high compared to henrik's 4 seconds..
what do you guys have on your servers ?
Olivier
Also, I'm a bit curious as to the chronlogical breakdown of the bbtest events;Service definitions loaded 1112644638.545180 0.002916 Tests loaded 1112644659.079652 20.534472specifically the "Tests Loaded" section. Does 20 seconds seem reasonable for a bb-hosts file our our size?No, it seems a bit much. My main server has about 1500 hosts in it, and spends 4 seconds loading that file (actually, it is split into about 30 files). If you look at the "bbgen" status, what's the time reported for the "Load bbhosts done" line ? I've looked over the code, but can't really spot anything that would explain why it takes so long.
list Henrik Størner
▸
On Mon, Apr 04, 2005 at 05:09:47PM -0400, Schwimmer, Eric E *HS wrote:
No, it seems a bit much. My main server has about 1500 hosts in it, and spends 4 seconds loading that file (actually, it is split into about 30 files). If you look at the "bbgen" status, what's the time reported for the "Load bbhosts done" line ?From bbgen:Load bbhosts done 1112648694.989501 0.178303
I tried playing around with this, and there is a bug. It adds the hostnames to the DNS resolver queue, and since you have more than 500 hosts the DNS resolver does kick in and tries to do DNS lookups of those hostnames. I think this little patch should do wonders for your runtime. Regards, Henrik -------------- next part -------------- --- bbnet/bbtest-net.c 2005/04/03 15:39:02 1.210 +++ bbnet/bbtest-net.c 2005/04/04 21:48:08 @@ -811,7 +811,7 @@ } strcpy(h->ip, bbh_item(hwalk, BBH_IP)); - if (!h->testip) add_host_to_dns_queue(h->hostname); + if (!h->testip && (dnsmethod != IP_ONLY)) add_host_to_dns_queue(h->hostname); h->next = testhosthead; testhosthead = h; }
list Terry Barnes
I have 103 hosts - 362 tests - my timing - just numbers with significant time. Tests loaded 1112651231.141276 0.046555 TCP tests completed 1112651236.224276 5.049537 PING test completed (103 hosts) 1112651237.102731 0.878455 bbtest-net completed 1112651238.206582 0.909244 TIME TOTAL 7.116877 Terry Barnes Siemens Com @ HFHS XXX-XXX-XXXX (Office) XXX-XXX-XXXX (Cellular) XXX-XXX-XXXX (Fax) user-34ea5ff61ded@xymon.invalid (Text Pager) user-0e29285d9a67@xymon.invalid
user-fe6e0e6a0d05@xymon.invalid 4/4/05 5:29:24 PM >>>
▸
Hi, my test hobbit server has 1500 entry in bbhost (only doing network test for now) and bbtest-net's report says for me : Tests loaded 1112649412.475584 56.090490 that seems awfully high compared to henrik's 4 seconds.. what do you guys have on your servers ? Olivier
Also, I'm a bit curious as to the chronlogical breakdown of the bbtest events;Service definitions loaded 1112644638.545180 0.002916
Tests loaded 1112644659.079652 20.534472
specifically the "Tests Loaded" section. Does 20 seconds seem reasonable for a bb-hosts file our our size?No, it seems a bit much. My main server has about 1500 hosts in it, and spends 4 seconds loading that file (actually, it is split into about 30 files). If you look at the "bbgen" status, what's the time reported for the "Load bbhosts done" line ? I've looked over the code, but can't really spot anything that would explain why it takes so long.
==============================================================================
HFHS CONFIDENTIALITY NOTICE: This email contains information from the sender that may be CONFIDENTIAL, LEGALLY PRIVILEGED, PROPRIETARY or otherwise protected from disclosure. This email is intended for use only by the person or entity to whom it is addressed. If you are not the intended recipient, any use, disclosure, copying, distribution, printing, or any action taken in reliance on the contents of this email, is strictly prohibited. If you received this email in error, please contact the sending party by replying in an email to the sender, delete the email from your computer system and shred any paper copies of the email you printed. Note to Patients: There are a number of risks you should consider before using e-mail to communicate with us. These risks are described in our Privacy Policy at http://henryford.com. Review that policy carefully before continuing to communicate with us by e-mail. For greater Internet security, our policy describes the Henry Ford MyHealth electronic communication process - you may register at http://henryford.com. If you do not believe that our policy gives you the privacy and security protection you need, do not send e-mail or Internet communications to us. ==============================================================================
list Eric E *hs Schwimmer
Ahhh, wonders achieved!! TIME SPENT Event Starttime Duration bbtest-net startup 1112653079.892656 - Service definitions loaded 1112653079.895452 0.002796 Tests loaded 1112653080.000170 0.104718 DNS lookups completed 1112653080.000448 0.000278 Test engine setup completed 1112653080.010469 0.010021 TCP tests completed 1112653080.011328 0.000859 PING test completed (1229 hosts) 1112653119.138755 39.127427 PING test results sent 1112653119.383178 0.244423 Test result collection completed 1112653119.383202 0.000024 LDAP test engine setup completed 1112653119.383217 0.000015 LDAP tests executed 1112653119.383247 0.000030 LDAP tests result collection completed 1112653119.383262 0.000015 Test results transmitted 1112653119.432869 0.049607 bbtest-net completed 1112653119.436212 0.003343 TIME TOTAL 39.543556 Thanks Henrik, you rock!! -Eric
▸
-----Original Message-----
From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: Mon 4/4/2005 5:50 PM
To: user-ae9b8668bcde@xymon.invalid
Cc:
Subject: Re: [hobbit] bbtest-net output & poll time
On Mon, Apr 04, 2005 at 05:09:47PM -0400, Schwimmer, Eric E *HS wrote:No, it seems a bit much. My main server has about 1500 hosts in it, and spends 4 seconds loading that file (actually, it is split into about 30 files). If you look at the "bbgen" status, what's the time reported for the "Load bbhosts done" line ?From bbgen:Load bbhosts done 1112648694.989501 0.178303
I tried playing around with this, and there is a bug. It adds the hostnames to the DNS resolver queue, and since you have more than 500 hosts the DNS resolver does kick in and tries to do DNS lookups of those hostnames. I think this little patch should do wonders for your runtime. Regards, Henrik