Which DNS query is causing delays
list Martin Ward
Hi all,
I have noticed recently that my bb-test runs are taking a long time to complete, 6500 seconds instead of the maximum of 600. The info on the bb-test results page shows that the DNS tests are taking most of the time, but I have around 200 of these, does anyone have any suggestions on how I can find out which DNS tests are causing the delays without having to click on every DNS test in all of my Xymon website?
Thanks,
|\/|artin
*************************************************************************************
The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way.
The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing user-61c7f445d564@xymon.invalid and delete the message and any attachments without retaining any copies.
Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses.
No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party.
Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.
list Henrik Størner
▸
In <user-f2650c92cc26@xymon.invalid> "Ward, Martin" <user-2d33a6eb6a05@xymon.invalid> writes:
I have noticed recently that my bb-test runs are taking a long time to complete, 6500 seconds instead of the maximum of 600. The info on the bb-test results page shows that the DNS tests are taking most of the time, but I have around 200 of these, does anyone have any suggestions on how I can find out which DNS tests are causing the delays without having to click on every DNS test in all of my Xymon website?
Are you using a local DNS cache ? I would strongly recommend that you do - the resolver library in Xymon is pretty aggressive, and has been known to knock out DNS servers if you have many hosts (although 200 doesn't sound like an awful lot). It shouldn't take that long unless you're using the standard resolver library (i.e. you are running bbtest-net with the "--no-ares" option), in which case Xymon cannot control the timeout for the DNS lookup. Try running with the option "--dnslog=FILENAME", this will log those DNS requests that fail. It is probably one of those. Regards, Henrik
list Martin Ward
Thanks Henrik, I am not using the --no-ares option so have put the dnslog entry in as you recommend. Thanks and I'll let you know how I get on. |\/|artin
▸
-----Original Message----- From: Henrik "Størner [mailto:user-ce4a2c883f75@xymon.invalid] Sent: 09 October 2009 12:26 To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] Which DNS query is causing delays In <user-f2650c92cc26@xymon.invalid> "Ward, Martin" <user-2d33a6eb6a05@xymon.invalid> writes:I have noticed recently that my bb-test runs are taking a long time to complete, 6500 seconds instead of the maximum of 600. The info on the bb-test results page shows that the DNS tests are taking most of the time, but I have around 200 of these, does anyone have any suggestions on how I can find out which DNS tests are causing the delays without having to click on every DNS test in all of my Xymon website?Are you using a local DNS cache ? I would strongly recommend that you do - the resolver library in Xymon is pretty aggressive, and has been known to knock out DNS servers if you have many hosts (although 200 doesn't sound like an awful lot). It shouldn't take that long unless you're using the standard resolver library (i.e. you are running bbtest-net with the "--no-ares" option), in which case Xymon cannot control the timeout for the DNS lookup. Try running with the option "--dnslog=FILENAME", this will log those DNS requests that fail. It is probably one of those. Regards, Henrik
*************************************************************************************
The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way.
The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing user-61c7f445d564@xymon.invalid and delete the message and any attachments without retaining any copies.
Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses.
No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party.
Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.
list Milan Kocian
▸
On Fri, Oct 09, 2009 at 11:26:23AM +0000, Henrik Størner wrote:
In <user-f2650c92cc26@xymon.invalid> "Ward, Martin" <user-2d33a6eb6a05@xymon.invalid> writes:I have noticed recently that my bb-test runs are taking a long time to >complete, 6500 seconds instead of the maximum of 600. The info on the >bb-test results page shows that the DNS tests are taking most of the time, >but I have around 200 of these, does anyone have any suggestions on how >I can find out which DNS tests are causing the delays without having to >click on every DNS test in all of my Xymon website?Are you using a local DNS cache ? I would strongly recommend that you do - the resolver library in Xymon is pretty aggressive, and has been known to knock out DNS servers if you have many hosts (although 200 doesn't sound like an awful lot). It shouldn't take that long unless you're using the standard resolver library (i.e. you are running bbtest-net with the "--no-ares" option), in which case Xymon cannot control the timeout for the DNS lookup. Try running with the option "--dnslog=FILENAME", this will log those DNS requests that fail. It is probably one of those. Regards, Henrik
Hello, when we are talking about, I have one question in this area (DNS). I don't understand why http tests resolve hostnames when you explicitly define ip address in test. Is this intention to make dns test together ? (I have small patch to disable it :-). Thank for answer, Regards, -- Milan Kocian
list Henrik Størner
▸
In <user-f2f6f8e2dc9f@xymon.invalid> Milan Kocian <user-d0e37a854cb1@xymon.invalid> writes:
when we are talking about, I have one question in this area (DNS). I don't understand why http tests resolve hostnames when you explicitly define ip address in test. Is this intention to make dns test together ? (I have small patch to disable it :-).
It's because webservers often use the hostname-part of the URL to determine how to process the request. E.g. if you have 10 sites hosted on the same physical box (= same IP), then you must specify the "hostname" in the URL to hit the right virtual webserver. That's why you can override the DNS lookup: 10.0.0.1 www.foo.com # http://abc.foo.com=192.168.1.2/ Regards, Henrik
list Milan Kocian
▸
On Fri, Oct 09, 2009 at 02:37:36PM +0000, Henrik Størner wrote:
In <user-f2f6f8e2dc9f@xymon.invalid> Milan Kocian <user-d0e37a854cb1@xymon.invalid> writes:when we are talking about, I have one question in this area (DNS). I don't understand why http tests resolve hostnames when you explicitly define ip address in test. Is this intention to make dns test together ? (I have small patch to disable it :-).It's because webservers often use the hostname-part of the URL to determine how to process the request. E.g. if you have 10 sites hosted on the same physical box (= same IP), then you must specify the "hostname" in the URL to hit the right virtual webserver.
Many thanks for answer. Yes, I understand this behaviour.
That's why you can override the DNS lookup: 10.0.0.1 www.foo.com # http://abc.foo.com=192.168.1.2/
But my question is: Why is done dns query in your case ? IMHO when I specify ip address explicitly (via =), I don't need dns query in bbtest-net.
Regards, Henrik
May be my patch helps to understand what I mean :
--- xymon-4.2.3/bbnet/bbtest-net.c 2009-02-09 10:28:57.000000000 +0100
+++ xymon-sources-4.2.3/bbnet/bbtest-net.c 2009-09-10 17:05:25.000000000 +0200
@@ -540,7 +540,8 @@
}
else {
s = httptest;
- add_url_to_dns_queue(testspec);
+ if (!url.desturl->ip)
+ add_url_to_dns_queue(testspec);
}
}
else if (argnmatch(testspec, "apache") || argnmatch(testspec, "apache=")) {
Regard,
--
Milan Kocian
list Martin Ward
Hey Henrik/all, I need more hand-holding.
I modified the command line for the bbtest-net command to include the dnslog option, the entry now looks like this:
[bbnet]
ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
NEEDS hobbitd
CMD bbtest-net --report --ping --checkresponse --timeout=20 --dns-timeout=10 --dnslog=/var/log/hobbit/dns.log
LOGFILE $BBSERVERLOGS/bb-network.log
INTERVAL 10m
After restarting the whole Xymon application I looked at the open connections that the bbtest-net process had and saw this:
user-a948b03128cf@xymon.invalid:/home/mward>pfiles /proc/2944
2944: bbtest-net --report --ping --checkresponse --timeout=20 --dns-timeout=
Current rlimit: 256 file descriptors
0: S_IFCHR mode:0666 dev:285,3 ino:41103 uid:0 gid:3 rdev:13,2
O_RDONLY
/dev/null
1: S_IFREG mode:0644 dev:61,261 ino:41257 uid:1985 gid:1985 size:1319913
O_WRONLY|O_APPEND|O_CREAT
/var/log/hobbit/bb-network.log
2: S_IFREG mode:0644 dev:61,261 ino:41257 uid:1985 gid:1985 size:1319913
O_WRONLY|O_APPEND|O_CREAT
/var/log/hobbit/bb-network.log
3: S_IFREG mode:0644 dev:61,261 ino:42256 uid:1985 gid:1985 size:0
O_WRONLY|O_CREAT|O_TRUNC
/var/log/hobbit/dns.log
4: S_IFSOCK mode:0666 dev:280,0 ino:25389 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK
SOCK_STREAM
SO_SNDBUF(49152),SO_RCVBUF(49640),IP_NEXTHOP(232.193.0.0)
sockname: AF_INET6 ::ffff:10.44.107.48 port: 43062
peername: AF_INET6 ::ffff:10.44.107.51 port: 389
5: S_IFSOCK mode:0666 dev:280,0 ino:59068 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK FD_CLOEXEC
SOCK_DGRAM
SO_DGRAM_ERRIND,SO_SNDBUF(57344),SO_RCVBUF(57344),IP_NEXTHOP(0.224.0.0)
sockname: AF_INET 212.74.77.181 port: 50902
peername: AF_INET 212.74.77.108 port: 53
So you can see that the bbtest-net process has the log file opened OK on file ID 3.
The problem is that the log file is empty :-(
user-a948b03128cf@xymon.invalid:/home/mward>ls -al /var/log/hobbit/dns.log
-rw-r--r-- 1 hobbit hobbit 0 Oct 12 12:05 /var/log/hobbit/dns.log
I did reduce the number of tests being made as there were a load of duplicates but I have plenty of DNS tests still configured, here you can see one configured entry:
user-a948b03128cf@xymon.invalid:/home/mward>grep dns= /opt/hobbit/server/etc/bb-hosts /opt/hobbit/server/etc/bb-hosts-* | egrep -v ":#" | wc -l
79
user-a948b03128cf@xymon.invalid:/home/mward>grep dns= /opt/hobbit/server/etc/bb-hosts | head -1
/opt/hobbit/server/etc/bb-hosts:212.121.128.1 ns0.de.colt.net # ssh dns=ns:de.colt.net COMMENT:212.121.128.1
and all these take 1,351 seconds to complete testing.
So it seems I need to add another step in my process to resolve my DNS issues, how can I persuade bbtest-net to actually log something?
|\/|artin
▸
-----Original Message----- From: Ward, Martin [mailto:user-2d33a6eb6a05@xymon.invalid] Sent: 09 October 2009 14:26 To: user-ae9b8668bcde@xymon.invalid Subject: RE: [hobbit] Which DNS query is causing delays Thanks Henrik, I am not using the --no-ares option so have put the dnslog entry in as you recommend. Thanks and I'll let you know how I get on. |\/|artin-----Original Message----- From: Henrik "Størner [mailto:user-ce4a2c883f75@xymon.invalid] Sent: 09 October 2009 12:26 To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] Which DNS query is causing delays In <user-f2650c92cc26@xymon.invalid> "Ward, Martin" <user-2d33a6eb6a05@xymon.invalid> writes:I have noticed recently that my bb-test runs are taking a long time to complete, 6500 seconds instead of the maximum of 600. The info on the bb-test results page shows that the DNS tests are taking most of the time, but I have around 200 of these, does anyone have any suggestions on how I can find out which DNS tests are causing the delays without having to click on every DNS test in all of my Xymon website?Are you using a local DNS cache ? I would strongly recommend that you do - the resolver library in Xymon is pretty aggressive, and has been known to knock out DNS servers if you have many hosts (although 200 doesn't sound like an awful lot). It shouldn't take that long unless you're using the standard resolver library (i.e. you are running bbtest-net with the "--no-ares" option), in which case Xymon cannot control the timeout for the DNS lookup. Try running with the option "--dnslog=FILENAME", this will log those DNS requests that fail. It is probably one of those. Regards, Henrik*********************************************************************** ************** The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way. The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing user-61c7f445d564@xymon.invalid and delete the message and any attachments without retaining any copies. Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses. No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party. Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.
*************************************************************************************
The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way.
The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing user-61c7f445d564@xymon.invalid and delete the message and any attachments without retaining any copies.
Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses.
No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party.
Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.
list Henrik Størner
In <user-40bc17b07875@xymon.invalid> "Ward, Martin" <user-2d33a6eb6a05@xymon.invalid> writes:
I modified the command line for the bbtest-net command to include the dnslo= g option, the entry now looks like this: The problem is that the log file is empty :-(
It will only log those DNS lookups that fail, not those that succeed. Regards, Henrik
list Martin Ward
▸
-----Original Message----- From: Henrik "Størner [mailto:user-ce4a2c883f75@xymon.invalid] Sent: 12 October 2009 15:49 To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] Which DNS query is causing delays In <user-40bc17b07875@xymon.invalid> "Ward, Martin" <user-2d33a6eb6a05@xymon.invalid> writes:I modified the command line for the bbtest-net command to include the dnslo= g option, the entry now looks like this: The problem is that the log file is empty :-(It will only log those DNS lookups that fail, not those that succeed.
OK, so it now seems likely that all the DNS tests are working, they are just taking a very long time to do it, so the question is: Is there an easy way to get the DNS resolving times for all my Xymon DNS queries without having to view every web page?
▸
|\/|artin
*************************************************************************************
The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way.
The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing user-61c7f445d564@xymon.invalid and delete the message and any attachments without retaining any copies.
Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses.
No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party.
Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.
list Henrik Størner
▸
In <user-4d9f326b31b0@xymon.invalid> "Ward, Martin" <user-2d33a6eb6a05@xymon.invalid> writes:
It will only log those DNS lookups that fail, not those that succeed.
OK, so it now seems likely that all the DNS tests are working, they are jus= t taking a very long time to do it, so the question is: Is there an easy wa= y to get the DNS resolving times for all my Xymon DNS queries without havin= g to view every web page?
The only way I can think of would be to run bbtest-net with the "--debug" option; that will log lines like (from the demo site): 2009-10-12 21:57:50 Adding hostname 'wifi.hswn.dk' to resolver queue 2009-10-12 21:57:50 Adding hostname 'mail.hswn.dk' to resolver queue 2009-10-12 21:57:50 Adding hostname 'www.hswn.dk' to resolver queue 2009-10-12 21:57:50 Adding hostname 'webmail.hswn.dk' to resolver queue 2009-10-12 21:57:50 Adding hostname 'news.hswn.dk' to resolver queue 2009-10-12 21:57:50 Processing 13 DNS lookups with ARES 2009-10-12 21:57:50 Got DNS result for host wifi.hswn.dk : 172.16.11.1 2009-10-12 21:57:50 Got DNS result for host mail.hswn.dk : 172.16.10.2 which shows that my DNS lookups take less than a second. You have to do the subtraction manually to get the time spent on each lookup, but it should still be fairly easy to spot those that are slow. Regards, Henrik