Large number of TIME_WAITs
list Gary Baluha
I noticed something strange on my Hobbit server, though I'm not sure if it's a problem or not. There are about 140 TIME_WAITs from the local IP of the Hobbit server, and about 30 TIME_WAITs from localhost, all using port 1984. They look something like this: 0 0 1.2.3.4:42389 1.2.3.4:1984 TIME_WAIT 0 0 1.2.3.4:42394 1.2.3.4:1984 TIME_WAIT 0 0 1.2.3.4:42399 1.2.3.4:1984 TIME_WAIT 0 0 1.2.3.4:42396 1.2.3.4:1984 TIME_WAIT 0 0 1.2.3.4:42397 1.2.3.4:1984 TIME_WAIT 0 0 127.0.0.1:42310 127.0.0.1:1984 TIME_WAIT 0 0 127.0.0.1:42309 127.0.0.1:1984 TIME_WAIT 0 0 127.0.0.1:42315 127.0.0.1:1984 TIME_WAIT 0 0 127.0.0.1:42322 127.0.0.1:1984 TIME_WAIT 0 0 127.0.0.1:42320 127.0.0.1:1984 TIME_WAIT There are also a few on various other ports: 0 0 127.0.0.1:42246 127.0.0.1:29021 TIME_WAIT 0 0 127.0.0.1:42301 127.0.0.1:29023 TIME_WAIT 0 0 127.0.0.1:42362 127.0.0.1:29017 TIME_WAIT 0 0 127.0.0.1:29030 127.0.0.1:42380 TIME_WAIT 0 0 127.0.0.1:42348 127.0.0.1:29019 TIME_WAIT Any idea why there should be so many of them, particularly on the localhost address? And any idea if this looks abnormal or not?
list Greg L Hubbard
What OS? Some operating systems allow network connections to hang around a bit after disconnection. On Solaris you can set a parameter to reduce the timeout period. Hobbit could trigger this because each client connects to the server, sends a data blast, and then disconnects.
▸
From: Gary Baluha [mailto:user-ae3e15c22de1@xymon.invalid]
Sent: Tuesday, January 29, 2008 8:58 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Large number of TIME_WAITs
I noticed something strange on my Hobbit server, though I'm not
sure if it's a problem or not. There are about 140 TIME_WAITs from the
local IP of the Hobbit server, and about 30 TIME_WAITs from localhost,
all using port 1984. They look something like this:
0 0 1.2.3.4:42389 1.2.3.4:1984 TIME_WAIT
0 0 1.2.3.4:42394 1.2.3.4:1984 TIME_WAIT
0 0 1.2.3.4:42399 1.2.3.4:1984 TIME_WAIT
0 0 1.2.3.4:42396 1.2.3.4:1984 TIME_WAIT
0 0 1.2.3.4:42397 1.2.3.4:1984 TIME_WAIT
0 0 127.0.0.1:42310 127.0.0.1:1984
TIME_WAIT
0 0 127.0.0.1:42309 127.0.0.1:1984
TIME_WAIT
0 0 127.0.0.1:42315 127.0.0.1:1984
TIME_WAIT
0 0 127.0.0.1:42322 127.0.0.1:1984
TIME_WAIT
0 0 127.0.0.1:42320 127.0.0.1:1984
TIME_WAIT
There are also a few on various other ports:
0 0 127.0.0.1:42246 127.0.0.1:29021
TIME_WAIT
0 0 127.0.0.1:42301 127.0.0.1:29023
TIME_WAIT
0 0 127.0.0.1:42362 127.0.0.1:29017
TIME_WAIT
0 0 127.0.0.1:29030 127.0.0.1:42380
TIME_WAIT
0 0 127.0.0.1:42348 127.0.0.1:29019
TIME_WAIT
Any idea why there should be so many of them, particularly on
the localhost address? And any idea if this looks abnormal or not?
list Gary Baluha
Red Hat ES 4 Update 5. That's what I normally would think, but it doesn't make any sense that the source AND destination are both localhost or the local IP. It's as though the Hobbit server is trying to communicate with itself using network sockets. It would make more sense if either the source or destination was a client IP, but that isn't the case.
▸
On Jan 29, 2008 10:15 AM, Hubbard, Greg L <user-d970b5e56ec9@xymon.invalid> wrote:
What OS? Some operating systems allow network connections to hang around a bit after disconnection. On Solaris you can set a parameter to reduce the timeout period. Hobbit could trigger this because each client connects to the server, sends a data blast, and then disconnects. *From:* Gary Baluha [mailto:user-ae3e15c22de1@xymon.invalid] *Sent:* Tuesday, January 29, 2008 8:58 AM *To:* user-ae9b8668bcde@xymon.invalid *Subject:* [hobbit] Large number of TIME_WAITs I noticed something strange on my Hobbit server, though I'm not sure if it's a problem or not. There are about 140 TIME_WAITs from the local IP of the Hobbit server, and about 30 TIME_WAITs from localhost, all using port 1984. They look something like this: 0 0 1.2.3.4:42389 1.2.3.4:1984 TIME_WAIT 0 0 1.2.3.4:42394 1.2.3.4:1984 TIME_WAIT 0 0 1.2.3.4:42399 1.2.3.4:1984 TIME_WAIT 0 0 1.2.3.4:42396 1.2.3.4:1984 TIME_WAIT 0 0 1.2.3.4:42397 1.2.3.4:1984 TIME_WAIT 0 0 127.0.0.1:42310 127.0.0.1:1984 TIME_WAIT 0 0 127.0.0.1:42309 127.0.0.1:1984 TIME_WAIT 0 0 127.0.0.1:42315 127.0.0.1:1984 TIME_WAIT 0 0 127.0.0.1:42322 127.0.0.1:1984 TIME_WAIT 0 0 127.0.0.1:42320 127.0.0.1:1984 TIME_WAIT There are also a few on various other ports: 0 0 127.0.0.1:42246 127.0.0.1:29021 TIME_WAIT 0 0 127.0.0.1:42301 127.0.0.1:29023 TIME_WAIT 0 0 127.0.0.1:42362 127.0.0.1:29017 TIME_WAIT 0 0 127.0.0.1:29030 127.0.0.1:42380 TIME_WAIT 0 0 127.0.0.1:42348 127.0.0.1:29019 TIME_WAIT Any idea why there should be so many of them, particularly on the localhost address? And any idea if this looks abnormal or not?
list Howard Snyder
All, I have a question about the client side of things. I have figured out how to get the client installed on one box and have started it but I'm getting an error that I can't figure out. Any suggestions would be helpful. This is the error in the hobbitclient.log that I'm getting: ld.so.1: hobbitd_client: fatal: libpcre.so.0: open failed: No such file or directory Killed Any Ideas?
list Galen Johnson
Wouldn't it make sense if you had a Hobbit client running on the same server as the Hobbit server? =G=
▸
From: Gary Baluha [mailto:user-ae3e15c22de1@xymon.invalid]
Sent: Tuesday, January 29, 2008 10:36 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Large number of TIME_WAITs
Red Hat ES 4 Update 5.
That's what I normally would think, but it doesn't make any sense that
the source AND destination are both localhost or the local IP. It's as
though the Hobbit server is trying to communicate with itself using
network sockets. It would make more sense if either the source or
destination was a client IP, but that isn't the case.
On Jan 29, 2008 10:15 AM, Hubbard, Greg L <user-d970b5e56ec9@xymon.invalid> wrote:
What OS? Some operating systems allow network connections to hang
around a bit after disconnection. On Solaris you can set a parameter to
reduce the timeout period. Hobbit could trigger this because each
client connects to the server, sends a data blast, and then disconnects.
From: Gary Baluha [mailto:user-ae3e15c22de1@xymon.invalid]
Sent: Tuesday, January 29, 2008 8:58 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Large number of TIME_WAITs
I noticed something strange on my Hobbit server, though I'm not
sure if it's a problem or not. There are about 140 TIME_WAITs from the
local IP of the Hobbit server, and about 30 TIME_WAITs from localhost,
all using port 1984. They look something like this:
0 0 1.2.3.4:42389 1.2.3.4:1984 TIME_WAIT
0 0 1.2.3.4:42394 1.2.3.4:1984 TIME_WAIT
0 0 1.2.3.4:42399 1.2.3.4:1984 TIME_WAIT
0 0 1.2.3.4:42396 1.2.3.4:1984 TIME_WAIT
0 0 1.2.3.4:42397 1.2.3.4:1984 TIME_WAIT
0 0 127.0.0.1:42310 127.0.0.1:1984
TIME_WAIT
0 0 127.0.0.1:42309 127.0.0.1:1984
TIME_WAIT
0 0 127.0.0.1:42315 127.0.0.1:1984
TIME_WAIT
0 0 127.0.0.1:42322 127.0.0.1:1984
TIME_WAIT
0 0 127.0.0.1:42320 127.0.0.1:1984
TIME_WAIT
There are also a few on various other ports:
0 0 127.0.0.1:42246 127.0.0.1:29021
TIME_WAIT
0 0 127.0.0.1:42301 127.0.0.1:29023
TIME_WAIT
0 0 127.0.0.1:42362 127.0.0.1:29017
TIME_WAIT
0 0 127.0.0.1:29030 127.0.0.1:42380
TIME_WAIT
0 0 127.0.0.1:42348 127.0.0.1:29019
TIME_WAIT
Any idea why there should be so many of them, particularly on
the localhost address? And any idea if this looks abnormal or not?
list Arun Thangappandi
HI. This error is due to the problem that pcre is not installed in the box or the path of library or include path is mentioned. Thanks & Regards, Arunkumar
▸
From: Snyder, Howard [mailto:user-e66ae0780d99@xymon.invalid]
Sent: Tuesday, January 29, 2008 9:10 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Client Question
All,
I have a question about the client side of things. I have
figured out how to get the client installed on one box and have started
it but I'm getting an error that I can't figure out. Any suggestions
would be helpful. This is the error in the hobbitclient.log that I'm
getting:
ld.so.1: hobbitd_client: fatal: libpcre.so.0: open failed: No
such file or directory
Killed
Any Ideas?
list Gary Baluha
▸
Wouldn't it make sense if you had a Hobbit client running on the same server as the Hobbit server?
But that many TIME_WAITs?
▸
=G= *From:* Gary Baluha [mailto:user-ae3e15c22de1@xymon.invalid] *Sent:* Tuesday, January 29, 2008 10:36 AM *To:* user-ae9b8668bcde@xymon.invalid *Subject:* Re: [hobbit] Large number of TIME_WAITs Red Hat ES 4 Update 5. That's what I normally would think, but it doesn't make any sense that the source AND destination are both localhost or the local IP. It's as though the Hobbit server is trying to communicate with itself using network sockets. It would make more sense if either the source or destination was a client IP, but that isn't the case. On Jan 29, 2008 10:15 AM, Hubbard, Greg L <user-d970b5e56ec9@xymon.invalid> wrote: What OS? Some operating systems allow network connections to hang around a bit after disconnection. On Solaris you can set a parameter to reduce the timeout period. Hobbit could trigger this because each client connects to the server, sends a data blast, and then disconnects. *From:* Gary Baluha [mailto:user-ae3e15c22de1@xymon.invalid] *Sent:* Tuesday, January 29, 2008 8:58 AM *To:* user-ae9b8668bcde@xymon.invalid *Subject:* [hobbit] Large number of TIME_WAITs I noticed something strange on my Hobbit server, though I'm not sure if it's a problem or not. There are about 140 TIME_WAITs from the local IP of the Hobbit server, and about 30 TIME_WAITs from localhost, all using port 1984. They look something like this: 0 0 1.2.3.4:42389 1.2.3.4:1984 TIME_WAIT 0 0 1.2.3.4:42394 1.2.3.4:1984 TIME_WAIT 0 0 1.2.3.4:42399 1.2.3.4:1984 TIME_WAIT 0 0 1.2.3.4:42396 1.2.3.4:1984 TIME_WAIT 0 0 1.2.3.4:42397 1.2.3.4:1984 TIME_WAIT 0 0 127.0.0.1:42310 127.0.0.1:1984 TIME_WAIT 0 0 127.0.0.1:42309 127.0.0.1:1984 TIME_WAIT 0 0 127.0.0.1:42315 127.0.0.1:1984 TIME_WAIT 0 0 127.0.0.1:42322 127.0.0.1:1984 TIME_WAIT 0 0 127.0.0.1:42320 127.0.0.1:1984 TIME_WAIT There are also a few on various other ports: 0 0 127.0.0.1:42246 127.0.0.1:29021 TIME_WAIT 0 0 127.0.0.1:42301 127.0.0.1:29023 TIME_WAIT 0 0 127.0.0.1:42362 127.0.0.1:29017 TIME_WAIT 0 0 127.0.0.1:29030 127.0.0.1:42380 TIME_WAIT 0 0 127.0.0.1:42348 127.0.0.1:29019 TIME_WAIT Any idea why there should be so many of them, particularly on the localhost address? And any idea if this looks abnormal or not?
list Buchan Milne
▸
On Tuesday 29 January 2008 17:53:59 Gary Baluha wrote:
Wouldn't it make sense if you had a Hobbit client running on the same server as the Hobbit server?But that many TIME_WAITs?
On a server receiving monitoring information and monitoring itself a total of ~ 150 hosts, I have ~ 500 hobbit ports in TIME_WAIT, and about 200 htto ports in TIME_WAIT. More answers in-line (ug, thread re-assembly ...).
▸
=G= *From:* Gary Baluha [mailto:user-ae3e15c22de1@xymon.invalid] *Sent:* Tuesday, January 29, 2008 10:36 AM *To:* user-ae9b8668bcde@xymon.invalid *Subject:* Re: [hobbit] Large number of TIME_WAITs Red Hat ES 4 Update 5.
same on mine.
▸
That's what I normally would think, but it doesn't make any sense that the source AND destination are both localhost or the local IP. It's as though the Hobbit server is trying to communicate with itself using network sockets. It would make more sense if either the source or destination was a client IP, but that isn't the case.
It is the case, bbtest-net connects to hobbitd. The client on the server also does. Any other server-side extensions may. bbgen may also (haven't checked, but I assume ...). The cgi's will probably also ...
▸
On Jan 29, 2008 10:15 AM, Hubbard, Greg L <user-d970b5e56ec9@xymon.invalid> wrote: What OS? Some operating systems allow network connections to hang around a bit after disconnection. On Solaris you can set a parameter to reduce the timeout period.
On Linux, it is net.ipv4.tcp_fin_timeout, e.g.: net.ipv4.tcp_fin_timeout = 30 in /etc/sysctl.conf, defaults to 60s. Regards, Buchan
list Howard Snyder
That is kinda what I thought. I have installed pcre twice and still get the error and no data to the server. Thank you, Howard Snyder user-e66ae0780d99@xymon.invalid SR Network Control Engineer Network Services XXX-XXX-XXXX This email, and any attachments, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. It is the property of Cingular Wireless and its Affiliates. If you are not the intended recipient of this email, you are hereby notified that any dissemination, distribution or copying of this email, any attachments thereto, and any use of the information contained is strictly prohibited. If you have received this email in error, please notify me at XXX.XXX.XXXX and permanently delete the original and any copy thereof. For login requests and password resets, you will need to open a request on the "My Logins" website: https://nslogins.edc.cingular.net/login.cfm?CFID=13246&CFTOKEN=41726264 <https://nslogins.edc.cingular.net/login.cfm?CFID=13246&CFTOKEN=41726264
For OSS problems: DL-MNOC-OSS Adjunct DL-MNOC-OSS Siemens DL-MNOC-OSS Nortel DL-MNOC-OSS Nokia DL-MNOC-OSS Lucent DL-MNOC-OSS Ericsson From: Thangappandi, Arun (Ge Money, consultant) [mailto:user-8a9ba39d15af@xymon.invalid] Sent: Tuesday, January 29, 2008 10:53 AM To: user-ae9b8668bcde@xymon.invalid Subject: RE: [hobbit] Client Question
▸
HI.
This error is due to the problem that pcre is not installed in the box
or the path of library or include path is mentioned.
Thanks & Regards,
Arunkumar
From: Snyder, Howard [mailto:user-e66ae0780d99@xymon.invalid]
Sent: Tuesday, January 29, 2008 9:10 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Client Question
All,
I have a question about the client side of things. I have
figured out how to get the client installed on one box and have started
it but I'm getting an error that I can't figure out. Any suggestions
would be helpful. This is the error in the hobbitclient.log that I'm
getting:
ld.so.1: hobbitd_client: fatal: libpcre.so.0: open failed: No
such file or directory
Killed
Any Ideas?
list Galen Johnson
What OS? If linux, you'll need to make sure the dev libraries are installed. If Solaris, you may need to add the lib path to the system lib path (man crle). =G=
▸
From: Snyder, Howard [mailto:user-e66ae0780d99@xymon.invalid] Sent: Tuesday, January 29, 2008 12:00 PM To: user-ae9b8668bcde@xymon.invalid Subject: RE: [hobbit] Client Question That is kinda what I thought. I have installed pcre twice and still get the error and no data to the server. Thank you, Howard Snyder user-e66ae0780d99@xymon.invalid SR Network Control Engineer Network Services XXX-XXX-XXXX This email, and any attachments, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. It is the property of Cingular Wireless and its Affiliates. If you are not the intended recipient of this email, you are hereby notified that any dissemination, distribution or copying of this email, any attachments thereto, and any use of the information contained is strictly prohibited. If you have received this email in error, please notify me at XXX.XXX.XXXX and permanently delete the original and any copy thereof. For login requests and password resets, you will need to open a request on the "My Logins" website: https://nslogins.edc.cingular.net/login.cfm?CFID=13246&CFTOKEN=41726264 For OSS problems: DL-MNOC-OSS Adjunct DL-MNOC-OSS Siemens DL-MNOC-OSS Nortel DL-MNOC-OSS Nokia DL-MNOC-OSS Lucent DL-MNOC-OSS Ericsson From: Thangappandi, Arun (Ge Money, consultant) [mailto:user-8a9ba39d15af@xymon.invalid] Sent: Tuesday, January 29, 2008 10:53 AM To: user-ae9b8668bcde@xymon.invalid Subject: RE: [hobbit] Client Question HI. This error is due to the problem that pcre is not installed in the box or the path of library or include path is mentioned. Thanks & Regards, Arunkumar From: Snyder, Howard [mailto:user-e66ae0780d99@xymon.invalid] Sent: Tuesday, January 29, 2008 9:10 PM To: user-ae9b8668bcde@xymon.invalid Subject: [hobbit] Client Question All, I have a question about the client side of things. I have figured out how to get the client installed on one box and have started it but I'm getting an error that I can't figure out. Any suggestions would be helpful. This is the error in the hobbitclient.log that I'm getting: ld.so.1: hobbitd_client: fatal: libpcre.so.0: open failed: No such file or directory Killed Any Ideas?
list Henrik Størner
▸
On Tue, Jan 29, 2008 at 10:36:17AM -0500, Gary Baluha wrote:
Red Hat ES 4 Update 5. That's what I normally would think, but it doesn't make any sense that the source AND destination are both localhost or the local IP. It's as though the Hobbit server is trying to communicate with itself using network sockets.
It makes perfect sense, actually. Various parts of Hobbit does talk to the hobbitd daemon - most of the webpages get their data this way, and all of the network test results are also reported using a network connection. TIME_WAIT states aren't evil, just confusing if you don't know what they are for. They will clear themselves after a few seconds. Regards, Henrik
list Gary Baluha
▸
On Jan 29, 2008 4:19 PM, Henrik Stoerner <user-ce4a2c883f75@xymon.invalid> wrote:
On Tue, Jan 29, 2008 at 10:36:17AM -0500, Gary Baluha wrote:Red Hat ES 4 Update 5. That's what I normally would think, but it doesn't make any sense that the source AND destination are both localhost or the local IP. It's as though the Hobbit server is trying to communicate with itself using network sockets.It makes perfect sense, actually. Various parts of Hobbit does talk to the hobbitd daemon - most of the webpages get their data this way, and all of the network test results are also reported using a network connection. TIME_WAIT states aren't evil, just confusing if you don't know what they are for. They will clear themselves after a few seconds.
Indeed. Thanks for everyone's input.