TCP Connection Issues?
list Ryan Skadberg
Anyone ever seen this? I have a few hundred servers in my xymon setup and have about 200 http checks, as well as some other server side scripts that check things via tcp. Recently all of my http and telnet connections will start to flap for 5-10 mins and cause all/most of my http and telnet checks to go red. I added a delayred to stop the paging, but I am still getting a wonderful Christmas tree a few times a day, just yellow now. It feels like an issue with the number of file handles or maybe the number of tcp connections, but I can't seem to get the right formula to fix it up. Anyone have any suggestions? Skadz PS - Running 4.3.17. Haven't had a time to upgrade in a while (and had some issues with some of the web changes last time I tried)
list John Thurston
▸
On 1/30/2017 1:06 PM, Ryan Skadberg wrote:
It feels like an issue with the number of file handles or maybe the number of tcp connections, but I can't seem to get the right formula to fix it up. Anyone have any suggestions?
It seems a little early to be hypothesizing causes with such a loose description of the failure.
Have you enabled --debug on the xymonnet process? That might tell you if the requests are all going out but the responses are not coming back. Or if there is a significant hicup in name resolution several times a day. Or . . .
If you run 'xymonnet --no-update' when the problem is occurring, do you see anything interesting in the output?
'twould also be helpful to know what operating system and network arrangement you have. Is this all Linux on VMWare with fake nics? Is your server an AX-400 on token ring?
--
Do things because you should, not just because you can.
John Thurston XXX-XXX-XXXX
user-ce4d79d99bab@xymon.invalid
Enterprise Technology Services
Department of Administration
State of Alaska
list Ryan Skadberg
So, we were able to figure this out on our own. The fix was to add "--concurrency=25" to the xymonnet test and everything got much much better. Skadz
▸
On Mon, Jan 30, 2017 at 5:06 PM, Ryan Skadberg <user-fd74ceebcd5d@xymon.invalid> wrote:
Anyone ever seen this? I have a few hundred servers in my xymon setup and have about 200 http checks, as well as some other server side scripts that check things via tcp. Recently all of my http and telnet connections will start to flap for 5-10 mins and cause all/most of my http and telnet checks to go red. I added a delayred to stop the paging, but I am still getting a wonderful Christmas tree a few times a day, just yellow now. It feels like an issue with the number of file handles or maybe the number of tcp connections, but I can't seem to get the right formula to fix it up. Anyone have any suggestions? Skadz PS - Running 4.3.17. Haven't had a time to upgrade in a while (and had some issues with some of the web changes last time I tried)