XYMON Proxy Issue

Andy Smith wrote:

Andy Smith wrote:

Hi,

In February, Gautier reported this issue with xymonproxy on Solaris :-

http://lists.xymon.com/pipermail/xymon/2014-February/039160.html

I have come this week to update an installation of 4.2.3 on Solaris 9 
and have encountered the exact same issue as Gautier, but this time on 
the latest 4.3.17 code :-

2014-05-04 13:05:36 xymonproxy version 4.3.17 starting
2014-05-04 13:20:41 Listening on 0.0.0.0:1984 <http://0.0.0.0:1984>;
2014-05-04 13:20:41 Sending to Xymon server(s) xx.xx.xx.xx:1984
2014-05-04 13:20:41 select() failed: Invalid argument
2014-05-04 13:20:41 select() failed: Invalid argument
2014-05-04 13:20:41 select() failed: Invalid argument
2014-05-04 13:20:41 select() failed: Invalid argument
2014-05-04 13:20:41 select() failed: Invalid argument
2014-05-04 13:20:41 select() failed: Invalid argument
2014-05-04 13:20:41 Too many select failures, aborting
2014-05-04 13:20:46 xymonproxy version 4.3.17 starting

I do not suffer the connections in TIME_WAIT, just the constant 
restarting of the proxy every 15 minutes.  Here is the truss as it 
gasps when falling over :-

poll(0xFFBFF208, 1, 1000)                       = 0
time()                                          = 1399206937
poll(0xFFBFF208, 1, 1000)                       = 0
time()                                          = 1399206938
poll(0xFFBFF208, 1, 1000)                       = 0
time()                                          = 1399206939
poll(0xFFBFF208, 1, 1000)                       = 0
time()                                          = 1399206940
poll(0xFFBFF208, 1, 1000)                       = 0
time()                                          = 1399206941
poll(0xFFBFF208, 1, 1000)                       = 0
time()                                          = 1399206942
poll(0xFFBFF208, 1, 1000)                       = 1
accept(3, 0x0003AC60, 0xFFBFF310, 1)            = 4
fcntl(4, F_SETFL, 0x00000080)                   = 0
time()                                          = 1399206942
poll(0xFFBFF200, 2, 1000)                       = 1
read(4, " s t a t u s + 4 5   c s".., 8185)     = 140
time()                                          = 1399206942
poll(0xFFBFF200, 2, 1000)                       = 1
read(4, 0x00038CE2, 8045)                       = 0
time()                                          = 1399206942
shutdown(4, 2, 1)                               = 0
close(4)                                        = 0
poll(0xFFBFF208, 1, 1000)                       = 1
accept(3, 0x0003ACD0, 0xFFBFF310, 1)            = 4
fcntl(4, F_SETFL, 0x00000080)                   = 0
time()                                          = 1399206942
time()                                          = 1399206942
write(2, " 2 0 1 4 - 0 5 - 0 4   1".., 19)      = 19
write(2, "  ", 1)                               = 1
write(2, " s e l e c t ( )   f a i".., 34)      = 34
time()                                          = 1399206942
time()                                          = 1399206942
write(2, " 2 0 1 4 - 0 5 - 0 4   1".., 19)      = 19
write(2, "  ", 1)                               = 1
write(2, " s e l e c t ( )   f a i".., 34)      = 34
time()                                          = 1399206942
time()                                          = 1399206942
write(2, " 2 0 1 4 - 0 5 - 0 4   1".., 19)      = 19
write(2, "  ", 1)                               = 1
write(2, " s e l e c t ( )   f a i".., 34)      = 34
time()                                          = 1399206942
time()                                          = 1399206942
write(2, " 2 0 1 4 - 0 5 - 0 4   1".., 19)      = 19
write(2, "  ", 1)                               = 1
write(2, " s e l e c t ( )   f a i".., 34)      = 34
time()                                          = 1399206942
time()                                          = 1399206942
write(2, " 2 0 1 4 - 0 5 - 0 4   1".., 19)      = 19
write(2, "  ", 1)                               = 1
write(2, " s e l e c t ( )   f a i".., 34)      = 34
time()                                          = 1399206942
time()                                          = 1399206942
write(2, " 2 0 1 4 - 0 5 - 0 4   1".., 19)      = 19
write(2, "  ", 1)                               = 1
write(2, " s e l e c t ( )   f a i".., 34)      = 34
time()                                          = 1399206942
write(2, " 2 0 1 4 - 0 5 - 0 4   1".., 19)      = 19
write(2, "  ", 1)                               = 1
write(2, " T o o   m a n y   s e l".., 35)      = 35
_exit(1)

So, question to Gautier, are you using Solaris 9 and have you managed 
to resolve this?

Another question to the rest of the list, this is actually the only 
proxy I have on Solaris, all the otehrs are on Redhat, is anyone else 
using xymonproxy on Solaris and if so, what version?  For the time 
being, I am running the old bbproxy until I get this fixed, the rest 
of 4.3.17 seems to be working OK.

Done a bit more digging around.  Firstly, if I regress to r#7368 
(4.3.13) then xymonproxy on Solaris is stable.  This just hides the 
problem of course and might be a factor in Gautier's performance issue.

If I modify the code for 4.3.17 to remove the exit after 5 select() 
failures and add in some further debugging, I can observe that on 
Solaris 9 at least :-

- every 900 seconds, select() fails
- select continues to fail for 2 seconds then succeeds and the proxy 
continues as normal.
- during these 2 seconds, there are no further calls to poll(), but 
somewhere in the region of 50,000 calls to time().
- the values for the selecttmo structure and maxfd are reasonable, so 
the invalid argument must be one of the fdread or fdwrite structures.

Continuing to collect information but still not sure if I am looking at 
a Sol9 issue or if this affects later Solaris versions.

This issue affected Solaris 10 as well, the attached patch resolves all 
my xymonproxy stability problems on Solaris platforms, I believe the 
patch is relevant to other platforms also, just that the select() on 
other platforms is more tolerant.

-- 
Andy