On Tue, 28 Jun 2016, 23:27 Junaid Shahid <user-bfbf9229dbc9@xymon.invalid> wrote:
Thanks JC!
Now that makes it very clear how CPU stats contain server's timestamp (and
why).
I have checked we are running version 4.3.21.
Now lets look at the reasons of skew:
a) your xymon server itself is wrong
Our server's time is correct (as I have manually checked it multiple times
manually and also with "ntpstats"). Plus, we have some 300+ clients under
Xymon monitoring, and none of them exhibit any time skew in their CLOCK
Offset trends
b) you have a xymonproxy in the middle and messages are delayed getting to
xymond
We don't use any xymon proxy
c) your xymond_client process is backlogged with [client] messages
This also can't be the reason because all other clients don't exhibit any
noticeable skew in their respective Clock Offset trends
d) your xymon server is overloaded and has a long period between transmission
and TCP processing by xymond
This also must not be the case as no other client show any noticeable
Clock Offset trend.
In our case there is one specific server (out of 300+) that has a
clock offset trend that alternates b/w 2-15 secs (like a sinusoidal wave).
This machine's time is in perfect sync with our NTP server though (no
clock drift exists actually). This machine has a little complicated network
topology though (behind various layers such as firewalls, load balancers
etc). My only guess now is that this is because of its weird network
location, what do you think JC?
I tend to agree. If it takes a few seconds to make a TCP connection to the
xymon server and transmit the client message, you will see such a delay.
Try manually sending a client message and see how long it takes. Something
like:
$ time $XYMON $XYMSRV "client/timetest $MACHINE.$SERVEROSTYPE"
(run within a xymoncmd shell on the client)
J