no client results arrive, lot of TIME_WAIT at server 🔗 link

On 19 Oct 2017, at 07:30, Norbert Kriegenburg <user-501bbe9c5409@xymon.invalid> wrote:

experts,

i have a new (not my first) Xymon installation (4.3.28 on RH 7.4, self compiled).
All looks good, but the clients only show the network tests (conn, ssh). Only the Xymon server itself shows everything (disk, cpu,...).
- all clients are RH 7.4, same install image used (i just copied over the client part with client config, create a cronjob for automatic start, so no root access nor extra install needed)
- nothing suspicious in server or client logs (even in debug mode)
- no ghosts
- if i send a test manually (the old BB way) this shows at Xymon immediately (so the client settings are ok)
- if i shutdown the xymond at server, the client logs complains:

2017-10-19 05:09:18.051994 Whoops ! Failed to send message (Connection failed)
2017-10-19 05:09:18.052104 -> Could not connect to Xymon daemon at 10.137.56.43:1984 (Connection refused)

(so obviously they communicate)

Querying the xymond shows:
xymon 127.0.0.1 "query clientname.ssh"
green <!-- [flags:OrdastLe] --> Thu Oct 19 05:21:29 2017 ssh ok
is ok, but
xymon 127.0.0.1 "query clientname.disk"
shows nothing

Any ideas?

Norbert

Norbert,

I assume your manual (BB way) is something like `xymon 1.2.3.4 “status
example.com.test …”`.

The xymon-client send `clientdata` that is processed by a “channel
listener” and _that_ sends status-messages similar to above (a bit
simplified).

That channel-listener may not be running/crashing:
Check our `tasks.cfg` for the section `[clientdata]` and verify, if the
`xymond_channel`-process is running (I suspect it’s not). Maybe the
`clientdata.log` helps with debugging. You may also run the task manually
to get further information.

What’s your problem with the "lot of TIME_WAIT”? Are these just for
tcp/1984-connections?

Cheers
Thomas

      On 19 Oct 2017, at 07:30, Norbert Kriegenburg <
      user-501bbe9c5409@xymon.invalid> wrote:


      experts,

      i have a new (not my first) Xymon installation (4.3.28 on RH 7.4,
      self compiled).
      All looks good, but the clients only show the network tests (conn,
      ssh). Only the Xymon server itself shows everything (disk, cpu,...).
      - all clients are RH 7.4, same install image used (i just copied over
      the client part with client config, create a cronjob for automatic
      start, so no root access nor extra install needed)
      - nothing suspicious in server or client logs (even in debug mode)
      - no ghosts
      - if i send a test manually (the old BB way) this shows at Xymon
      immediately (so the client settings are ok)
      - if i shutdown the xymond at server, the client logs complains:

      2017-10-19 05:09:18.051994 Whoops ! Failed to send message
      (Connection failed)
      2017-10-19 05:09:18.052104 -> Could not connect to Xymon
      daemon at 10.137.56.43:1984 (Connection refused)

      (so obviously they communicate)

      Querying the xymond shows:
      xymon 127.0.0.1 "query clientname.ssh"
      green <!-- [flags:OrdastLe] --> Thu Oct 19 05:21:29 2017 ssh ok
      is ok, but
      xymon 127.0.0.1 "query clientname.disk"
      shows nothing

      Any ideas?

      Norbert

On 19 Oct 2017, at 14:49, Norbert Kriegenburg <user-501bbe9c5409@xymon.invalid> wrote:

Yes,
that's what i checked at first. But i assume that also the info from the servers own client data are captured by the same client channel listener, and that obviously works.

Nothing in the clientdata.log, even if i activate the DEBUG mode with a
kill -USR2 PID

My clientdata section:
# The client back-end module. You need this if you are running the Xymon client on any system.
[clientdata]
ENVFILE $XYMONSERVERHOME/etc/xymonserver.cfg
NEEDS xymond
CMD /home/xymon/server/bin/xymond_channel --channel=client \
--log=$XYMONSERVERLOGS/clientdata.log xymond_client \
--no-clear-files \
--no-clear-ports \
--no-port-listing \
--uptime-status
INTERVAL 2m

ps listing:
$ ps -ef|grep "xymond_channel --channel=client"|grep -v grep
xymon 18865 18705 0 05:13 ? 00:00:00 /home/xymon/server/bin/xymond_channel --channel=client --log=/home/xymon/server/logs/clientdata.log xymond_client --no-clear-files --no-clear-ports --no-port-listing --uptime-status

I have a lot of TIME_WAITS from time to time, no idea why this goes up and down:

$ netstat -atn|grep 1984
tcp 0 0 0.0.0.0:1984 0.0.0.0:* LISTEN
tcp 0 0 10.137.56.43:37752 10.137.56.43:1984 TIME_WAIT
tcp 0 0 10.137.56.43:37822 10.137.56.43:1984 TIME_WAIT
...

10.137.56.43 is the Xymon server IP.

$ netstat -atn|grep 1984|wc -l
165
(sometimes much much more)

(But goes down to 7 after some mins)

From: Thomas Eckert <user-2a86d6cd6326@xymon.invalid>
To: xymon <xymon at xymon.com>
Date: 19/10/2017 11:14
Subject: Re: [Xymon] no client results arrive, lot of TIME_WAIT at server
Sent by: "Xymon" <xymon-bounces at xymon.com>


Norbert,

I assume your manual (BB way) is something like `xymon 1.2.3.4 “status example.com.test …”`.

The xymon-client send `clientdata` that is processed by a “channel listener” and _that_ sends status-messages similar to above (a bit simplified).

That channel-listener may not be running/crashing:
Check our `tasks.cfg` for the section `[clientdata]` and verify, if the `xymond_channel`-process is running (I suspect it’s not). Maybe the `clientdata.log` helps with debugging. You may also run the task manually to get further information.

What’s your problem with the "lot of TIME_WAIT”? Are these just for tcp/1984-connections?

Cheers
Thomas
On 19 Oct 2017, at 07:30, Norbert Kriegenburg <user-501bbe9c5409@xymon.invalid <mailto:user-501bbe9c5409@xymon.invalid>> wrote:
experts,

i have a new (not my first) Xymon installation (4.3.28 on RH 7.4, self compiled).
All looks good, but the clients only show the network tests (conn, ssh). Only the Xymon server itself shows everything (disk, cpu,...).
- all clients are RH 7.4, same install image used (i just copied over the client part with client config, create a cronjob for automatic start, so no root access nor extra install needed)
- nothing suspicious in server or client logs (even in debug mode)
- no ghosts
- if i send a test manually (the old BB way) this shows at Xymon immediately (so the client settings are ok)
- if i shutdown the xymond at server, the client logs complains:

2017-10-19 05:09:18.051994 Whoops ! Failed to send message (Connection failed)

(so obviously they communicate)

Querying the xymond shows:
xymon 127.0.0.1 "query clientname.ssh"
green <!-- [flags:OrdastLe] --> Thu Oct 19 05:21:29 2017 ssh ok
is ok, but
xymon 127.0.0.1 "query clientname.disk"
shows nothing

Any ideas?

Norbert

From:	Thomas Eckert <user-2a86d6cd6326@xymon.invalid>
To:	xymon <xymon at xymon.com>
Date:	19/10/2017 16:42
Subject:	Re: [Xymon] no client results arrive, lot of TIME_WAIT at
            server
Sent by:	"Xymon" <xymon-bounces at xymon.com>


      On 19 Oct 2017, at 14:49, Norbert Kriegenburg <
      user-501bbe9c5409@xymon.invalid> wrote:


      Yes,
      that's what i checked at first. But i assume that also the info from
      the servers own client data are captured by the same client channel
      listener, and that obviously works.


valid point


      Nothing in the clientdata.log, even if i activate the DEBUG mode with
      a
      kill -USR2 PID

      My clientdata section:
      # The client back-end module. You need this if you are running the
      Xymon client on any system.
      [clientdata]
      ENVFILE $XYMONSERVERHOME/etc/xymonserver.cfg
      NEEDS xymond
      CMD /home/xymon/server/bin/xymond_channel --channel=client \
      --log=$XYMONSERVERLOGS/clientdata.log xymond_client \
      --no-clear-files \
      --no-clear-ports \
      --no-port-listing \
      --uptime-status
      INTERVAL 2m


The `INTERVAL` here is not im my and xymon.com’s-version of tasks.cfg— not
sure if it does harm (it _may_ try to start a new channel every 2 minutes).


      ps listing:
      $ ps -ef|grep "xymond_channel --channel=client"|grep -v grep
      xymon 18865 18705 0 05:13 ?
      00:00:00 /home/xymon/server/bin/xymond_channel --channel=client
      --log=/home/xymon/server/logs/clientdata.log xymond_client
      --no-clear-files --no-clear-ports --no-port-listing --uptime-status


If this process does not change it should be fine; your xymon server was
restarted 5:13 this morning?


      I have a lot of TIME_WAITS from time to time, no idea why this goes
      up and down:

      $ netstat -atn|grep 1984
      tcp 0 0 0.0.0.0:1984 0.0.0.0:* LISTEN
      tcp 0 0 10.137.56.43:37752 10.137.56.43:1984 TIME_WAIT
      tcp 0 0 10.137.56.43:37822 10.137.56.43:1984 TIME_WAIT
      ...

      10.137.56.43 is the Xymon server IP.

      $ netstat -atn|grep 1984|wc -l
      165
      (sometimes much much more)

      (But goes down to 7 after some mins)


Looks like internal xymon-communication. As long as the TIME_WAIT-count
does not go >20k  that should not be the cause.

Thing’s I’d try from here:

1. double-check the XYMSRV / XYMONSERVERS setting on the client(s), e.g.
with:

xymoncmd /usr/bin/env | egrep -i 'xymsrv|xymonservers|bbdisp’

2. just to double-check: tcpdump on xymon server and a client to check for
tcp/1984 communication

3. run a channel-listener an the server to see if the client messages
(other than the ones from the local client) make it

xymoncmd xymond_channel --channel=client cat | grep ‘^client '
xymoncmd xymond_channel --channel=client cat

The latter might be very noisy — the former just prints the “header”/1st
line of each connection.
There should be one message for each client sending data every 5 minutes.

HTH
Thomas


      <graycol.gif>Thomas Eckert ---19/10/2017 11:14:56---Norbert, I assume
      your manual (BB way) is something like `xymon 1.2.3.4 “status
      example.com.test …”`

      From: Thomas Eckert <user-2a86d6cd6326@xymon.invalid>
      To: xymon <xymon at xymon.com>
      Date: 19/10/2017 11:14
      Subject: Re: [Xymon] no client results arrive, lot of TIME_WAIT at
      server
      Sent by: "Xymon" <xymon-bounces at xymon.com>


      Norbert,

      I assume your manual (BB way) is something like `xymon 1.2.3.4
      “status example.com.test …”`.

      The xymon-client send `clientdata` that is processed by a “channel
      listener” and _that_ sends status-messages similar to above (a bit
      simplified).

      That channel-listener may not be running/crashing:
      Check our `tasks.cfg` for the section `[clientdata]` and verify, if
      the `xymond_channel`-process is running (I suspect it’s not). Maybe
      the `clientdata.log` helps with debugging. You may also run the task
      manually to get further information.

      What’s your problem with the "lot of TIME_WAIT”? Are these just for
      tcp/1984-connections?

      Cheers
      Thomas
                  On 19 Oct 2017, at 07:30, Norbert Kriegenburg <
                  user-501bbe9c5409@xymon.invalid> wrote:


                  experts,

                  i have a new (not my first) Xymon installation (4.3.28 on
                  RH 7.4, self compiled).
                  All looks good, but the clients only show the network
                  tests (conn, ssh). Only the Xymon server itself shows
                  everything (disk, cpu,...).
                  - all clients are RH 7.4, same install image used (i just
                  copied over the client part with client config, create a
                  cronjob for automatic start, so no root access nor extra
                  install needed)
                  - nothing suspicious in server or client logs (even in
                  debug mode)
                  - no ghosts
                  - if i send a test manually (the old BB way) this shows
                  at Xymon immediately (so the client settings are ok)
                  - if i shutdown the xymond at server, the client logs
                  complains:

                  2017-10-19 05:09:18.051994 Whoops ! Failed to send
                  message (Connection failed)
                  2017-10-19 05:09:18.052104 -> Could not connect to Xymon
                  daemon at 10.137.56.43:1984 (Connection refused)

                  (so obviously they communicate)

                  Querying the xymond shows:
                  xymon 127.0.0.1 "query clientname.ssh"
                  green <!-- [flags:OrdastLe] --> Thu Oct 19 05:21:29 2017
                  ssh ok
                  is ok, but
                  xymon 127.0.0.1 "query clientname.disk"
                  shows nothing

                  Any ideas?

                  Norbert

Hi Thomas,

thank you for your hints.
Lets go through your suggestions:

- you were right with the INTERVAL. This was added by accident, but doesn't harm (no more channels are started), removed, no effect
- yes, server process was restartet at 5:13am UTC

1.
- environment looks as expected:
$ xymoncmd /usr/bin/env | egrep -i 'xymsrv|xymonservers|bbdisp'
XYMONSERVERS=10.137.56.43
XYMSRV=10.137.56.43
BBDISP=10.137.56.43
BBDISPLAYS=10.137.56.43

2.
- unfortunately i don't have root access here for a tcpdump

3.
- now it gets strange:
if i try your suggestion i get this (doublechecked at other server and the syntax works ok):

$ xymoncmd xymond_channel --channel=client cat
2017-10-19 15:58:58.843500 execvp() failed: No such file or directory
(but of curse xymoncmd is in the path)

if i try with relative path i get this (run from bin path):
$ ./xymoncmd ./xymond_channel --channel=client cat
2017-10-19 16:01:02.173601 Could not get shm of size 524288: No such file or directory
2017-10-19 16:01:02.173943 Channel not available

some of these messages i can find in some logs:

$ grep "Could not get shm" *
alert.log:2017-10-19 10:38:00.733526 Could not get shm of size 1048576: No such file or directory
history.log:2017-10-19 10:38:00.730686 Could not get shm of size 1048576: No such file or directory
hostdata.log:2017-10-19 10:38:00.730538 Could not get shm of size 524288: No such file or directory
rrd-data.log:2017-10-19 10:38:00.734198 Could not get shm of size 524288: No such file or directory

I searched the web for those messages, and found some in hobbit/xymon mailing list, but no details or solution.
http://lists.xymon.com/oldarchive/2008/06/msg00183.html

But: the client messages from server still arrive and the tests are all visible...
very strange

Norbert

From: Thomas Eckert <user-2a86d6cd6326@xymon.invalid>
To: xymon <xymon at xymon.com>
Date: 19/10/2017 16:42
Subject: Re: [Xymon] no client results arrive, lot of TIME_WAIT at server
Sent by: "Xymon" <xymon-bounces at xymon.com>


On 19 Oct 2017, at 14:49, Norbert Kriegenburg <user-501bbe9c5409@xymon.invalid<mailto:user-501bbe9c5409@xymon.invalid>> wrote:

Yes,
that's what i checked at first. But i assume that also the info from the servers own client data are captured by the same client channel listener, and that obviously works.

valid point

Nothing in the clientdata.log, even if i activate the DEBUG mode with a
kill -USR2 PID

My clientdata section:
# The client back-end module. You need this if you are running the Xymon client on any system.
[clientdata]
ENVFILE $XYMONSERVERHOME/etc/xymonserver.cfg
NEEDS xymond
CMD /home/xymon/server/bin/xymond_channel --channel=client \
--log=$XYMONSERVERLOGS/clientdata.log xymond_client \
--no-clear-files \
--no-clear-ports \
--no-port-listing \
--uptime-status
INTERVAL 2m

ps listing:
$ ps -ef|grep "xymond_channel --channel=client"|grep -v grep
xymon 18865 18705 0 05:13 ? 00:00:00 /home/xymon/server/bin/xymond_channel --channel=client --log=/home/xymon/server/logs/clientdata.log xymond_client --no-clear-files --no-clear-ports --no-port-listing --uptime-status

If this process does not change it should be fine; your xymon server was restarted 5:13 this morning?

I have a lot of TIME_WAITS from time to time, no idea why this goes up and down:

$ netstat -atn|grep 1984
tcp 0 0 0.0.0.0:1984 0.0.0.0:* LISTEN
tcp 0 0 10.137.56.43:37752 10.137.56.43:1984 TIME_WAIT
tcp 0 0 10.137.56.43:37822 10.137.56.43:1984 TIME_WAIT
...

10.137.56.43 is the Xymon server IP.

$ netstat -atn|grep 1984|wc -l
165
(sometimes much much more)

(But goes down to 7 after some mins)

Looks like internal xymon-communication. As long as the TIME_WAIT-count does not go >20k that should not be the cause.

Thing’s I’d try from here:

1. double-check the XYMSRV / XYMONSERVERS setting on the client(s), e.g. with:

xymoncmd /usr/bin/env | egrep -i 'xymsrv|xymonservers|bbdisp’

2. just to double-check: tcpdump on xymon server and a client to check for tcp/1984 communication

3. run a channel-listener an the server to see if the client messages (other than the ones from the local client) make it

xymoncmd xymond_channel --channel=client cat | grep ‘^client '
xymoncmd xymond_channel --channel=client cat

The latter might be very noisy — the former just prints the “header”/1st line of each connection.
There should be one message for each client sending data every 5 minutes.

HTH
Thomas

<graycol.gif>Thomas Eckert ---19/10/2017 11:14:56---Norbert, I assume your manual (BB way) is something like `xymon 1.2.3.4 “status example.com.test …”`

From: Thomas Eckert <user-2a86d6cd6326@xymon.invalid<mailto:user-2a86d6cd6326@xymon.invalid>>
To: xymon <xymon at xymon.com<mailto:xymon at xymon.com>>
Date: 19/10/2017 11:14
Subject: Re: [Xymon] no client results arrive, lot of TIME_WAIT at server

Norbert,

I assume your manual (BB way) is something like `xymon 1.2.3.4 “status example.com.test …”`.

The xymon-client send `clientdata` that is processed by a “channel listener” and _that_ sends status-messages similar to above (a bit simplified).

That channel-listener may not be running/crashing:
Check our `tasks.cfg` for the section `[clientdata]` and verify, if the `xymond_channel`-process is running (I suspect it’s not). Maybe the `clientdata.log` helps with debugging. You may also run the task manually to get further information.

What’s your problem with the "lot of TIME_WAIT”? Are these just for tcp/1984-connections?

Cheers
Thomas
On 19 Oct 2017, at 07:30, Norbert Kriegenburg <user-501bbe9c5409@xymon.invalid<mailto:user-501bbe9c5409@xymon.invalid>> wrote:

experts,

i have a new (not my first) Xymon installation (4.3.28 on RH 7.4, self compiled).
All looks good, but the clients only show the network tests (conn, ssh). Only the Xymon server itself shows everything (disk, cpu,...).
- all clients are RH 7.4, same install image used (i just copied over the client part with client config, create a cronjob for automatic start, so no root access nor extra install needed)
- nothing suspicious in server or client logs (even in debug mode)
- no ghosts
- if i send a test manually (the old BB way) this shows at Xymon immediately (so the client settings are ok)
- if i shutdown the xymond at server, the client logs complains:

2017-10-19 05:09:18.051994 Whoops ! Failed to send message (Connection failed)

(so obviously they communicate)

Querying the xymond shows:
xymon 127.0.0.1 "query clientname.ssh"
green <!-- [flags:OrdastLe] --> Thu Oct 19 05:21:29 2017 ssh ok
is ok, but
xymon 127.0.0.1 "query clientname.disk"
shows nothing

Any ideas?

Norbert