Xymon Mailing List Archive search

log monitoring hostname matching trouble

5 messages in this thread

list Schminke_Erik_D · Fri, 8 Jan 2016 09:54:52 -0600 · 
Hello,

I'm trying to get log monitoring to work on a new implementation of Xymon
(v4.3.24)

For whatever reason, my client doesn't seem to be receiving a logfetch.cfg
file from the server after it checks in.  I've tried a number of
combinations of hostname and none of them seem to work:

[lpcobwas2p] (hostname only)
[lpcobwas2p.hfc.ad] (FQDN, periods)
[lpcobwas2p,hfc,ad] (FQDN, commas)

None of them seem to work.

Any ideas?

Server side:

[root at lxcobxymonp server]# pwd
/home/xymon/server
[root at lxcobxymonp server]# hostname
lxcobxymonp.hfc.ad
[root at lxcobxymonp server]#

[root at lxcobxymonp server]# tail -2 etc/client-local.cfg
[lpcobwas2p]
log:/var/log/messages:20480
[root at lxcobxymonp server]#

[root at lxcobxymonp server]# grep lpcobwas2p etc/hosts.cfg
0.0.0.0         lpcobwas2p # CLIENT:lpcobwas2p.hfc.ad
[root at lxcobxymonp server]#

Client side:

[root at lpcobwas2p client]# pwd
/home/xymon/client
[root at lpcobwas2p client]# hostname
lpcobwas2p.hfc.ad
[root at lpcobwas2p client]#

[root at lpcobwas2p client]# head -2 tmp/msg.lpcobwas2p.hfc.ad.txt
@@client#1|1|127.0.0.1|lpcobwas2p.hfc.ad|linux
client lpcobwas2p,hfc,ad.linux linux
[root at lpcobwas2p client]#


Erik D. Schminke | Associate Systems Programmer
Hormel Foods Corporation | One Hormel Way | Austin, MN XXXXX
Phone: (XXX) XXX-XXXX
user-15513f33c451@xymon.invalid | www.hormelfoods.com
list Schminke_Erik_D · Wed, 13 Jan 2016 08:43:14 -0600 · 
Hi Jeremy,

Thanks for your reply.

First, I am aware of the 10 minute interval.  I did observe that when
working with an AIX system.  I mention that because I am able to get log
monitoring  to work there, but not with a Linux system.  That is to say, I
do get a "logfetch.[hostname].cfg" file in $XYMONHOME/tmp, and changes come
in approx ten minutes after being made on the XYMON server at
$XYMONHOME/etc/client-local.cfg

Is there anything I can do to reduce the 10 minute interval for testing
purposes?

Entering "xymoncmd"

[root at lpcobwas2p ~]# su - xymon
-bash-4.1$ export XYMONCLIENTHOME=/home/xymon/client
-bash-4.1$ client/bin/xymoncmd
sh-4.1$ whoami
xymon
sh-4.1$ pwd
/home/xymon
sh-4.1$

----
All files in /home/xymon are owned by xymon, group xymon and are writable

sh-4.1$ find . ! -writable
sh-4.1$ find . ! -user xymon
sh-4.1$ find . ! -group xymon
sh-4.1$


$XYMONTMP/msg.$(hostname).txt is getting updated every five minutes, as
expected:

sh-4.1$ ls -la client/tmp/msg.$(hostname).txt
-rw-rw-r--. 1 xymon xymon 152566 Jan 13 08:00
client/tmp/msg.lpcobwas2p.hfc.ad.txt
sh-4.1$ ls -la client/tmp/msg.$(hostname).txt
-rw-rw-r--. 1 xymon xymon 148917 Jan 13 08:05
client/tmp/msg.lpcobwas2p.hfc.ad.txt
sh-4.1$

I sent this file manually to the xymon server, but got no output

sh-4.1$ $XYMON $XYMSRV "@" < $XYMONTMP/msg.lpcobwas2p.hfc.ad.txt
sh-4.1$

Sending your suggested test string yields the following:

sh-4.1$ $XYMON $XYMSRV "@" << EOT
client $MACHINE.linux linux
EOT
log:/var/log/messages:10240
ignore MARK
sh-4.1$ echo $MACHINE
lpcobwas2p,hfc,ad
sh-4.1$

Interesting... but a) why doesn't that come back with the message file and
b) it still doesn't match the hostname


Lastly.. I'm still not clear on the correct format of a hostname definition
for client/etc/client-local.cfg

Should it be

lpcobwas2p (just hostname)
lpcobwas2p.hfc.ad (period)
lpcobwas2p,hfc,ad (commas)

Thanks,
▸ quoted from Schminke_Erik_D


Erik D. Schminke | Associate Systems Programmer
Hormel Foods Corporation | One Hormel Way | Austin, MN XXXXX
Phone: (XXX) XXX-XXXX
user-15513f33c451@xymon.invalid | www.hormelfoods.com


From:	Jeremy Laidman <user-71895fb2e44c@xymon.invalid>
To:	user-15513f33c451@xymon.invalid
Date:	01/12/2016 08:03 PM
Subject:	Re: [Xymon] log monitoring hostname matching trouble



Erik

Hmm. It's hard to know what might be wrong.  First, note that it can take
some time (10 minutes maybe) for the cfg file to appear.

Check your ~xymon/client/tmp/ directory and ensure that the xymon user can
write there (probably should have owner and group both as "xymon").

Then, see if you have a client message temp file there with something like

ls -l ~xymon/client/tmp/msg.*.txt

This should show the hostname that the Xymon client is using, as part of
the filename.  It will also show the permissions, which should make the
file writeable by the xymon user.  Also, check the first line of the file
(eg: head -1 ~xymon/client/tmp/msg.*.txt) and see if the hostname listed
there is what you're expecting.

Try sending the msg.*.txt file via the xymon client message, and see if you
get the cfg file as output:

$ ~xymon/client/bin/xymoncmd
$ $XYMON $XYMSRV @ < $XYMONTMP/msg.*.txt
$ exit

Or even just:

$ ~xymon/client/bin/xymoncmd
$ echo "client $MACHINE.linux linux" | $XYMON $XYMSRV @

(Substitute "linux" for your OS.)

Cheers
Jeremy
▸ quoted from Schminke_Erik_D

On Sat, Jan 9, 2016 at 3:10 AM <user-15513f33c451@xymon.invalid> wrote:

  Hello,

  I'm trying to get log monitoring to work on a new implementation of Xymon
  (v4.3.24)

  For whatever reason, my client doesn't seem to be receiving a
  logfetch.cfg
  file from the server after it checks in.  I've tried a number of
  combinations of hostname and none of them seem to work:

  [lpcobwas2p] (hostname only)
  [lpcobwas2p.hfc.ad] (FQDN, periods)
  [lpcobwas2p,hfc,ad] (FQDN, commas)

  None of them seem to work.

  Any ideas?

  Server side:

  [root at lxcobxymonp server]# pwd
  /home/xymon/server
  [root at lxcobxymonp server]# hostname
  lxcobxymonp.hfc.ad
  [root at lxcobxymonp server]#

  [root at lxcobxymonp server]# tail -2 etc/client-local.cfg
  [lpcobwas2p]
  log:/var/log/messages:20480
  [root at lxcobxymonp server]#

  [root at lxcobxymonp server]# grep lpcobwas2p etc/hosts.cfg
  0.0.0.0         lpcobwas2p # CLIENT:lpcobwas2p.hfc.ad
  [root at lxcobxymonp server]#

  Client side:

  [root at lpcobwas2p client]# pwd
  /home/xymon/client
  [root at lpcobwas2p client]# hostname
  lpcobwas2p.hfc.ad
  [root at lpcobwas2p client]#

  [root at lpcobwas2p client]# head -2 tmp/msg.lpcobwas2p.hfc.ad.txt
  @@client#1|1|127.0.0.1|lpcobwas2p.hfc.ad|linux
  client lpcobwas2p,hfc,ad.linux linux
  [root at lpcobwas2p client]#


  Erik D. Schminke | Associate Systems Programmer
  Hormel Foods Corporation | One Hormel Way | Austin, MN XXXXX
  Phone: (XXX) XXX-XXXX
  user-15513f33c451@xymon.invalid | www.hormelfoods.com
list Japheth Cleaver · Wed, 13 Jan 2016 07:57:00 -0800 · 
Hi Ed,

The 10 minute interval is actually an artifact of the run frequency (1x to
download the updated file, but not used until the next run). For testing,
or any other purpose, simply increase the 'INTERVAL' in the client section
of clientlaunch.cfg to something more frequent than that. (I tend to use
100s instead of 5m.)

There's nothing sacrosanct about that interval, so as long as your systems
can handle additional traffic and incoming message load, it can be lowered
below 5m as desired (shouldn't be raised higher than 5m). The only real
negative side effect is that you'll see more "process spam" in the form of
simultaneous backgrounded vmstat processes.


HTH,
-jc
▸ quoted from Schminke_Erik_D


On Wed, January 13, 2016 6:43 am, user-15513f33c451@xymon.invalid wrote:
Hi Jeremy,

Thanks for your reply.

First, I am aware of the 10 minute interval.  I did observe that when
working with an AIX system.  I mention that because I am able to get log
monitoring  to work there, but not with a Linux system.  That is to say, I
do get a "logfetch.[hostname].cfg" file in $XYMONHOME/tmp, and changes
come
in approx ten minutes after being made on the XYMON server at
$XYMONHOME/etc/client-local.cfg

Is there anything I can do to reduce the 10 minute interval for testing
purposes?

Entering "xymoncmd"

[root at lpcobwas2p ~]# su - xymon
-bash-4.1$ export XYMONCLIENTHOME=/home/xymon/client
-bash-4.1$ client/bin/xymoncmd
sh-4.1$ whoami
xymon
sh-4.1$ pwd
/home/xymon
sh-4.1$

----
All files in /home/xymon are owned by xymon, group xymon and are writable

sh-4.1$ find . ! -writable
sh-4.1$ find . ! -user xymon
sh-4.1$ find . ! -group xymon
sh-4.1$


$XYMONTMP/msg.$(hostname).txt is getting updated every five minutes, as
expected:

sh-4.1$ ls -la client/tmp/msg.$(hostname).txt
-rw-rw-r--. 1 xymon xymon 152566 Jan 13 08:00
client/tmp/msg.lpcobwas2p.hfc.ad.txt
sh-4.1$ ls -la client/tmp/msg.$(hostname).txt
-rw-rw-r--. 1 xymon xymon 148917 Jan 13 08:05
client/tmp/msg.lpcobwas2p.hfc.ad.txt
sh-4.1$

I sent this file manually to the xymon server, but got no output

sh-4.1$ $XYMON $XYMSRV "@" < $XYMONTMP/msg.lpcobwas2p.hfc.ad.txt
sh-4.1$

Sending your suggested test string yields the following:

sh-4.1$ $XYMON $XYMSRV "@" << EOT
client $MACHINE.linux linux
EOT
log:/var/log/messages:10240
ignore MARK
sh-4.1$ echo $MACHINE
lpcobwas2p,hfc,ad
sh-4.1$

Interesting... but a) why doesn't that come back with the message file and
b) it still doesn't match the hostname


Lastly.. I'm still not clear on the correct format of a hostname
definition
for client/etc/client-local.cfg

Should it be

lpcobwas2p (just hostname)
lpcobwas2p.hfc.ad (period)
lpcobwas2p,hfc,ad (commas)

Thanks,


Erik D. Schminke | Associate Systems Programmer
Hormel Foods Corporation | One Hormel Way | Austin, MN XXXXX
Phone: (XXX) XXX-XXXX
user-15513f33c451@xymon.invalid | www.hormelfoods.com


From:	Jeremy Laidman <user-71895fb2e44c@xymon.invalid>
To:	user-15513f33c451@xymon.invalid
Date:	01/12/2016 08:03 PM
Subject:	Re: [Xymon] log monitoring hostname matching trouble


Erik


Hmm. It's hard to know what might be wrong.  First, note that it can take
▸ quoted from Schminke_Erik_D
some time (10 minutes maybe) for the cfg file to appear.

Check your ~xymon/client/tmp/ directory and ensure that the xymon user can
write there (probably should have owner and group both as "xymon").

Then, see if you have a client message temp file there with something like

ls -l ~xymon/client/tmp/msg.*.txt

This should show the hostname that the Xymon client is using, as part of

the filename.  It will also show the permissions, which should make the
file writeable by the xymon user.  Also, check the first line of the file
▸ quoted from Schminke_Erik_D
(eg: head -1 ~xymon/client/tmp/msg.*.txt) and see if the hostname listed
there is what you're expecting.

Try sending the msg.*.txt file via the xymon client message, and see if
you
get the cfg file as output:

$ ~xymon/client/bin/xymoncmd
$ $XYMON $XYMSRV @ < $XYMONTMP/msg.*.txt
$ exit

Or even just:

$ ~xymon/client/bin/xymoncmd
$ echo "client $MACHINE.linux linux" | $XYMON $XYMSRV @

(Substitute "linux" for your OS.)

Cheers
Jeremy

On Sat, Jan 9, 2016 at 3:10 AM <user-15513f33c451@xymon.invalid> wrote:

  Hello,

  I'm trying to get log monitoring to work on a new implementation of
Xymon
  (v4.3.24)

  For whatever reason, my client doesn't seem to be receiving a
  logfetch.cfg

  file from the server after it checks in.  I've tried a number of
▸ quoted from Schminke_Erik_D
  combinations of hostname and none of them seem to work:

  [lpcobwas2p] (hostname only)
  [lpcobwas2p.hfc.ad] (FQDN, periods)
  [lpcobwas2p,hfc,ad] (FQDN, commas)

  None of them seem to work.

  Any ideas?

  Server side:

  [root at lxcobxymonp server]# pwd
  /home/xymon/server
  [root at lxcobxymonp server]# hostname
  lxcobxymonp.hfc.ad
  [root at lxcobxymonp server]#

  [root at lxcobxymonp server]# tail -2 etc/client-local.cfg
  [lpcobwas2p]
  log:/var/log/messages:20480
  [root at lxcobxymonp server]#

  [root at lxcobxymonp server]# grep lpcobwas2p etc/hosts.cfg

  0.0.0.0         lpcobwas2p # CLIENT:lpcobwas2p.hfc.ad
▸ quoted from Schminke_Erik_D
  [root at lxcobxymonp server]#

  Client side:

  [root at lpcobwas2p client]# pwd
  /home/xymon/client
  [root at lpcobwas2p client]# hostname
  lpcobwas2p.hfc.ad
  [root at lpcobwas2p client]#

  [root at lpcobwas2p client]# head -2 tmp/msg.lpcobwas2p.hfc.ad.txt
  @@client#1|1|127.0.0.1|lpcobwas2p.hfc.ad|linux
  client lpcobwas2p,hfc,ad.linux linux
  [root at lpcobwas2p client]#


  Erik D. Schminke | Associate Systems Programmer
  Hormel Foods Corporation | One Hormel Way | Austin, MN XXXXX
  Phone: (XXX) XXX-XXXX
  user-15513f33c451@xymon.invalid | www.hormelfoods.com
list Jeremy Laidman · Thu, 14 Jan 2016 03:41:14 +0000 ·
▸ quoted from Japheth Cleaver
On Thu, Jan 14, 2016 at 1:43 AM <user-15513f33c451@xymon.invalid> wrote:

Is there anything I can do to reduce the 10 minute interval for testing
purposes?

You can do as JC said.  Alternatively, manually run the xymonclient.sh
process (under xymoncmd as xymon user) and it should immediately pick up
any changes from the server.  This is a bit of an overkill for simply
fetching the update, and I don't know how it might interfere with the
normal process of running the client every 5 minutes.  So you can instead
do: echo "client $MACHINE.linux linux" | $XYMON $XYMSRV >
/path/to/logfetch.$MACHINE.cfg

I've never really been in enough of a hurry to do this, and I simply waited
out the 5+5 minutes.
▸ quoted from Japheth Cleaver

I sent this file manually to the xymon server, but got no output
sh-4.1$ $XYMON $XYMSRV "@" < $XYMONTMP/msg.lpcobwas2p.hfc.ad.txt
sh-4.1$

Hmm.  Check out the first line of the file, see if it's different from the
"EOT" command-line you used.
▸ quoted from Japheth Cleaver


Sending your suggested test string yields the following:

sh-4.1$ $XYMON $XYMSRV "@" << EOT
client $MACHINE.linux linux
EOT
log:/var/log/messages:10240
ignore MARK

That's a good sign.  I think this proves that all is OK on the Xymon server
(except possibly a mismatch of the hostname somehow).
▸ quoted from Japheth Cleaver


sh-4.1$ echo $MACHINE
lpcobwas2p,hfc,ad
sh-4.1$

Interesting... but a) why doesn't that come back with the message file and

Yes, that is indeed the crux of the problem.  What you need to do is to
modify the command-line to be close to the file's first line, until it
fails.  Or modify the file's first line close to the command-line until it
works.  Then the last thing you change is where the problem is.


b) it still doesn't match the hostname

Exactly which hostname are you referring to?  The hostname in hosts.cfg?
The hostname in the CLIENT: specification in hosts.cfg?  The output of
"uname -n" on the client?  Or $MACHINE on the client?
▸ quoted from Japheth Cleaver

Lastly.. I'm still not clear on the correct format of a hostname definition
for client/etc/client-local.cfg

For the client config to be fetched correctly, the important thing is that
$MACHINE on the client, under xymoncmd, is a match to the entry in the
client-local.cfg file, and that's all.  It doesn't matter what appears in
the hosts.cfg file.  The client gives xymond its name (and OS), and xymond
gives the client the matching section - the hostname if it is listed,
otherwise the OS name.


Should it be

lpcobwas2p (just hostname)
lpcobwas2p.hfc.ad (period)
lpcobwas2p,hfc,ad (commas)

Based on your commands showing $MACHINE, and the name of the msg file, it
should be the middle entry, FQDN with periods.  But you said none of these
works.

Cheers
Jeremy
list Schminke_Erik_D · Mon, 18 Jan 2016 13:01:33 -0600 · 
I think I've got this figured out.  It's now behaving as expected.

The root of my problem seemed to be the fact that the xymon client on the
monitored system was running in "local mode".  I removed the "--local" from
the CMD line of the [client] block in $XYMONCLIENTHOME/etc/clientlaunch.cfg

The existence of this line in my msg file was what clued me to that; this
line no longer exists now that the client message.
@@client#1|1|127.0.0.1|lpcobwas2p.hfc.ad|linux

Once I cleared that up, I determined that when trying to match hostname in
client-local.cfg, it matches based on what you have defined in hosts.cfg

Server:
[root at lxcobxymonp etc]# grep lpcobwas2p hosts.cfg
0.0.0.0         lpcobwas2p # CLIENT:lpcobwas2p.hfc.ad
[root at lxcobxymonp etc]# sed -n '70,71 p' client-local.cfg
[lpcobwas2p]
log:/var/log/messages:9997
[root at lxcobxymonp etc]#

Thanks everyone for your help!
▸ signature


Erik D. Schminke | Associate Systems Programmer
Hormel Foods Corporation | One Hormel Way | Austin, MN XXXXX
Phone: (XXX) XXX-XXXX
user-15513f33c451@xymon.invalid | www.hormelfoods.com
▸ quoted from Jeremy Laidman
From:	Jeremy Laidman <user-71895fb2e44c@xymon.invalid>
To:	user-15513f33c451@xymon.invalid
Date:	01/12/2016 08:03 PM
Subject:	Re: [Xymon] log monitoring hostname matching trouble


Erik

Hmm. It's hard to know what might be wrong.  First, note that it can take
some time (10 minutes maybe) for the cfg file to appear.

Check your ~xymon/client/tmp/ directory and ensure that the xymon user can
write there (probably should have owner and group both as "xymon").

Then, see if you have a client message temp file there with something like

ls -l ~xymon/client/tmp/msg.*.txt

This should show the hostname that the Xymon client is using, as part of
the filename.  It will also show the permissions, which should make the
file writeable by the xymon user.  Also, check the first line of the file
(eg: head -1 ~xymon/client/tmp/msg.*.txt) and see if the hostname listed
there is what you're expecting.

Try sending the msg.*.txt file via the xymon client message, and see if you
get the cfg file as output:

$ ~xymon/client/bin/xymoncmd
$ $XYMON $XYMSRV @ < $XYMONTMP/msg.*.txt
$ exit

Or even just:

$ ~xymon/client/bin/xymoncmd
$ echo "client $MACHINE.linux linux" | $XYMON $XYMSRV @

(Substitute "linux" for your OS.)

Cheers
Jeremy

On Sat, Jan 9, 2016 at 3:10 AM <user-15513f33c451@xymon.invalid> wrote:

  Hello,

  I'm trying to get log monitoring to work on a new implementation of Xymon
  (v4.3.24)

  For whatever reason, my client doesn't seem to be receiving a
  logfetch.cfg
  file from the server after it checks in.  I've tried a number of
  combinations of hostname and none of them seem to work:

  [lpcobwas2p] (hostname only)
  [lpcobwas2p.hfc.ad] (FQDN, periods)
  [lpcobwas2p,hfc,ad] (FQDN, commas)

  None of them seem to work.

  Any ideas?

  Server side:

  [root at lxcobxymonp server]# pwd
  /home/xymon/server
  [root at lxcobxymonp server]# hostname
  lxcobxymonp.hfc.ad
  [root at lxcobxymonp server]#

  [root at lxcobxymonp server]# tail -2 etc/client-local.cfg
  [lpcobwas2p]
  log:/var/log/messages:20480
  [root at lxcobxymonp server]#

  [root at lxcobxymonp server]# grep lpcobwas2p etc/hosts.cfg
  0.0.0.0         lpcobwas2p # CLIENT:lpcobwas2p.hfc.ad
  [root at lxcobxymonp server]#

  Client side:

  [root at lpcobwas2p client]# pwd
  /home/xymon/client
  [root at lpcobwas2p client]# hostname
  lpcobwas2p.hfc.ad
  [root at lpcobwas2p client]#

  [root at lpcobwas2p client]# head -2 tmp/msg.lpcobwas2p.hfc.ad.txt
  @@client#1|1|127.0.0.1|lpcobwas2p.hfc.ad|linux
  client lpcobwas2p,hfc,ad.linux linux
  [root at lpcobwas2p client]#


  Erik D. Schminke | Associate Systems Programmer
  Hormel Foods Corporation | One Hormel Way | Austin, MN XXXXX
  Phone: (XXX) XXX-XXXX
  user-15513f33c451@xymon.invalid | www.hormelfoods.com