Serial alarm feeds
list Kevin King
Fellow xymoners,
I am looking for some ideas on how to monitor some alarm feeds (serial ROP).
I have to collect them from the xymon server, The platform involved cannot
support a xymon client, I have the collection part down to stream it to a
log file. But I want to use the log monitor feature for the clients, so I
can build the alerts and post the results under the hosts display on xymon.
If it helps here is a typical message:
HOSTNAME 13-08-24 06:28:04 CST EAGLE 44.0.4-64.34.24
7879.1056 CARD 5107 INFO SCCP rcvd inv SCMG - bad subsystem
SIO=a3 OPC= xxx-xxx-xxx DPC= xxx-xxx-xxx
SCMG: MSG TYPE=003 MSG LEN=006
AFTPC= 001-151-001 AFTSS=008 MULT=000
LSN=m209250004
Report Date:13-08-24 Time:06:28:02
;
Any thoughts or a different way to deal with it is welcome. Fresh eyes and
ideas help break out of my box.
Thanks
-Kevin
list Jeremy Laidman
Doesn't this device support SNMP? If so, you could perhaps do something using devmon. But I see no problem with what you've proposed. Should work just fine. J
▸
On 26 August 2013 22:34, Kevin King <user-4c6c31ddcab2@xymon.invalid> wrote:
Fellow xymoners,****
** **
I am looking for some ideas on how to monitor some alarm feeds (serial
ROP). I have to collect them from the xymon server, The platform involved
cannot support a xymon client, I have the collection part down to stream it
to a log file. But I want to use the log monitor feature for the clients,
so I can build the alerts and post the results under the hosts display on
xymon. ****
** **
If it helps here is a typical message:****
** **
HOSTNAME 13-08-24 06:28:04 CST EAGLE 44.0.4-64.34.24****
7879.1056 CARD 5107 INFO SCCP rcvd inv SCMG - bad subsystem*
***
SIO=a3 OPC= xxx-xxx-xxx DPC= xxx-xxx-xxx****
SCMG: MSG TYPE=003 MSG LEN=006****
AFTPC= 001-151-001 AFTSS=008 MULT=000****
LSN=m209250004****
Report Date:13-08-24 Time:06:28:02****
;****
** **
** **
** **
Any thoughts or a different way to deal with it is welcome. Fresh eyes and
ideas help break out of my box.****
** **
Thanks****
** **
-Kevin****
** **
list Jeremy Laidman
On 27 August 2013 13:31, KING, KEVIN <user-ca972c0c43a8@xymon.invalid> wrote:
No no snmp. It is a very closed system. The messages we get are out of a serial port to a term server.
:-(
I guess I do not know the client tool then. How can I save the files on my xymon server and use entries in the client config and alert rules to make if show up under the correct server? Maybe I am getting myself stuck in the mud, but if the file is on the xymon host how can I process it like it was on the client?
A server is almost always also a client. You should have the Xymon server
name in hosts.cfg, yes? Also, the Xymon server can be listed in
client-local.cfg and analysis.cfg. Just create appropriate entries in
those two files for the server and the logfile, and it will report the
errors as they appear.
The down-side to this is that the errors are assigned to the Xymon server.
If you want them assigned to the device name instead, then you will
probably have to do something a bit more complex, such as writing a custom
script on the Xymon server to do the log analysis and reporting. I don't
see a way that you can do this with the built-in log monitoring features.
Hmm, here's an idea. You could create a script that simply replicates the
log file into a client message, which would be a substitute for the
client-local.cfg entry. So then you'd be able to use the standard "LOG"
definitions in analysis.cfg to match whatever you want.
The easiest way to do this, I think, would be to run a second copy of
logfetch on the Xymon server (the standard one is usually run from
xymonclient.sh, from the [xymonclient] section of tasks.cfg). The logfetch
tool requires a configuration file and a state file as parameters.
Normally the config file is simply extracted from client-local.cfg and
pushed to the client during the client message interaction, but you can
simply create your own. For example (untested), create in $XYMONTMP the
file logfetch-special.cfg, containing the following:
log:/var/log/mylogfile:10240
ignore THIS
ignore THAT
Then add the following into tasks.cfg or a new file in tasks.d:
[xymonclient-special]
NEEDS xymond
ENVFILE $XYMONCLIENTHOME/etc/xymonclient.cfg
LOGFILE $XYMONCLIENTLOGS/xymonclient-special.log
INTERVAL 5m
CMD sh -c '( printf "eagle.example.com.eagle
eagle\n[msgs:/var/log/mylogfile]\n"; $XYMONCLIENTHOME/bin/logfetch
$XYMONTMP/logfetch-special.cfg $XYMONTMP/logfetch-special.status ) | $XYMON
$XYMSRV "@"'
This creates a header, using the device's hostname, and a [msgs:] section
for the logfile being monitored. Then it uses logfetch to get the latest
bunch of lines from the logfile, and filter out any "ignore" entries from
the config file logfetch-special.cfg. Finally, all of this is piped into
an instance of xymon that creates a client message and sends it to the
server.
The end result is a client message that should start to generate a "msgs"
dot against the device name (assuming you added the device name into
hosts.cfg - probably with a "noping" tag). If this works, you can then
start using analysis.cfg to have the Xymon server parse the log file from
the client data.
This is all a bit kludgy. You might prefer to extract the CMD bit into a
separate script, and just put the script name after the CMD keyword.
J
list Kevin King
Jeremy, This is good food for thought. I think I might need to run this as an external script. That way I can pass the host information, (I have more than one of these) Then I should be able to enter a hostname in the logfetch-special.cfg file. I then could do this for almost any collector that cannot use a local client. I seem to remember we had a tool like this way back in the BB days, I ether found something or I modified bb. Wish I still had that server! I built many a tool for BB. That is when I was on top of my game, I am a retread in this admin stuff. I look at some of the stuff I wrote in the early 2000s and I say wow nice code, what does it do? Thanks for the kick start! -Kevin
▸
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Jeremy Laidman
Sent: Tuesday, August 27, 2013 12:14 AM
To: KING, KEVIN; xymon at xymon.com
Subject: Re: [Xymon] Serial alarm feeds
On 27 August 2013 13:31, KING, KEVIN <user-ca972c0c43a8@xymon.invalid> wrote:
No no snmp. It is a very closed system. The messages we get are out of a
serial port to a term server.
:-(
▸
I guess I do not know the client tool then. How can I save the files on my
xymon server and use entries in the client config and alert rules to make
if show up under the correct server? Maybe I am getting myself stuck in the
mud, but if the file is on the xymon host how can I process it like it was
on the client?
A server is almost always also a client. You should have the Xymon server
name in hosts.cfg, yes? Also, the Xymon server can be listed in
client-local.cfg and analysis.cfg. Just create appropriate entries in those
two files for the server and the logfile, and it will report the errors as
they appear.
The down-side to this is that the errors are assigned to the Xymon server.
If you want them assigned to the device name instead, then you will probably
have to do something a bit more complex, such as writing a custom script on
the Xymon server to do the log analysis and reporting. I don't see a way
that you can do this with the built-in log monitoring features.
Hmm, here's an idea. You could create a script that simply replicates the
log file into a client message, which would be a substitute for the
client-local.cfg entry. So then you'd be able to use the standard "LOG"
definitions in analysis.cfg to match whatever you want.
The easiest way to do this, I think, would be to run a second copy of
logfetch on the Xymon server (the standard one is usually run from
xymonclient.sh, from the [xymonclient] section of tasks.cfg). The logfetch
tool requires a configuration file and a state file as parameters. Normally
the config file is simply extracted from client-local.cfg and pushed to the
client during the client message interaction, but you can simply create your
own. For example (untested), create in $XYMONTMP the file
logfetch-special.cfg, containing the following:
log:/var/log/mylogfile:10240
ignore THIS
ignore THAT
Then add the following into tasks.cfg or a new file in tasks.d:
[xymonclient-special]
NEEDS xymond
ENVFILE $XYMONCLIENTHOME/etc/xymonclient.cfg
LOGFILE $XYMONCLIENTLOGS/xymonclient-special.log
INTERVAL 5m
CMD sh -c '( printf "eagle.example.com.eagle
eagle\n[msgs:/var/log/mylogfile]\n"; $XYMONCLIENTHOME/bin/logfetch
$XYMONTMP/logfetch-special.cfg $XYMONTMP/logfetch-special.status ) | $XYMON
$XYMSRV "@"'
This creates a header, using the device's hostname, and a [msgs:] section
for the logfile being monitored. Then it uses logfetch to get the latest
bunch of lines from the logfile, and filter out any "ignore" entries from
the config file logfetch-special.cfg. Finally, all of this is piped into an
instance of xymon that creates a client message and sends it to the server.
The end result is a client message that should start to generate a "msgs"
dot against the device name (assuming you added the device name into
hosts.cfg - probably with a "noping" tag). If this works, you can then
start using analysis.cfg to have the Xymon server parse the log file from
the client data.
This is all a bit kludgy. You might prefer to extract the CMD bit into a
separate script, and just put the script name after the CMD keyword.
J
list Galen Johnson
Look on the front page of xymonton.org. There is a link to a "clientless" xymon client (xymon-rclient) that might do what you are looking for. =G=
▸
From: Xymon [xymon-bounces at xymon.com] on behalf of Kevin King [user-4c6c31ddcab2@xymon.invalid]
Sent: Tuesday, August 27, 2013 10:38 AM
To: 'Jeremy Laidman'; 'KING, KEVIN'; xymon at xymon.com
Subject: Re: [Xymon] Serial alarm feeds
Jeremy,
This is good food for thought. I think I might need to run this as an external script. That way I can pass the host information, (I have more than one of these) Then I should be able to enter a hostname in the logfetch-special.cfg file. I then could do this for almost any collector that cannot use a local client.
I seem to remember we had a tool like this way back in the BB days, I ether found something or I modified bb. Wish I still had that server! I built many a tool for BB. That is when I was on top of my game, I am a retread in this admin stuff. I look at some of the stuff I wrote in the early 2000s and I say wow nice code, what does it do?
Thanks for the kick start!
-Kevin
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Jeremy Laidman
Sent: Tuesday, August 27, 2013 12:14 AM
To: KING, KEVIN; xymon at xymon.com
Subject: Re: [Xymon] Serial alarm feeds
On 27 August 2013 13:31, KING, KEVIN <user-ca972c0c43a8@xymon.invalid<mailto:user-ca972c0c43a8@xymon.invalid>> wrote:
No no snmp. It is a very closed system. The messages we get are out of a serial port to a term server.
:-(
▸
I guess I do not know the client tool then. How can I save the files on my xymon server and use entries in the client config and alert rules to make if show up under the correct server? Maybe I am getting myself stuck in the mud, but if the file is on the xymon host how can I process it like it was on the client?
A server is almost always also a client. You should have the Xymon server name in hosts.cfg, yes? Also, the Xymon server can be listed in client-local.cfg and analysis.cfg. Just create appropriate entries in those two files for the server and the logfile, and it will report the errors as they appear.
The down-side to this is that the errors are assigned to the Xymon server. If you want them assigned to the device name instead, then you will probably have to do something a bit more complex, such as writing a custom script on the Xymon server to do the log analysis and reporting. I don't see a way that you can do this with the built-in log monitoring features.
Hmm, here's an idea. You could create a script that simply replicates the log file into a client message, which would be a substitute for the client-local.cfg entry. So then you'd be able to use the standard "LOG" definitions in analysis.cfg to match whatever you want.
The easiest way to do this, I think, would be to run a second copy of logfetch on the Xymon server (the standard one is usually run from xymonclient.sh, from the [xymonclient] section of tasks.cfg). The logfetch tool requires a configuration file and a state file as parameters. Normally the config file is simply extracted from client-local.cfg and pushed to the client during the client message interaction, but you can simply create your own. For example (untested), create in $XYMONTMP the file logfetch-special.cfg, containing the following:
log:/var/log/mylogfile:10240
ignore THIS
ignore THAT
Then add the following into tasks.cfg or a new file in tasks.d:
[xymonclient-special]
NEEDS xymond
ENVFILE $XYMONCLIENTHOME/etc/xymonclient.cfg
LOGFILE $XYMONCLIENTLOGS/xymonclient-special.log
INTERVAL 5m
CMD sh -c '( printf "eagle.example.com.eagle eagle\n[msgs:/var/log/mylogfile]\n"; $XYMONCLIENTHOME/bin/logfetch $XYMONTMP/logfetch-special.cfg $XYMONTMP/logfetch-special.status ) | $XYMON $XYMSRV "@"'
This creates a header, using the device's hostname, and a [msgs:] section for the logfile being monitored. Then it uses logfetch to get the latest bunch of lines from the logfile, and filter out any "ignore" entries from the config file logfetch-special.cfg. Finally, all of this is piped into an instance of xymon that creates a client message and sends it to the server.
The end result is a client message that should start to generate a "msgs" dot against the device name (assuming you added the device name into hosts.cfg - probably with a "noping" tag). If this works, you can then start using analysis.cfg to have the Xymon server parse the log file from the client data.
This is all a bit kludgy. You might prefer to extract the CMD bit into a separate script, and just put the script name after the CMD keyword.
J
list Kevin King
I do use that script. However it does not do anything with message files. (Unless the author has done some updates to it.) The other issue is I am dealing with a ROP feed. This is a serial stream of message data from port we have tied to a term server. So I have no OS to login to and run commands. Thanks for the tip! -Kevin
▸
From: Galen Johnson [mailto:user-87f955643e3d@xymon.invalid]
Sent: Tuesday, August 27, 2013 11:02 AM
To: Kevin King; 'Jeremy Laidman'; 'KING, KEVIN'; xymon at xymon.com
Subject: RE: [Xymon] Serial alarm feeds
Look on the front page of xymonton.org. There is a link to a "clientless"
xymon client (xymon-rclient) that might do what you are looking for.
=G=
From: Xymon [xymon-bounces at xymon.com] on behalf of Kevin King
[user-4c6c31ddcab2@xymon.invalid]
Sent: Tuesday, August 27, 2013 10:38 AM
To: 'Jeremy Laidman'; 'KING, KEVIN'; xymon at xymon.com
Subject: Re: [Xymon] Serial alarm feeds
Jeremy,
This is good food for thought. I think I might need to run this as an
external script. That way I can pass the host information, (I have more than
one of these) Then I should be able to enter a hostname in the
logfetch-special.cfg file. I then could do this for almost any collector
that cannot use a local client.
I seem to remember we had a tool like this way back in the BB days, I ether
found something or I modified bb. Wish I still had that server! I built many
a tool for BB. That is when I was on top of my game, I am a retread in this
admin stuff. I look at some of the stuff I wrote in the early 2000s and I
say wow nice code, what does it do?
Thanks for the kick start!
-Kevin
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Jeremy Laidman
Sent: Tuesday, August 27, 2013 12:14 AM
To: KING, KEVIN; xymon at xymon.com
Subject: Re: [Xymon] Serial alarm feeds
On 27 August 2013 13:31, KING, KEVIN <user-ca972c0c43a8@xymon.invalid> wrote:
No no snmp. It is a very closed system. The messages we get are out of a
serial port to a term server.
:-(
▸
I guess I do not know the client tool then. How can I save the files on my
xymon server and use entries in the client config and alert rules to make
if show up under the correct server? Maybe I am getting myself stuck in the
mud, but if the file is on the xymon host how can I process it like it was
on the client?
A server is almost always also a client. You should have the Xymon server
name in hosts.cfg, yes? Also, the Xymon server can be listed in
client-local.cfg and analysis.cfg. Just create appropriate entries in those
two files for the server and the logfile, and it will report the errors as
they appear.
The down-side to this is that the errors are assigned to the Xymon server.
If you want them assigned to the device name instead, then you will probably
have to do something a bit more complex, such as writing a custom script on
the Xymon server to do the log analysis and reporting. I don't see a way
that you can do this with the built-in log monitoring features.
Hmm, here's an idea. You could create a script that simply replicates the
log file into a client message, which would be a substitute for the
client-local.cfg entry. So then you'd be able to use the standard "LOG"
definitions in analysis.cfg to match whatever you want.
The easiest way to do this, I think, would be to run a second copy of
logfetch on the Xymon server (the standard one is usually run from
xymonclient.sh, from the [xymonclient] section of tasks.cfg). The logfetch
tool requires a configuration file and a state file as parameters. Normally
the config file is simply extracted from client-local.cfg and pushed to the
client during the client message interaction, but you can simply create your
own. For example (untested), create in $XYMONTMP the file
logfetch-special.cfg, containing the following:
log:/var/log/mylogfile:10240
ignore THIS
ignore THAT
Then add the following into tasks.cfg or a new file in tasks.d:
[xymonclient-special]
NEEDS xymond
ENVFILE $XYMONCLIENTHOME/etc/xymonclient.cfg
LOGFILE $XYMONCLIENTLOGS/xymonclient-special.log
INTERVAL 5m
CMD sh -c '( printf "eagle.example.com.eagle
eagle\n[msgs:/var/log/mylogfile]\n"; $XYMONCLIENTHOME/bin/logfetch
$XYMONTMP/logfetch-special.cfg $XYMONTMP/logfetch-special.status ) | $XYMON
$XYMSRV "@"'
This creates a header, using the device's hostname, and a [msgs:] section
for the logfile being monitored. Then it uses logfetch to get the latest
bunch of lines from the logfile, and filter out any "ignore" entries from
the config file logfetch-special.cfg. Finally, all of this is piped into an
instance of xymon that creates a client message and sends it to the server.
The end result is a client message that should start to generate a "msgs"
dot against the device name (assuming you added the device name into
hosts.cfg - probably with a "noping" tag). If this works, you can then
start using analysis.cfg to have the Xymon server parse the log file from
the client data.
This is all a bit kludgy. You might prefer to extract the CMD bit into a
separate script, and just put the script name after the CMD keyword.
J
list Galen Johnson
Since it sounds like you already have it pumping to a term server...is this using something like conserver, http://conserver.com/, or some other solution that would provide you more flexibility with running a client locally and mining the logs that way? While I'm sure the recommendations below would work, you would also gain a central log collector (assuming you don;t already have one). =G=
▸
From: Kevin King [user-4c6c31ddcab2@xymon.invalid]
Sent: Tuesday, August 27, 2013 11:27 AM
To: Galen Johnson; xymon at xymon.com
Subject: RE: [Xymon] Serial alarm feeds
I do use that script. However it does not do anything with message files. (Unless the author has done some updates to it.) The other issue is I am dealing with a ROP feed. This is a serial stream of message data from port we have tied to a term server. So I have no OS to login to and run commands.
Thanks for the tip!
-Kevin
From: Galen Johnson [mailto:user-87f955643e3d@xymon.invalid]
Sent: Tuesday, August 27, 2013 11:02 AM
To: Kevin King; 'Jeremy Laidman'; 'KING, KEVIN'; xymon at xymon.com
Subject: RE: [Xymon] Serial alarm feeds
Look on the front page of xymonton.org. There is a link to a "clientless" xymon client (xymon-rclient) that might do what you are looking for.
=G=
From: Xymon [xymon-bounces at xymon.com] on behalf of Kevin King [user-4c6c31ddcab2@xymon.invalid]
Sent: Tuesday, August 27, 2013 10:38 AM
To: 'Jeremy Laidman'; 'KING, KEVIN'; xymon at xymon.com<mailto:xymon at xymon.com>
Subject: Re: [Xymon] Serial alarm feeds
Jeremy,
This is good food for thought. I think I might need to run this as an external script. That way I can pass the host information, (I have more than one of these) Then I should be able to enter a hostname in the logfetch-special.cfg file. I then could do this for almost any collector that cannot use a local client.
I seem to remember we had a tool like this way back in the BB days, I ether found something or I modified bb. Wish I still had that server! I built many a tool for BB. That is when I was on top of my game, I am a retread in this admin stuff. I look at some of the stuff I wrote in the early 2000s and I say wow nice code, what does it do?
Thanks for the kick start!
-Kevin
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Jeremy Laidman
Sent: Tuesday, August 27, 2013 12:14 AM
To: KING, KEVIN; xymon at xymon.com<mailto:xymon at xymon.com>
Subject: Re: [Xymon] Serial alarm feeds
On 27 August 2013 13:31, KING, KEVIN <user-ca972c0c43a8@xymon.invalid<mailto:user-ca972c0c43a8@xymon.invalid>> wrote:
No no snmp. It is a very closed system. The messages we get are out of a serial port to a term server.
:-(
▸
I guess I do not know the client tool then. How can I save the files on my xymon server and use entries in the client config and alert rules to make if show up under the correct server? Maybe I am getting myself stuck in the mud, but if the file is on the xymon host how can I process it like it was on the client?
A server is almost always also a client. You should have the Xymon server name in hosts.cfg, yes? Also, the Xymon server can be listed in client-local.cfg and analysis.cfg. Just create appropriate entries in those two files for the server and the logfile, and it will report the errors as they appear.
The down-side to this is that the errors are assigned to the Xymon server. If you want them assigned to the device name instead, then you will probably have to do something a bit more complex, such as writing a custom script on the Xymon server to do the log analysis and reporting. I don't see a way that you can do this with the built-in log monitoring features.
Hmm, here's an idea. You could create a script that simply replicates the log file into a client message, which would be a substitute for the client-local.cfg entry. So then you'd be able to use the standard "LOG" definitions in analysis.cfg to match whatever you want.
The easiest way to do this, I think, would be to run a second copy of logfetch on the Xymon server (the standard one is usually run from xymonclient.sh, from the [xymonclient] section of tasks.cfg). The logfetch tool requires a configuration file and a state file as parameters. Normally the config file is simply extracted from client-local.cfg and pushed to the client during the client message interaction, but you can simply create your own. For example (untested), create in $XYMONTMP the file logfetch-special.cfg, containing the following:
log:/var/log/mylogfile:10240
ignore THIS
ignore THAT
Then add the following into tasks.cfg or a new file in tasks.d:
[xymonclient-special]
NEEDS xymond
ENVFILE $XYMONCLIENTHOME/etc/xymonclient.cfg
LOGFILE $XYMONCLIENTLOGS/xymonclient-special.log
INTERVAL 5m
CMD sh -c '( printf "eagle.example.com.eagle eagle\n[msgs:/var/log/mylogfile]\n"; $XYMONCLIENTHOME/bin/logfetch $XYMONTMP/logfetch-special.cfg $XYMONTMP/logfetch-special.status ) | $XYMON $XYMSRV "@"'
This creates a header, using the device's hostname, and a [msgs:] section for the logfile being monitored. Then it uses logfetch to get the latest bunch of lines from the logfile, and filter out any "ignore" entries from the config file logfetch-special.cfg. Finally, all of this is piped into an instance of xymon that creates a client message and sends it to the server.
The end result is a client message that should start to generate a "msgs" dot against the device name (assuming you added the device name into hosts.cfg - probably with a "noping" tag). If this works, you can then start using analysis.cfg to have the Xymon server parse the log file from the client data.
This is all a bit kludgy. You might prefer to extract the CMD bit into a separate script, and just put the script name after the CMD keyword.
J
list Kevin King
I have this running. I am getting the logfetch-special.status it is tracking the lines it has processed. I am not getting a msg file or a column and nothing in xymonclientl-speacial.log. Still searching.
▸
-Kevin
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Jeremy Laidman
Sent: Tuesday, August 27, 2013 12:14 AM
To: KING, KEVIN; xymon at xymon.com
Subject: Re: [Xymon] Serial alarm feeds
On 27 August 2013 13:31, KING, KEVIN <user-ca972c0c43a8@xymon.invalid> wrote:
No no snmp. It is a very closed system. The messages we get are out of a
serial port to a term server.
:-(
▸
I guess I do not know the client tool then. How can I save the files on my
xymon server and use entries in the client config and alert rules to make
if show up under the correct server? Maybe I am getting myself stuck in the
mud, but if the file is on the xymon host how can I process it like it was
on the client?
A server is almost always also a client. You should have the Xymon server
name in hosts.cfg, yes? Also, the Xymon server can be listed in
client-local.cfg and analysis.cfg. Just create appropriate entries in those
two files for the server and the logfile, and it will report the errors as
they appear.
The down-side to this is that the errors are assigned to the Xymon server.
If you want them assigned to the device name instead, then you will probably
have to do something a bit more complex, such as writing a custom script on
the Xymon server to do the log analysis and reporting. I don't see a way
that you can do this with the built-in log monitoring features.
Hmm, here's an idea. You could create a script that simply replicates the
log file into a client message, which would be a substitute for the
client-local.cfg entry. So then you'd be able to use the standard "LOG"
definitions in analysis.cfg to match whatever you want.
The easiest way to do this, I think, would be to run a second copy of
logfetch on the Xymon server (the standard one is usually run from
xymonclient.sh, from the [xymonclient] section of tasks.cfg). The logfetch
tool requires a configuration file and a state file as parameters. Normally
the config file is simply extracted from client-local.cfg and pushed to the
client during the client message interaction, but you can simply create your
own. For example (untested), create in $XYMONTMP the file
logfetch-special.cfg, containing the following:
log:/var/log/mylogfile:10240
ignore THIS
ignore THAT
Then add the following into tasks.cfg or a new file in tasks.d:
[xymonclient-special]
NEEDS xymond
ENVFILE $XYMONCLIENTHOME/etc/xymonclient.cfg
LOGFILE $XYMONCLIENTLOGS/xymonclient-special.log
INTERVAL 5m
CMD sh -c '( printf "eagle.example.com.eagle
eagle\n[msgs:/var/log/mylogfile]\n"; $XYMONCLIENTHOME/bin/logfetch
$XYMONTMP/logfetch-special.cfg $XYMONTMP/logfetch-special.status ) | $XYMON
$XYMSRV "@"'
This creates a header, using the device's hostname, and a [msgs:] section
for the logfile being monitored. Then it uses logfetch to get the latest
bunch of lines from the logfile, and filter out any "ignore" entries from
the config file logfetch-special.cfg. Finally, all of this is piped into an
instance of xymon that creates a client message and sends it to the server.
The end result is a client message that should start to generate a "msgs"
dot against the device name (assuming you added the device name into
hosts.cfg - probably with a "noping" tag). If this works, you can then
start using analysis.cfg to have the Xymon server parse the log file from
the client data.
This is all a bit kludgy. You might prefer to extract the CMD bit into a
separate script, and just put the script name after the CMD keyword.
J
list Jeremy Laidman
▸
On 28 August 2013 01:27, Kevin King <user-4c6c31ddcab2@xymon.invalid> wrote:
I do use that script. However it does not do anything with message files. (Unless the author has done some updates to it.)
The author is me. And there have been no updates to support messages, and probably never will. It's too much trouble to re-implement the logfetch utility. J
list Kevin King
I did get log fetch to work with your script, sorta. I added the file in the script to collect and it collects it and posts it to the message column but of course it is not processed for alerts, Just collected. I added this to the xymonclient-linux.sh that I use for the hosts I use your script for. Hmmm If I get this other log fetch working I might be able to smash your script with this and make that work. Ok better stick to one thing at a time. echo "[msgs:/var/log/messages]" tail -c 10240 /var/log/messages | egrep -v MARK echo "[msgs:/var/TKLC/log/syscheck/fail_log]" tail -c 10240 /var/TKLC/log/syscheck/fail_log
▸
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Jeremy Laidman
Sent: Tuesday, August 27, 2013 4:34 PM
To: Kevin King
Cc: xymon at xymon.com
Subject: Re: [Xymon] Serial alarm feeds
On 28 August 2013 01:27, Kevin King <user-4c6c31ddcab2@xymon.invalid<mailto:user-4c6c31ddcab2@xymon.invalid>> wrote:
I do use that script. However it does not do anything with message files. (Unless the author has done some updates to it.)
The author is me. And there have been no updates to support messages, and probably never will. It's too much trouble to re-implement the logfetch utility.
J
list Jeremy Laidman
▸
On 28 August 2013 06:20, Kevin King <user-4c6c31ddcab2@xymon.invalid> wrote:
I have this running. I am getting the logfetch-special.status it is tracking the lines it has processed.
Great.
▸
I am not getting a msg file or a column and nothing in xymonclientl-speacial.log.
*sad face*
Check your ghost report. Also see if you can view a client data page for
the device. Do this by viewing the conn page for another device, click on
the link for "client data" for that device, and then change the device name
in the URL to eagle.example.com (or whatever). You should get a client
message page.
If there's no client data page, then the xymond_client handler is rejecting
the message for some reason. Have a look at the output for xymond_client
and see if it's complaining about something. I wonder if there's a
critical section from the client message that's critical. For example,
maybe xymond_client rejects the client data message if it has no [date]
and/or [uname] sections.
By the way, as you're going to implement this as a separate script, you
might consider capturing the output from the command that sends the client
message, and that can be used for your logfetch-special.cfg file. The
"client" protocol not only accepts the client data but it gives back the
section from client-local.cfg that matches the hostname in the client
message header. But I'd add this bit later.
J
list Kevin King
The link hack did not work. If it would then I would expect to see the msg.host.txt in the tmp directory. I am thinking we might need more in the client message. Still trying. -Kevin
▸
From: Jeremy Laidman [mailto:user-71895fb2e44c@xymon.invalid]
Sent: Tuesday, August 27, 2013 4:54 PM
To: Kevin King
Cc: KING, KEVIN; xymon at xymon.com
Subject: Re: [Xymon] Serial alarm feeds
On 28 August 2013 06:20, Kevin King <user-4c6c31ddcab2@xymon.invalid> wrote:
I have this running. I am getting the logfetch-special.status it is
tracking the lines it has processed.
Great.
I am not getting a msg file or a column and nothing in
xymonclientl-speacial.log.
*sad face*
Check your ghost report. Also see if you can view a client data page for
the device. Do this by viewing the conn page for another device, click on
the link for "client data" for that device, and then change the device name
in the URL to eagle.example.com (or whatever). You should get a client
message page.
If there's no client data page, then the xymond_client handler is rejecting
the message for some reason. Have a look at the output for xymond_client
and see if it's complaining about something. I wonder if there's a critical
section from the client message that's critical. For example, maybe
xymond_client rejects the client data message if it has no [date] and/or
[uname] sections.
By the way, as you're going to implement this as a separate script, you
might consider capturing the output from the command that sends the client
message, and that can be used for your logfetch-special.cfg file. The
"client" protocol not only accepts the client data but it gives back the
section from client-local.cfg that matches the hostname in the client
message header. But I'd add this bit later.
J
list Jeremy Laidman
▸
On 28 August 2013 06:51, KING, KEVIN <user-ca972c0c43a8@xymon.invalid> wrote:
I did get log fetch to work with your script, sorta. I added the file in the script to collect and it collects it and posts it to the message column
Good.
but of course it is not processed for alerts, Just collected.
So it just needs an entry in analysis.cfg?
▸
****I added this to the xymonclient-linux.sh that I use for the hosts I use your script for. Hmmm If I get this other log fetch working I might be able to smash your script with this and make that work. Ok better stick to one thing at a time.
Always.
▸
echo "[msgs:/var/log/messages]" tail -c 10240 /var/log/messages | egrep -v MARK**** echo "[msgs:/var/TKLC/log/syscheck/fail_log]"**** tail -c 10240 /var/TKLC/log/syscheck/fail_log
Well this works, in as much as you'll get message lines in your client data, but you'll end up with repeated and skipped messages. So if you alert on the text "WARNING" and it appears in the log, sure you'll get an alert. But if the log gets less than 10k of data appended throughout the day, Xymon will repeat the alert all day. Or if, between one sample and the next, the "WARNING" message and then 10k of log data is added, the second sample won't see the matching message because it always gets the last 10k. These issues are dealt with by the logfetch utility. It keeps state so that it (generally) gives out each log message line exactly once - none is missed and none is repeated. You can handle this yourself, by keeping state of the size of the log file you're watching, and always tailing what's after that size. You also have to detect and handle log rotation. Doable, but lots of trouble. If you can get logfetch to work, then why not use it. Especially as it can give you your (its) config file from client-local.cfg to boot. J
list Kevin King
Ok made some progress today. I am getting a formatted client message, just not getting it to post. It may have something to do with it being local. Anyway I do not have anything showing up in ghost clients and have not found anything in any logs barking about this. But this is looking very promising. I am sure tomorrow will bring more advances. Your perspective has been very helpful Jeremy.
Here is the script I cobbled together from xymonclient.sh
Bash-3.2 $ cat xymonclient-special.sh
#!/bin/sh
# attempt to use log fetch to collect from log files
# on main xymon and report dots to correct line.
# -kking 8/28/2013
# Must make sure the commands return standard (english) texts.
LANG=C
LC_ALL=C
LC_MESSAGES=C
export LANG LC_ALL LC_MESSAGES
MACHINEDOTS="lkmrflmf00w"
MSGFILE="$XYMONTMP/msg.$MACHINEDOTS.txt"
MSGTMPFILE="$MSGFILE.$$"
LOGFETCHCFG=$XYMONTMP/logfetch.$MACHINEDOTS.cfg
LOGFETCHSTATUS=$XYMONTMP/logfetch.$MACHINEDOTS.status
export LOGFETCHCFG LOGFETCHSTATUS
rm -f $MSGTMPFILE
touch $MSGTMPFILE
echo "client $MACHINEDOTS.eagle eagle" >> $MSGTMPFILE
$XYMONHOME/bin/xymonclient-eagle.sh >> $MSGTMPFILE
# logfiles
if test -f $LOGFETCHCFG
then
$XYMONHOME/bin/logfetch $LOGFETCHCFG $LOGFETCHSTATUS >>$MSGTMPFILE
fi
# System clock
#echo "[clock]" >> $MSGTMPFILE
#$XYMONHOME/bin/logfetch --clock >> $MSGTMPFILE
$XYMON $XYMSRV "@" < $MSGTMPFILE >$LOGFETCHCFG.tmp
if test -f $LOGFETCHCFG.tmp
then
if test -s $LOGFETCHCFG.tmp
then
mv $LOGFETCHCFG.tmp $LOGFETCHCFG
else
rm -f $LOGFETCHCFG.tmp
fi
fi
# Save the latest file for debugging.
rm -f $MSGFILE
mv $MSGTMPFILE $MSGFILE
exit 0
and this is the file it creates:
client lkmrflmf00w.eagle eagle
[date]
Wed Aug 28 19:16:46 CDT 2013
[uname]
EAGLE 44.0.4-64.34.24
[uptime]
[who]
[msgs:/usr/local/xymon/server/www/html/lkmrflmf00w/lkmrflmf00w]
<...SKIPPED...>
15:55
;
lkmrflmf00w 13-08-28 18:16:02 CST EAGLE 44.0.4-64.34.24
9261.1056 CARD 2203 INFO SCCP rcvd inv SCMG - bad subsystem
SIO=a3 OPC= 209-049-005 DPC= 001-151-001
SCMG: MSG TYPE=003 MSG LEN=006
AFTPC= 001-151-001 AFTSS=146 MULT=000
LSN=m209049005
Report Date:13-08-28 Time:18:16:02
;
lkmrflmf00w 13-08-28 18:16:05 CST EAGLE 44.0.4-64.34.24
9262.1056 CARD 5107 INFO SCCP rcvd inv SCMG - bad subsystem
SIO=a3 OPC= 209-250-002 DPC= 001-151-001
SCMG: MSG TYPE=003 MSG LEN=006
AFTPC= 001-151-001 AFTSS=008 MULT=000
LSN=m209250002
Report Date:13-08-28 Time:18:16:05
<SNIP>
[logfile:/usr/local/xymon/server/www/html/lkmrflmf00w/lkmrflmf00w]
type:100000 (file)
mode:640 (-rw-r-----)
linkcount:1
owner:100 (xymon)
group:100 (bbgroup)
size:17902673
clock:1377735406 (2013/08/28-19:16:46)
atime:1377735406 (2013/08/28-19:16:46)
ctime:1377735403 (2013/08/28-19:16:43)
mtime:1377735403 (2013/08/28-19:16:43)
▸
From: Kevin King [mailto:user-4c6c31ddcab2@xymon.invalid]
Sent: Tuesday, August 27, 2013 5:07 PM
To: 'Jeremy Laidman'
Cc: KING, KEVIN; xymon at xymon.com
Subject: RE: [Xymon] Serial alarm feeds
The link hack did not work. If it would then I would expect to see the msg.host.txt in the tmp directory. I am thinking we might need more in the client message.
Still trying.
-Kevin
From: Jeremy Laidman [mailto:user-71895fb2e44c@xymon.invalid]
Sent: Tuesday, August 27, 2013 4:54 PM
To: Kevin King
Cc: KING, KEVIN; xymon at xymon.com<mailto:xymon at xymon.com>
Subject: Re: [Xymon] Serial alarm feeds
On 28 August 2013 06:20, Kevin King <user-4c6c31ddcab2@xymon.invalid<mailto:user-4c6c31ddcab2@xymon.invalid>> wrote:
I have this running. I am getting the logfetch-special.status it is tracking the lines it has processed.
Great.
I am not getting a msg file or a column and nothing in xymonclientl-speacial.log.
*sad face*
Check your ghost report. Also see if you can view a client data page for the device. Do this by viewing the conn page for another device, click on the link for "client data" for that device, and then change the device name in the URL to eagle.example.com<http://eagle.example.com>; (or whatever). You should get a client message page.
▸
If there's no client data page, then the xymond_client handler is rejecting the message for some reason. Have a look at the output for xymond_client and see if it's complaining about something. I wonder if there's a critical section from the client message that's critical. For example, maybe xymond_client rejects the client data message if it has no [date] and/or [uname] sections.
By the way, as you're going to implement this as a separate script, you might consider capturing the output from the command that sends the client message, and that can be used for your logfetch-special.cfg file. The "client" protocol not only accepts the client data but it gives back the section from client-local.cfg that matches the hostname in the client message header. But I'd add this bit later.
J
list Jeremy Laidman
▸
On 29 August 2013 10:24, KING, KEVIN <user-ca972c0c43a8@xymon.invalid> wrote:
Ok made some progress today. I am getting a formatted client message, just not getting it to post. It may have something to do with it being local.
Nope, that should not be a problem. Xymon doesn't normally care where a client message comes from as long as the hostname matches an entry in hosts.cfg.
▸
Anyway I do not have anything showing up in ghost clients and have not found anything in any logs barking about this.
Strange.
▸
But this is looking very promising. I am sure tomorrow will bring more advances. Your perspective has been very helpful Jeremy.
Happy to help. I'd be running the xymond_channel process to see if it catches the client messages you're sending, and perhaps compare with other client messages. Something like this: $ xymoncmd xymond_channel --filter="lkmrflmf00w" --channel=client cat This will capture client messages where the first line contains "lkmrflmf00w". You can replace "cat" with "grep" or "sed" or other commands that limit the output. Although if this is the only message for this host then probably not necessary in this case. J
list Kevin King
Ok I got it to post!.... now for the reason, or what I did to make it work. Jeremy, I was using your example of hostname.eagle eagle, that was the problem. I changed it to sunos and it posted fine. I did make an entry in local config for eagle and it would build the config file correctly so I thought I was good to go. Looks like some deeper work is needed to support custom clients. With this I might go back and try your first script suggestion to see if it will work with sunos replacing eagle.. Wait I have a very dim light bulb flashing above my head.. the host I am running this on is sun, so maybe somewhere it is checking a uname -a that I overlooked. So it might be a local issue, see I came back to that. J This was fun and I can see this working with your xymon remote client to process the log files. Now to dig a bit deeper to see what has to be done to create an eagle hostclass and ostype
▸
-Kevin
From: Jeremy Laidman [mailto:user-71895fb2e44c@xymon.invalid]
Sent: Thursday, August 29, 2013 12:37 AM
To: KING, KEVIN
Cc: Kevin King; xymon at xymon.com
Subject: Re: [Xymon] Serial alarm feeds
On 29 August 2013 10:24, KING, KEVIN <user-ca972c0c43a8@xymon.invalid> wrote:
Ok made some progress today. I am getting a formatted client message, just
not getting it to post. It may have something to do with it being local.
Nope, that should not be a problem. Xymon doesn't normally care where a
client message comes from as long as the hostname matches an entry in
hosts.cfg.
Anyway I do not have anything showing up in ghost clients and have not
found anything in any logs barking about this.
Strange.
But this is looking very promising. I am sure tomorrow will bring more
advances. Your perspective has been very helpful Jeremy.
Happy to help.
I'd be running the xymond_channel process to see if it catches the client
messages you're sending, and perhaps compare with other client messages.
Something like this:
$ xymoncmd xymond_channel --filter="lkmrflmf00w" --channel=client cat
This will capture client messages where the first line contains
"lkmrflmf00w". You can replace "cat" with "grep" or "sed" or other commands
that limit the output. Although if this is the only message for this host
then probably not necessary in this case.
J
list Jeremy Laidman
▸
On Aug 30, 2013 8:54 AM, "Kevin King" <user-4c6c31ddcab2@xymon.invalid> wrote:
Jeremy, I was using your example of hostname.eagle eagle, that was the problem. I changed it to sunos and it posted fine.
Maybe Xymon just wants to see a matching section in client-local.cfg. J
list Ralph Mitchell
I think xymon wants to see a matching section in xymond/xymond_client.c.
There's a big switch statement down at the bottom of that file, selecting
between the various client functions in xymond/client/*.c. Those files are
included into xymond_client.c right above main(). If it hits the end of
the switch without matching an OS, it says:
errprintf("No client backend for OS '%s' sent by %s\n", clientos,
sender);
That message is probably showing up in the log somewhere.
Ralph Mitchell
▸
On Thu, Aug 29, 2013 at 8:07 PM, Jeremy Laidman <user-71895fb2e44c@xymon.invalid>wrote:
On Aug 30, 2013 8:54 AM, "Kevin King" <user-4c6c31ddcab2@xymon.invalid> wrote:Jeremy, I was using your example of hostname.eagle eagle, that was the problem. I changed it to sunos and it posted fine.Maybe Xymon just wants to see a matching section in client-local.cfg. J
list Japheth Cleaver
Assuming you're using 4.3.12 here, the --unknownclientsok option to xymond_client should allow it through. If there's another mis-parse, hopefully it would show up in --debug mode :/ (or when given a -USR2 signal) HTH, -jc Sent from my iPad
▸
On Aug 29, 2013, at 5:33 PM, Ralph Mitchell <user-00a5e44c48c0@xymon.invalid> wrote:
I think xymon wants to see a matching section in xymond/xymond_client.c. There's a big switch statement down at the bottom of that file, selecting between the various client functions in xymond/client/*.c. Those files are included into xymond_client.c right above main(). If it hits the end of the switch without matching an OS, it says: errprintf("No client backend for OS '%s' sent by %s\n", clientos, sender); That message is probably showing up in the log somewhere. Ralph Mitchell On Thu, Aug 29, 2013 at 8:07 PM, Jeremy Laidman <user-71895fb2e44c@xymon.invalid> wrote:On Aug 30, 2013 8:54 AM, "Kevin King" <user-4c6c31ddcab2@xymon.invalid> wrote:Jeremy, I was using your example of hostname.eagle eagle, that was the problem. I changed it to sunos and it posted fine.Maybe Xymon just wants to see a matching section in client-local.cfg. J
list Kevin King
I had eagle in client-local.cfg. I think the other posts about the c code is where the issue lies.
▸
-Kevin
From: Jeremy Laidman [mailto:user-71895fb2e44c@xymon.invalid]
Sent: Thursday, August 29, 2013 8:08 PM
To: Kevin King
Cc: KING, KEVIN; xymon at xymon.com
Subject: RE: [Xymon] Serial alarm feeds
On Aug 30, 2013 8:54 AM, "Kevin King" <user-4c6c31ddcab2@xymon.invalid> wrote:
Jeremy, I was using your example of hostname.eagle eagle, that was the problem. I changed it to sunos and it posted fine.
Maybe Xymon just wants to see a matching section in client-local.cfg. J
list Kevin King
Ralph, spot on! I missed it, it was right there in the clientdata.log -Kevin
▸
From: Ralph Mitchell [mailto:user-00a5e44c48c0@xymon.invalid]
Sent: Thursday, August 29, 2013 8:34 PM
To: Jeremy Laidman
Cc: Kevin King; xymon at xymon.com
Subject: Re: [Xymon] Serial alarm feeds
I think xymon wants to see a matching section in xymond/xymond_client.c.
There's a big switch statement down at the bottom of that file, selecting
between the various client functions in xymond/client/*.c. Those files are
included into xymond_client.c right above main(). If it hits the end of the
switch without matching an OS, it says:
errprintf("No client backend for OS '%s' sent by %s\n", clientos,
sender);
That message is probably showing up in the log somewhere.
Ralph Mitchell
On Thu, Aug 29, 2013 at 8:07 PM, Jeremy Laidman <user-71895fb2e44c@xymon.invalid>
wrote:
On Aug 30, 2013 8:54 AM, "Kevin King" <user-4c6c31ddcab2@xymon.invalid> wrote:
Jeremy, I was using your example of hostname.eagle eagle, that was the problem. I changed it to sunos and it posted fine.
Maybe Xymon just wants to see a matching section in client-local.cfg. J