Breaking msgs in multiple columns for various log files monitoring 🔗 link

From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Pierre Malenfant
Sent: 20 January 2014 13:54
To: xymon at xymon.com
Subject: [Xymon] Breaking msgs in multiple columns for various log files monitoring

Hello All,

I searched the Xymon Archive without success about this.  Found a few inquiries but no solutions.

I would like to have multiple columns for log file monitoring.  I need the exact same thing as msgs, but in many columns to alert differently.

Ex:
System logs -> Alert to sysadmins
Application logs -> Alert to Analysts
DB logs -> Alert to DBA.

Having the columns named like msgs_sys / msgs_app / msgs_db  would make sense to me.

There may be good ext scripts for this.  I tought it would be simpler to have a central setup with the msgs "style".

For thoses of you that had the issue, what's your recommendation?

Thanks in advance!

Pierre Malenfant

Neil Simmonds wrote:

Possibly a better way would be to stick with the single msgs column but in your analysis.cfg where you specifiy the log strings you want to alert on, you can add GROUP=

 
i.e.  LOG /applogs/was/server/SystemOut.log "java.lang.OutOfMemoryError" COLOR=red GROUP=waslogs

 
That way when you set up the alerts you can specify the host, the service and the group

 
HOST=server SERVICE=msgs GROUP=waslogs

            MAIL etc…..

 
*From:* Xymon [mailto:xymon-bounces at xymon.com] *On Behalf Of *Pierre Malenfant
*Sent:* 20 January 2014 13:54
*To:* xymon at xymon.com
*Subject:* [Xymon] Breaking msgs in multiple columns for various log files monitoring

 
Hello All,

I searched the Xymon Archive without success about this.  Found a few inquiries but no solutions.

I would like to have multiple columns for log file monitoring.  I need the exact same thing as msgs, but in many columns to alert differently.

Ex:
System logs -> Alert to sysadmins
Application logs -> Alert to Analysts
DB logs -> Alert to DBA.

Having the columns named like msgs_sys / msgs_app / msgs_db  would make sense to me.

There may be good ext scripts for this.  I tought it would be simpler to have a central setup with the msgs "style".

For thoses of you that had the issue, what's your recommendation?

Thanks in advance!

Pierre Malenfant

Name & Registered Office: EXPRESS GIFTS LIMITED, 2 GREGORY ST, HYDE, CHESHIRE, ENGLAND, SK14 4TH, Company No. 00718151.

On 25 January 2014 17:56, Andy Smith <user-982f5f6d4d28@xymon.invalid> wrote:

Neil Simmonds wrote:

Possibly a better way would be to stick with the single msgs column but
in your analysis.cfg where you specifiy the log strings you want to alert
on, you can add GROUP=


i.e.  LOG /applogs/was/server/SystemOut.log "java.lang.OutOfMemoryError"
COLOR=red GROUP=waslogs


That way when you set up the alerts you can specify the host, the service
and the group


HOST=server SERVICE=msgs GROUP=waslogs

            MAIL etc…..


*From:* Xymon [mailto:xymon-bounces at xymon.com] *On Behalf Of *Pierre
Malenfant
*Sent:* 20 January 2014 13:54
*To:* xymon at xymon.com
*Subject:* [Xymon] Breaking msgs in multiple columns for various log
files monitoring


Hello All,

I searched the Xymon Archive without success about this.  Found a few
inquiries but no solutions.

I would like to have multiple columns for log file monitoring.  I need
the exact same thing as msgs, but in many columns to alert differently.

Ex:
System logs -> Alert to sysadmins
Application logs -> Alert to Analysts
DB logs -> Alert to DBA.

Having the columns named like msgs_sys / msgs_app / msgs_db  would make
sense to me.

There may be good ext scripts for this.  I tought it would be simpler to
have a central setup with the msgs "style".

For thoses of you that had the issue, what's your recommendation?

Thanks in advance!

Pierre Malenfant

Name & Registered Office: EXPRESS GIFTS LIMITED, 2 GREGORY ST, HYDE,
CHESHIRE, ENGLAND, SK14 4TH, Company No. 00718151.

Express Gifts Limited is authorised and regulated by the Financial
Conduct Authority


NOTE: This email and any information contained within or attached in a
separate file is confidential and intended solely for the Individual to
whom it is addressed. The information or data included is solely for the
purpose indicated or previously agreed. Any information or data included
with this e-mail remains the property of Findel PLC and the recipient will
refrain from utilising the information for any purpose other than that
indicated and upon request will destroy the information and remove it from
their records. Any views or opinions presented are solely those of the
author and do not necessarily represent those of Findel PLC. If you are not
the intended recipient, be advised that you have received this email in
error and that any use, dissemination, forwarding, printing, or copying of
this email is strictly prohibited. No warranties or assurances are made in
relation to the safety and content of this e-mail and any attachments. No
liability is accepted for any consequences arising from it. Findel Plc
reserves the right to monitor all e-mail communications through its
internal and external networks. If you have received this email in error
please notify our IT helpdesk on +44(0) 1254 303030

Using GROUPS is fine if your notification mechanism was solely e-mail
based.  We have a 24x7 team following the critical view and we need the
ability to separate faults found in different log files in exactly the same
way as Pierre describes.  For the moment we get round it as follows :-
1) define the logs as normal in client-local.cfg on the sever.
2) create a custom ext script on the server which :-
   a) extracts the logs from the client data :-
      xymon localhost "clientlog <myhost> section=msgs:/path/to/the/log"
   b) decides if this is good or bad.
   c) sends a status update to xymon for the new column :-
      xymon localhost "status <myhost>.msgs_apps RED|YELLOW|GREEN"

2b) is messy and hides stuff that should otherwise be in analysis.cfg, so
as a compromise, we use TRIGGER in client-local.cfg, which means 2b) can be
almost as simple as a non-zero line count.  I would much prefer support for
multiple msgs columns directly in core xymon but have not yet had chance to
see how difficult this would be to implement.

HTH
--
Andy

-- 

Vernon Everett wrote:

Hi all

My DBAs at a previous client wanted their own alert.log column for their database alerts logs.
It was really simple. I no longer have access to it, but in pseudo-code, it looked something like this.

LOGFILES=$(grep for files to check in clientlocal)
#This makes it configurable at server side
for LOGFILE in $LOGFILES
do
  colour=green
  # If this must go to a specific alert column in the display
  column_name=$(grep $LOGFILE.colname clientlocal)
  if [ -z $column_name ] && column_name=alt_logs

  # the next bit makes sure we only check the last bit of the file.
  if [ exist lengthfile.$LOGFILE ]
  then
     lastline=$(cat lengthfile.$LOGFILE)
  else
     lastline=0
  fi
   # get the current length of the log file
  file_len=$(wc $LOGFILE)

  if [ $lastline -gt $file_len ] && lastline=0 # Somebody cycled the log file
   echo $file_len > lengthfile.$LOGFILE # save the length for next time we run

  grepstringred=$(grep string_that_trigger_red_alerts.$LOGFILE clientlocal)
  grepstringyel=$(grep string_that_trigger_yellow_alerts.$LOGFILE clientlocal)
  # The above lines makes it configurable from server side
   tail -($file_len-$lastline)>$tmpfile
   grep $grepstringred $tmpfile && colour=red
  if [ $colour == green ] && grep $grepstringyel $tmpfile &&  colour=yellow

send results and $tmpfile to server

done
 This only worked if we need one log file per column. You may have to add some logic and come up with some clever html display to put more than one file per column.
For our DBAs it was easy, because they wanted one column per database instance, which I gave them on their own page with some clever use of the group-except and group-only.

Hope it helps.

Cheers
Vernon

P.S. Maybe I will rewrite this properly, and pop it on xymonton in the near future. I haven't contributed anything there for some time.


On 25 January 2014 17:56, Andy Smith <user-982f5f6d4d28@xymon.invalid <mailto:user-982f5f6d4d28@xymon.invalid>> wrote:

    Neil Simmonds wrote:

        Possibly a better way would be to stick with the single msgs
        column but in your analysis.cfg where you specifiy the log
        strings you want to alert on, you can add GROUP=

                 i.e.  LOG /applogs/was/server/SystemOut.__log
        "java.lang.OutOfMemoryError" COLOR=red GROUP=waslogs

                 That way when you set up the alerts you can specify the host,
        the service and the group

                 HOST=server SERVICE=msgs GROUP=waslogs

                    MAIL etc…..

                 *From:* Xymon [mailto:xymon-bounces at xymon.__com
        <mailto:xymon-bounces at xymon.com>] *On Behalf Of *Pierre Malenfant
        *Sent:* 20 January 2014 13:54

        *Subject:* [Xymon] Breaking msgs in multiple columns for various
        log files monitoring


                 Hello All,

        I searched the Xymon Archive without success about this.  Found
        a few inquiries but no solutions.

        I would like to have multiple columns for log file monitoring.
         I need the exact same thing as msgs, but in many columns to
        alert differently.

        Ex:
        System logs -> Alert to sysadmins
        Application logs -> Alert to Analysts
        DB logs -> Alert to DBA.

        Having the columns named like msgs_sys / msgs_app / msgs_db
         would make sense to me.

        There may be good ext scripts for this.  I tought it would be
        simpler to have a central setup with the msgs "style".

        For thoses of you that had the issue, what's your recommendation?

        Thanks in advance!

        Pierre Malenfant

        Name & Registered Office: EXPRESS GIFTS LIMITED, 2 GREGORY ST,
        HYDE, CHESHIRE, ENGLAND, SK14 4TH, Company No. 00718151.

        Express Gifts Limited is authorised and regulated by the
        Financial Conduct Authority


        NOTE: This email and any information contained within or
        attached in a separate file is confidential and intended solely
        for the Individual to whom it is addressed. The information or
        data included is solely for the purpose indicated or previously
        agreed. Any information or data included with this e-mail
        remains the property of Findel PLC and the recipient will
        refrain from utilising the information for any purpose other
        than that indicated and upon request will destroy the
        information and remove it from their records. Any views or
        opinions presented are solely those of the author and do not
        necessarily represent those of Findel PLC. If you are not the
        intended recipient, be advised that you have received this email
        in error and that any use, dissemination, forwarding, printing,
        or copying of this email is strictly prohibited. No warranties
        or assurances are made in relation to the safety and content of
        this e-mail and any attachments. No liability is accepted for
        any consequences arising from it. Findel Plc reserves the right
        to monitor all e-mail communications through its internal and
        external networks. If you have received this email in error
        please notify our IT helpdesk on +44(0) 1254 303030

         
        <


    Using GROUPS is fine if your notification mechanism was solely
    e-mail based.  We have a 24x7 team following the critical view and
    we need the ability to separate faults found in different log files
    in exactly the same way as Pierre describes.  For the moment we get
    round it as follows :-
    1) define the logs as normal in client-local.cfg on the sever.
    2) create a custom ext script on the server which :-
       a) extracts the logs from the client data :-
          xymon localhost "clientlog <myhost> section=msgs:/path/to/the/log"
       b) decides if this is good or bad.
       c) sends a status update to xymon for the new column :-
          xymon localhost "status <myhost>.msgs_apps RED|YELLOW|GREEN"

    2b) is messy and hides stuff that should otherwise be in
    analysis.cfg, so as a compromise, we use TRIGGER in
    client-local.cfg, which means 2b) can be almost as simple as a
    non-zero line count.  I would much prefer support for multiple msgs
    columns directly in core xymon but have not yet had chance to see
    how difficult this would be to implement.

    HTH
    --     Andy
    <


-- 
"Accept the challenges so that you can feel the exhilaration of victory"
- General George Patton