Automated remedial actions

6 messages in this thread

list Neil Simmonds · Thu, 17 Nov 2011 08:05:13 -0000 ·

Hi all,

I've got a requirement to take a specific action on a unix server should
we get a specific message in a log file.

I'm not sure this is even possible in Xymon and I suspect I'll have to
try and work out some way of doing it with a script.

Perhaps an external script that will somehow check for the message, run
the relevant command to fix it and report a warning alert so that we
know it has happened.

If there is some better way of doing it through Xymon I'd really like to
know how.

Thanks,

Neil Simmonds Operations Analyst
Operations Support Group Express Gifts Ltd. user-8188d25e65e4@xymon.invalid
Tel :- 01254 303092 Fax :- 01254 303100

Name & Registered Office: EXPRESS GIFTS LIMITED, 2 GREGORY ST, HYDE, CHESHIRE, ENGLAND, SK14 4TH, Company No. 00718151.
Express Gifts Limited is authorised and regulated by the Financial Services Authority
NOTE: This email and any information contained within or attached in a separate file is confidential and intended solely for the Individual to whom it is addressed. The information or data included is solely for the purpose indicated or previously agreed. Any information or data included with this e-mail remains the property of Findel PLC and the recipient will refrain from utilising the information for any purpose other than that indicated and upon request will destroy the information and remove it from their records. Any views or opinions presented are solely those of the author and do not necessarily represent those of Findel PLC. If you are not the intended recipient, be advised that you have received this email in error and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. No warranties or assurances are made in relation to the safety and content of this e-mail and any attachments. No liability is accepted for any consequences arising from it. Findel Plc reserves the right to monitor all e-mail communications through its internal and external networks. If you have received this email in error please notify our IT helpdesk on +44(0) 1254 303030

list Vernon Everett · Thu, 17 Nov 2011 17:05:45 +0800 ·

Check the man page for Alerts.
Xymon can, as part of normal alerts, kick off a script, which, in theory,
could do almost anything you want.

Should a monitoring tool take remedial action or not?
I leave that for you to decide. :-)

Regards
     Vernon


On 17 November 2011 16:05, Neil Simmonds

▸ quoted from Neil Simmonds

<user-8188d25e65e4@xymon.invalid>wrote:

 Hi all,****

** **

I’ve got a requirement to take a specific action on a unix server should
we get a specific message in a log file. ****

** **

I’m not sure this is even possible in Xymon and I suspect I’ll have to try
and work out some way of doing it with a script. ****

** **

Perhaps an external script that will somehow check for the message, run
the relevant command to fix it and report a warning alert so that we know
it has happened.****

** **

If there is some better way of doing it through Xymon I’d really like to
know how.****

** **

Thanks,****

Neil Simmonds
Operations Analyst
Operations Support Group
Express Gifts Ltd.
user-8188d25e65e4@xymon.invalid ****

Tel :- 01254 303092
Fax :- 01254 303100 ****

** **

Name & Registered Office: EXPRESS GIFTS LIMITED, 2 GREGORY ST, HYDE,
CHESHIRE, ENGLAND, SK14 4TH, Company No. 00718151.
Express Gifts Limited is authorised and regulated by the Financial
Services Authority
NOTE: This email and any information contained within or attached in a
separate file is confidential and intended solely for the Individual to
whom it is addressed. The information or data included is solely for the
purpose indicated or previously agreed. Any information or data included
with this e-mail remains the property of Findel PLC and the recipient will
refrain from utilising the information for any purpose other than that
indicated and upon request will destroy the information and remove it from
their records. Any views or opinions presented are solely those of the
author and do not necessarily represent those of Findel PLC. If you are not
the intended recipient, be advised that you have received this email in
error and that any use, dissemination, forwarding, printing, or copying of
this email is strictly prohibited. No warranties or assurances are made in
relation to the safety and content of this e-mail and any attachments. No
liability is accepted for any consequences arising from it. Findel Plc
reserves the right to monitor all e-mail communications through its
internal and external networks. If you have received this email in error
please notify our IT helpdesk on +44(0) 1254 303030

--


"While it is futile to try to eliminate risk, and questionable to try to
minimize it, it is essential that the risks taken be the right risks. "
- Peter F. Drucker

list Neil Simmonds · Thu, 17 Nov 2011 09:15:29 -0000 ·

Hi Vernon,

 
I've user-eecb411e3384@xymon.invalid and as far as I can see that would only allow
me to narrow it down to the service (i.e. I could specify the msgs
service on a specific host)

 
However that doesn't allow me to narrow it down to a specific message. I
had thought of perhaps triggering a script that way and then doing a
grep through the last 500 lines of the log to see if I find the message
I'm interested in (I don't expect it to come out too often).

 
As for whether a monitoring solution should take remedial action,
unfortunately we are currently using one that shall remain nameless but
let's just say it's one of the big 4 and we are looking to replace all
the functionality we are currently utilising from that with Xymon. As
that does remedial actions I have to try and find a way of doing it in
Xymon.

 
Cheers,

▸ signature


Neil Simmonds 
Operations Analyst
Operations Support Group 
Express Gifts Ltd. 
user-8188d25e65e4@xymon.invalid 

Tel :- 01254 303092 
Fax :- 01254 303100

▸ quoted from Vernon Everett

From: Vernon Everett [mailto:user-b3f8dacb72c8@xymon.invalid] 
Sent: 17 November 2011 09:06
To: Neil Simmonds
Cc: Xymon Mailing List
Subject: Re: [Xymon] Automated remedial actions

Check the man page for Alerts.

Xymon can, as part of normal alerts, kick off a script, which, in
theory, could do almost anything you want.

Should a monitoring tool take remedial action or not?

I leave that for you to decide. :-)

Regards

     Vernon

On 17 November 2011 16:05, Neil Simmonds
<user-8188d25e65e4@xymon.invalid> wrote:

Hi all,

I've got a requirement to take a specific action on a unix server should
we get a specific message in a log file. 

I'm not sure this is even possible in Xymon and I suspect I'll have to
try and work out some way of doing it with a script. 

Perhaps an external script that will somehow check for the message, run
the relevant command to fix it and report a warning alert so that we
know it has happened.

If there is some better way of doing it through Xymon I'd really like to
know how.

Thanks,

Neil Simmonds 
Operations Analyst
Operations Support Group 
Express Gifts Ltd. 
user-8188d25e65e4@xymon.invalid 

Tel :- 01254 303092 
Fax :- 01254 303100 

Name & Registered Office: EXPRESS GIFTS LIMITED, 2 GREGORY ST, HYDE,
CHESHIRE, ENGLAND, SK14 4TH, Company No. 00718151.
Express Gifts Limited is authorised and regulated by the Financial
Services Authority
NOTE: This email and any information contained within or attached in a
separate file is confidential and intended solely for the Individual to
whom it is addressed. The information or data included is solely for the
purpose indicated or previously agreed. Any information or data included
with this e-mail remains the property of Findel PLC and the recipient
will refrain from utilising the information for any purpose other than
that indicated and upon request will destroy the information and remove
it from their records. Any views or opinions presented are solely those
of the author and do not necessarily represent those of Findel PLC. If
you are not the intended recipient, be advised that you have received
this email in error and that any use, dissemination, forwarding,
printing, or copying of this email is strictly prohibited. No warranties
or assurances are made in relation to the safety and content of this
e-mail and any attachments. No liability is accepted for any
consequences arising from it. Findel Plc reserves the right to monitor
all e-mail communications through its internal and external networks. If
you have received this email in error please notify our IT helpdesk on


+44(0) 1254 303030 <tel:%2B44%280%29%201254%20303030>

▸ quoted from Vernon Everett

--
"While it is futile to try to eliminate risk, and questionable to try to
minimize it, it is essential that the risks taken be the right risks. "
- Peter F. Drucker

list Vernon Everett · Thu, 17 Nov 2011 17:25:34 +0800 ·

No need for that level of complexity.
Scroll down a little further.
The following environment variables will be defined within your script.
*BBALPHAMSG* The full text of the status log triggering the alert
So all you need to do is
echo $BBALPHAMSG | grep "Bad message" && do_stuff

Hope it helps

Cheers
     Vernon

On 17 November 2011 17:15, Neil Simmonds

▸ quoted from Neil Simmonds

<user-8188d25e65e4@xymon.invalid>wrote:

** ** ** ** **

Hi ****Vernon****,****

** **

I’ve user-eecb411e3384@xymon.invalid and as far as I can see that would only allow me
to narrow it down to the service (i.e. I could specify the msgs service on
a specific host)****

** **

However that doesn’t allow me to narrow it down to a specific message. I
had thought of perhaps triggering a script that way and then doing a grep
through the last 500 lines of the log to see if I find the message I’m
interested in (I don’t expect it to come out too often).****

** **

As for whether a monitoring solution should take remedial action,
unfortunately we are currently using one that shall remain nameless but
let’s just say it’s one of the big 4 and we are looking to replace all the
functionality we are currently utilising from that with Xymon. As that does
remedial actions I have to try and find a way of doing it in Xymon.****

** **

Cheers,****

Neil Simmonds
Operations Analyst
Operations Support Group
Express Gifts Ltd.
user-8188d25e65e4@xymon.invalid ****

Tel :- 01254 303092
Fax :- 01254 303100 ****

*From:* Vernon Everett [mailto:user-b3f8dacb72c8@xymon.invalid]
*Sent:* 17 November 2011 09:06
*To:* Neil Simmonds
*Cc:* Xymon Mailing List
*Subject:* Re: [Xymon] Automated remedial actions****

** **

Check the man page for Alerts.****

Xymon can, as part of normal alerts, kick off a script, which, in theory,
could do almost anything you want.****

** **

Should a monitoring tool take remedial action or not?****

I leave that for you to decide. :-)****

** **

Regards****

     ****Vernon********

** **

** **

** **

On 17 November 2011 16:05, Neil Simmonds <
user-8188d25e65e4@xymon.invalid> wrote:****

Hi all,****

 ****

I’ve got a requirement to take a specific action on a unix server should
we get a specific message in a log file. ****

 ****

I’m not sure this is even possible in Xymon and I suspect I’ll have to try
and work out some way of doing it with a script. ****

 ****

Perhaps an external script that will somehow check for the message, run
the relevant command to fix it and report a warning alert so that we know
it has happened.****

 ****

If there is some better way of doing it through Xymon I’d really like to
know how.****

 ****

Thanks,****

Neil Simmonds
Operations Analyst
Operations Support Group
Express Gifts Ltd.
user-8188d25e65e4@xymon.invalid ****

Tel :- 01254 303092
Fax :- 01254 303100 ****

 ****


Name & Registered Office: EXPRESS GIFTS LIMITED, ****2 GREGORY ST****,
HYDE, ****CHESHIRE**, **ENGLAND****, SK14 4TH, Company No. 00718151.
Express Gifts Limited is authorised and regulated by the Financial
Services Authority
NOTE: This email and any information contained within or attached in a
separate file is confidential and intended solely for the Individual to
whom it is addressed. The information or data included is solely for the
purpose indicated or previously agreed. Any information or data included
with this e-mail remains the property of Findel PLC and the recipient will
refrain from utilising the information for any purpose other than that
indicated and upon request will destroy the information and remove it from
their records. Any views or opinions presented are solely those of the
author and do not necessarily represent those of Findel PLC. If you are not
the intended recipient, be advised that you have received this email in
error and that any use, dissemination, forwarding, printing, or copying of
this email is strictly prohibited. No warranties or assurances are made in
relation to the safety and content of this e-mail and any attachments. No
liability is accepted for any consequences arising from it. Findel Plc
reserves the right to monitor all e-mail communications through its
internal and external networks. If you have received this email in error
please notify our IT helpdesk on +44(0) 1254 303030****


****

** **

--
"While it is futile to try to eliminate risk, and questionable to try to
minimize it, it is essential that the risks taken be the right risks. "
- Peter F. Drucker ****

Name & Registered Office: EXPRESS GIFTS LIMITED, 2 GREGORY ST, HYDE,
CHESHIRE, ENGLAND, SK14 4TH, Company No. 00718151.
Express Gifts Limited is authorised and regulated by the Financial
Services Authority
NOTE: This email and any information contained within or attached in a
separate file is confidential and intended solely for the Individual to
whom it is addressed. The information or data included is solely for the
purpose indicated or previously agreed. Any information or data included
with this e-mail remains the property of Findel PLC and the recipient will
refrain from utilising the information for any purpose other than that
indicated and upon request will destroy the information and remove it from
their records. Any views or opinions presented are solely those of the
author and do not necessarily represent those of Findel PLC. If you are not
the intended recipient, be advised that you have received this email in
error and that any use, dissemination, forwarding, printing, or copying of
this email is strictly prohibited. No warranties or assurances are made in
relation to the safety and content of this e-mail and any attachments. No
liability is accepted for any consequences arising from it. Findel Plc
reserves the right to monitor all e-mail communications through its
internal and external networks. If you have received this email in error
please notify our IT helpdesk on +44(0) 1254 303030

-- 
"While it is futile to try to eliminate risk, and questionable to try to
minimize it, it is essential that the risks taken be the right risks. "
- Peter F. Drucker

list Ken Connell · Thu, 17 Nov 2011 12:58:41 +0000 ·

swatch will do the trick.

It dynamically watches whatever log(s) file(s) looking for something you specified in the config and then exe a script to do whatever or even update xymon for that matter... 


Ken Connell
Intermediate Network Engineer
Computer & Communication Services
Ryerson University
XXX Victoria St
RM AB50
Toronto, Ont
M5B 2K3
XXX-XXX-XXXX x6709


-----Original Message-----
From: Neil Simmonds <user-8188d25e65e4@xymon.invalid>
Sender: xymon-bounces at xymon.com
Date: Thu, 17 Nov 2011 08:05:13 
To: Xymon Mailing List<xymon at xymon.com>
Subject: [Xymon] Automated remedial actions

list Ralph Mitchell · Thu, 17 Nov 2011 08:47:09 -0500 ·

On Thu, Nov 17, 2011 at 3:05 AM, Neil Simmonds <

▸ quoted from Vernon Everett

user-8188d25e65e4@xymon.invalid> wrote:

 Hi all,****

** **

I’ve got a requirement to take a specific action on a unix server should
we get a specific message in a log file. ****

** **

I’m not sure this is even possible in Xymon and I suspect I’ll have to try
and work out some way of doing it with a script. ****

** **

Perhaps an external script that will somehow check for the message, run
the relevant command to fix it and report a warning alert so that we know
it has happened.****

** **

If there is some better way of doing it through Xymon I’d really like to
know how.

Xymon clients mainly just report things to the Xymon server.  The Xymon
server then evaluates the report to discover any error conditions.  It
doesn't send anything back to the client if an error is found.  So, if you
want the client to take remedial action, you're probably going to have to
write an external test script for the client to run.

Ralph Mitchell

Automated remedial actions 🔗 link

Automated remedial actions