proc : multiple alerts?
list Bakkies Gatvol
I want to have a system procs (cron ntpd) alert and a dba procs (weblogic blah blah) alert. Based on my current understanding this is not possible under 4.3 ? I can write code to do this - I just do not want to do unnecessary work! -- Bakkies
list Mike Burger
Of course it's possible.
In analysis.cfg, you have something like this:
HOST=some.host.name
PROC ntpd 1
PROC <insert appropriate regex here> <minimum #> <maximum #>
And in alerts.cfg:
HOST=some.host.name
MAIL user-a445539a35b3@xymon.invalid COLOR=<yellow or red>
--
Mike Burger
http://www.bubbanfriends.org
"It's always suicide-mission this, save-the-planet that. No one ever just
stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1
▸
I want to have a system procs (cron ntpd) alert and a dba procs (weblogic blah blah) alert. Based on my current understanding this is not possible under 4.3 ? I can write code to do this - I just do not want to do unnecessary work! -- Bakkies
list Bakkies Gatvol
That is one alert - I want to be able to do Host=appel PROC=ntpd alert a sysadmin Host=appel PROC=weblogic alert a dba
▸
Date: Fri, 4 Oct 2013 18:57:33 -0400 Subject: Re: [Xymon] proc : multiple alerts? From: user-cc5c6e80f4c5@xymon.invalid To: user-66e2e196cd54@xymon.invalid CC: xymon at xymon.com Of course it's possible. In analysis.cfg, you have something like this: HOST=some.host.name PROC ntpd 1 PROC <insert appropriate regex here> <minimum #> <maximum #> And in alerts.cfg: HOST=some.host.name MAIL user-a445539a35b3@xymon.invalid COLOR=<yellow or red> -- Mike Burger http://www.bubbanfriends.org "It's always suicide-mission this, save-the-planet that. No one ever just stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1I want to have a system procs (cron ntpd) alert and a dba procs (weblogic blah blah) alert. Based on my current understanding this is not possible under 4.3 ? I can write code to do this - I just do not want to do unnecessary work! -- Bakkies
list Paul Root
Write a script to put in alerts.cfg.
HOST=apple COLOR=red SERVICE=proc
SCRIPT /usr/local/scripts/xymAppelProcs $NG-SYSADM $NG_DBA
Then the script will have to look at the alert, and see which one it is. And alert the proper person.
From: Xymon [mailto:xymon
-user-2ce0a6b6a2e5@xymon.invalid] On Behalf Of Bakkies Gatvol
Sent: Saturday, October 05, 2013 6:44 PM
To: Xymon Mailing List
▸
Subject: Re: [Xymon] proc : multiple alerts?
That is one alert - I want to be able to do
Host=appel
PROC=ntpd
alert a sysadmin
Host=appel
PROC=weblogic
alert a dba
Date: Fri, 4 Oct 2013 18:57:33 -0400 Subject: Re: [Xymon] proc : multiple alerts? From: user-cc5c6e80f4c5@xymon.invalid<mailto:user-cc5c6e80f4c5@xymon.invalid> To: user-66e2e196cd54@xymon.invalid<mailto:user-66e2e196cd54@xymon.invalid> CC: xymon at xymon.com<mailto:xymon at xymon.com> Of course it's possible. In analysis.cfg, you have something like this: HOST=some.host.name PROC ntpd 1 PROC <insert appropriate regex here> <minimum #> <maximum #> And in alerts.cfg: HOST=some.host.name
MAIL user-a445539a35b3@xymon.invalid<mailto:user-a445539a35b3@xymon.invalid> COLOR=<yellow or red>
▸
-- Mike Burger http://www.bubbanfriends.org "It's always suicide-mission this, save-the-planet that. No one ever just stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1I want to have a system procs (cron ntpd) alert and a dba procs (weblogic blah blah) alert. Based on my current understanding this is not possible under 4.3 ? I can write code to do this - I just do not want to do unnecessary work! -- Bakkies
list Carl Melgaard
Hi, BUT - if the procs-column is already in the red-state, when the 2nd process goes bad, it wont trigger a notification... Regards, Carl Melgaard
▸
Fra: Xymon [mailto:xymon-bounces at xymon.com] På vegne af Root, Paul T
Sendt: 6. oktober 2013 03:52
Til: 'Bakkies Gatvol'; 'Xymon Mailing List'
Emne: Re: [Xymon] proc : multiple alerts?
Write a script to put in alerts.cfg.
HOST=apple COLOR=red SERVICE=proc
SCRIPT /usr/local/scripts/xymAppelProcs $NG-SYSADM $NG_DBA
Then the script will have to look at the alert, and see which one it is. And alert the proper person.
From: Xymon [mailto:xymon
-user-2ce0a6b6a2e5@xymon.invalid] On Behalf Of Bakkies Gatvol
Sent: Saturday, October 05, 2013 6:44 PM
To: Xymon Mailing List
Subject: Re: [Xymon] proc : multiple alerts?
That is one alert - I want to be able to do
Host=appel
PROC=ntpd
alert a sysadmin
Host=appel
PROC=weblogic
alert a dba
Date: Fri, 4 Oct 2013 18:57:33 -0400 Subject: Re: [Xymon] proc : multiple alerts? From: user-cc5c6e80f4c5@xymon.invalid<mailto:user-cc5c6e80f4c5@xymon.invalid> To: user-66e2e196cd54@xymon.invalid<mailto:user-66e2e196cd54@xymon.invalid> CC: xymon at xymon.com<mailto:xymon at xymon.com> Of course it's possible. In analysis.cfg, you have something like this: HOST=some.host.name PROC ntpd 1 PROC <insert appropriate regex here> <minimum #> <maximum #> And in alerts.cfg: HOST=some.host.name MAIL user-a445539a35b3@xymon.invalid<mailto:user-a445539a35b3@xymon.invalid> COLOR=<yellow or red> -- Mike Burger http://www.bubbanfriends.org "It's always suicide-mission this, save-the-planet that. No one ever just stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1I want to have a system procs (cron ntpd) alert and a dba procs (weblogic blah blah) alert. Based on my current understanding this is not possible under 4.3 ? I can write code to do this - I just do not want to do unnecessary work! -- Bakkies
list Betsy Schwartz
▸
BUT – if the procs-column is already in the red-state, when the 2nd process goes bad, it wont trigger a notification…
That depends on your rules. If you have procs set to alert every five minutes while it is red, you will get repeated notifications. However if someone *acks* or signs it out, you wont. There may be another way to slice this: -if one of these services is listening on a port, you can do a custom ports test in protocols.cfg and alert on that -if one of these services is writing a log file or moving files around, use a files or msgs test -if there's a web service, use the CONT= feature to create a named http test and create an alert on that Depending on the size of your shop , you may be able to sidestep another way. In our case, it turns out that giving the NOC privileges to run sudo /sbin/service along with some debugging documentation cut out a lot of pages all around :-) I did end up doing a custom test in one case because testing on /sbin/service status was deemed to be preferable to just looking for the process . If you have to do a custom test for a proc, at least it's a very short test. Once you've written "do a system call and alert on results" you should have a fairly generic and reusable piece of code.
list Paul Root
That's true. Could you do something with REPEAT? In this specific case, he could use the ntp test. Otherwise you he'll probably need to write his own external test.
▸
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Carl Melgaard
Sent: Monday, October 07, 2013 5:21 AM
To: 'xymon at xymon.com'
Subject: Re: [Xymon] proc : multiple alerts?
Hi,
BUT - if the procs-column is already in the red-state, when the 2nd process goes bad, it wont trigger a notification...
Regards,
Carl Melgaard
Fra: Xymon [mailto:xymon-bounces at xymon.com] På vegne af Root, Paul T
Sendt: 6. oktober 2013 03:52
Til: 'Bakkies Gatvol'; 'Xymon Mailing List'
Emne: Re: [Xymon] proc : multiple alerts?
Write a script to put in alerts.cfg.
HOST=apple COLOR=red SERVICE=proc
SCRIPT /usr/local/scripts/xymAppelProcs $NG-SYSADM $NG_DBA
Then the script will have to look at the alert, and see which one it is. And alert the proper person.
From: Xymon [mailto:xymon
-user-2ce0a6b6a2e5@xymon.invalid<mailto:-user-2ce0a6b6a2e5@xymon.invalid>] On Behalf Of Bakkies Gatvol
▸
Sent: Saturday, October 05, 2013 6:44 PM
To: Xymon Mailing List
Subject: Re: [Xymon] proc : multiple alerts?
That is one alert - I want to be able to do
Host=appel
PROC=ntpd
alert a sysadmin
Host=appel
PROC=weblogic
alert a dbaDate: Fri, 4 Oct 2013 18:57:33 -0400 Subject: Re: [Xymon] proc : multiple alerts? From: user-cc5c6e80f4c5@xymon.invalid<mailto:user-cc5c6e80f4c5@xymon.invalid> To: user-66e2e196cd54@xymon.invalid<mailto:user-66e2e196cd54@xymon.invalid> CC: xymon at xymon.com<mailto:xymon at xymon.com> Of course it's possible. In analysis.cfg, you have something like this: HOST=some.host.name PROC ntpd 1 PROC <insert appropriate regex here> <minimum #> <maximum #> And in alerts.cfg: HOST=some.host.name MAIL user-a445539a35b3@xymon.invalid<mailto:user-a445539a35b3@xymon.invalid> COLOR=<yellow or red> -- Mike Burger http://www.bubbanfriends.org "It's always suicide-mission this, save-the-planet that. No one ever just stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1I want to have a system procs (cron ntpd) alert and a dba procs (weblogic blah blah) alert. Based on my current understanding this is not possible under 4.3 ? I can write code to do this - I just do not want to do unnecessary work! -- Bakkies
list Ralph Mitchell
Speaking of reusabe code - it might be possible to re-purpose Jeremy Laidman's fs-test: https://wiki.xymonton.org/doku.php/monitors:fs-test It looks at the "disk" column and creates new columns for any filesystems that show non-green. Once the "disk" column goes green again, it deletes the extra columns. You could adapt that to look in the procs list and create new columns that group system procs in one place, dba procs in another, etc. Ralph Mitchell
▸
On Mon, Oct 7, 2013 at 7:12 AM, Betsy Schwartz <user-c61747246f66@xymon.invalid>wrote:
BUT – if the procs-column is already in the red-state, when the 2nd process goes bad, it wont trigger a notification…That depends on your rules. If you have procs set to alert every five minutes while it is red, you will get repeated notifications. However if someone *acks* or signs it out, you wont. There may be another way to slice this: -if one of these services is listening on a port, you can do a custom ports test in protocols.cfg and alert on that -if one of these services is writing a log file or moving files around, use a files or msgs test -if there's a web service, use the CONT= feature to create a named http test and create an alert on that Depending on the size of your shop , you may be able to sidestep another way. In our case, it turns out that giving the NOC privileges to run sudo /sbin/service along with some debugging documentation cut out a lot of pages all around :-) I did end up doing a custom test in one case because testing on /sbin/service status was deemed to be preferable to just looking for the process . If you have to do a custom test for a proc, at least it's a very short test. Once you've written "do a system call and alert on results" you should have a fairly generic and reusable piece of code.
list Bakkies Gatvol
I am leaning towards some fancy footwork in an alert script. I believe this will be do-able in xymon 5, so I am just going to code something in the interim.
▸
Date: Mon, 7 Oct 2013 07:25:12 -0400 From: user-00a5e44c48c0@xymon.invalid To: user-c61747246f66@xymon.invalid CC: xymon at xymon.com Subject: Re: [Xymon] proc : multiple alerts? Speaking of reusabe code - it might be possible to re-purpose Jeremy Laidman's fs-test: https://wiki.xymonton.org/doku.php/monitors:fs-test It looks at the "disk" column and creates new columns for any filesystems that show non-green. Once the "disk" column goes green again, it deletes the extra columns. You could adapt that to look in the procs list and create new columns that group system procs in one place, dba procs in another, etc. Ralph Mitchell On Mon, Oct 7, 2013 at 7:12 AM, Betsy Schwartz <user-c61747246f66@xymon.invalid> wrote:
BUT – if the procs-column is already in the red-state, when the 2nd process goes bad, it wont trigger a notification…
That depends on your rules. If you have procs set to alert every five minutes while it is red, you will get repeated notifications. However if someone *acks* or signs it out, you wont. There may be another way to slice this: -if one of these services is listening on a port, you can do a custom ports test in protocols.cfg and alert on that -if one of these services is writing a log file or moving files around, use a files or msgs test -if there's a web service, use the CONT= feature to create a named http test and create an alert on that Depending on the size of your shop , you may be able to sidestep another way. In our case, it turns out that giving the NOC privileges to run sudo /sbin/service along with some debugging documentation cut out a lot of pages all around :-) I did end up doing a custom test in one case because testing on /sbin/service status was deemed to be preferable to just looking for the process . If you have to do a custom test for a proc, at least it's a very short test. Once you've written "do a system call and alert on results" you should have a fairly generic and reusable piece of code.
list Mike Burger
I note quite a few responses suggesting using a script, and that would definitely work. Another option, if your Xymon server is a Linux/Unix server, might be to forward all alarms to a local account, and have that local account make use of procmail to forward the alarms to the appropriate teams/individuals, based on content.
▸
-- Mike Burger http://www.bubbanfriends.org "It's always suicide-mission this, save-the-planet that. No one ever just stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1
That is one alert - I want to be able to do Host=appel PROC=ntpd alert a sysadmin Host=appel PROC=weblogic alert a dbaDate: Fri, 4 Oct 2013 18:57:33 -0400 Subject: Re: [Xymon] proc : multiple alerts? From: user-cc5c6e80f4c5@xymon.invalid To: user-66e2e196cd54@xymon.invalid CC: xymon at xymon.com Of course it's possible. In analysis.cfg, you have something like this: HOST=some.host.name PROC ntpd 1 PROC <insert appropriate regex here> <minimum #> <maximum #> And in alerts.cfg: HOST=some.host.name MAIL user-a445539a35b3@xymon.invalid COLOR=<yellow or red> -- Mike Burger http://www.bubbanfriends.org "It's always suicide-mission this, save-the-planet that. No one ever just stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1I want to have a system procs (cron ntpd) alert and a dba procs (weblogic blah blah) alert. Based on my current understanding this is not possible under 4.3 ? I can write code to do this - I just do not want to do unnecessary work! -- Bakkies