Hobbit client executing a script to be proactive if a problem occurs?

14 messages in this thread

list Chris Wopat · Fri, 11 Apr 2008 08:56:02 -0500 ·

Hello,

I've been running hobbit for a few years, after converting from Big Brother. I actively use it to monitor hundreds of hosts, send alerts, etc. However, one thing I have yet to do is use any custom scripts for anything. Looking at docs, I see two ways that scripts are called:

1) an alert can call a script instead of email directly. This script is passed variables, and just about anything can be done *on the hobbit server* from here.

2) A script can be launched on the client, from ext/, when the hobbit clien starts. This script is generally used to add another column to hobbit.

What I'd like to do is execute a script *on the client* when an alert has happened. Say, if CPU goes red on something, I'd like a script to run on the cilent. I could force the issue with #1 above, but this seems like it would invovle the server having the script, likely having SSH keys setup to get into the client, then run the script.

Is there a cleaner way built in that will just say "if service FOO is red then on client run script ext/BAR.sh"?

Thanks
--Chris

list Phil Wild · Fri, 11 Apr 2008 23:16:08 +0800 ·

Hi Chris,

I think it really depends on what you are testing? If you are using the
standard hobbit client and the standard tests, most of the client side is
pretty basic, I guess you could call it a dumb client in a way as it does a
simple job of pulling the data out and sends it on without any intelligent
decisions being made about thresholds etc.

To do what you want, you either have to do as you say (set up keys from the
server etc and have the server perform an action after a threshold breach
from a script initiated/configured in the hobbit-alerts file.

Or, which would be much simpler, put some code in your monitoring script to
take action, but then you are starting to move away from the simplicity of
hobbit. It decomes even harder if you want to take action based on something
picked up in the standard tests (like CPU that you mentioned in your post).
You may need to write your own test/new column that monitors the same metric
but in a different light. In my view, an automated action based on a
detected event probably does not belong in the monitoring system. If a
failure can be expected and an automated action is known to fix the issue,
perhaps that should be built into the startup process of the application (a
watchdog process etc). Hobbit can then be used to monitor the log for a
restart event, or a failed restart event etc. Actually, thinking about it
more, building the intelligent action into the agent is an ok idea and you
also have the opportunity of capturing and transmitting additional
information about why something dies if you run an action to fix and it
failed etc. I am waffling... You still need extra security based
configuration steps on the client with sudo or ssh anyway to get around
access permission to restart something anyway as your client is running as
the hobbit userid so this brings the client configuration closer to an ssh
setup on the server. I don't think either way is perfect but both would do
what you want...

Perhaps this is something that belongs on a request for feature list of a
future release of hobbit.

The hobbit client installation to configure sudo to allow it to run commands
as other users (on admins acceptance during the installation of course).
The ability of the hobbit server to send a series of actions to the hobbit
client for execution via the hobbit communication channel. Sounds like
something that could have lots of uses if done well...

Thoughts?

Cheers

Phil

2008/4/11 Chris Wopat <user-8ece45634613@xymon.invalid>:

▸ quoted from Chris Wopat

Hello,

I've been running hobbit for a few years, after converting from Big
Brother. I actively use it to monitor hundreds of hosts, send alerts, etc.
However, one thing I have yet to do is use any custom scripts for anything.
Looking at docs, I see two ways that scripts are called:

1) an alert can call a script instead of email directly. This script is
passed variables, and just about anything can be done *on the hobbit server*
from here.

2) A script can be launched on the client, from ext/, when the hobbit
clien starts. This script is generally used to add another column to hobbit.

What I'd like to do is execute a script *on the client* when an alert has
happened. Say, if CPU goes red on something, I'd like a script to run on the
cilent. I could force the issue with #1 above, but this seems like it would
invovle the server having the script, likely having SSH keys setup to get
into the client, then run the script.

Is there a cleaner way built in that will just say "if service FOO is red
then on client run script ext/BAR.sh"?

Thanks
--Chris

--


Tel: XXXX XXX XXX
Fax: XXXX XXX XXX
email: user-e365c1418192@xymon.invalid

list Dave Haertig · Fri, 11 Apr 2008 10:30:11 -0600 ·

Here is how I execute a remote "pkill" on a client.  Replace
"client_server" with your client hostname, and replace "client_userid"
with the userid (on the client) that you want to run the script (pkill)
under.  Also, set up ssh pubkey authentication between the Hobbit server
and client so that ssh does not prompt you for a password.


hobbit_alerts.cfg:

##############################
$KILL_SCRIPT=/hobbit/server/ext/kill_script
HOST=client_server SERVICE=procs COLOR=red
        SCRIPT $KILL_SCRIPT client_server,client_userid,grep_pattern
##############################


/hobbit/server/ext/kill_script:

##############################
#!/bin/perl
#$debug++;      # Comment-out this line for normal operation
($host, $userid, $pattern) = split(/,/, $ENV{RCPT}, 3);
exit unless ((defined $host) && (defined $userid) && (defined
$pattern));
$command = qq!ssh $userid\@$host "pkill -f '$pattern'"!;
if ($debug) {
        $command =~ s/pkill/pgrep -l/;
        print `$command`;
}
else {
        `$command`;
}
##############################


And here is how I execute an arbitrary script on the client (in this
example, "weblogics_kill_script").  It is a minor modification of the
above:


hobbit_alerts.cfg:

##############################
$REMOTE_SCRIPT=/hobbit/server/ext/remote_script
HOST=ssdpws001 SERVICE=weblogics COLOR=red
        SCRIPT $REMOTE_SCRIPT
client_server,client_userid,/export/home/hobbit/scripts/weblogics_kill_s
cript
##############################


Note: weblogics_kill_script exists on the client - I use it to force a
few thread dumps, archive logfiles, and then kill/restart weblogics

/hobbit/server/ext/remote_script:

##############################
#!/bin/perl
#$debug++;      # Comment-out this line for normal operation
($host, $userid, $script) = split(/,/, $ENV{RCPT}, 3);
exit unless ((defined $host) && (defined $userid) && (defined $script));
$command = qq!ssh $userid\@$host "$script"!;
if ($debug) {
        print `$command`;
}
else {
        `$command`;
}
##############################


-----Original Message-----
From: Chris Wopat [mailto:user-8ece45634613@xymon.invalid] 
Sent: Friday, April 11, 2008 7:56 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Hobbit client executing a script to be proactive if a
problem occurs?

▸ quoted from Phil Wild


Hello,

I've been running hobbit for a few years, after converting from Big
Brother. I actively use it to monitor hundreds of hosts, send alerts,
etc. However, one thing I have yet to do is use any custom scripts for
anything. Looking at docs, I see two ways that scripts are called:

1) an alert can call a script instead of email directly. This script is
passed variables, and just about anything can be done *on the hobbit
server* from here.

2) A script can be launched on the client, from ext/, when the hobbit
clien starts. This script is generally used to add another column to
hobbit.

What I'd like to do is execute a script *on the client* when an alert
has happened. Say, if CPU goes red on something, I'd like a script to
run on the cilent. I could force the issue with #1 above, but this seems
like it would invovle the server having the script, likely having SSH
keys setup to get into the client, then run the script.

Is there a cleaner way built in that will just say "if service FOO is
red then on client run script ext/BAR.sh"?

Thanks
--Chris

list Dave Haertig · Fri, 11 Apr 2008 10:51:53 -0600 ·

Sorry about the formating of the code I sent.  I don't know why I can't
successfully cut-n-paste from vi on Linux into Outlook on Windows and
not get screwed up format-wise.  I guess if you know Hobbit alert syntax
and PERL, they you can figure it out easy enough.

▸ quoted from Dave Haertig

-----Original Message-----
From: Haertig, David F (Dave) [mailto:user-68874b735d77@xymon.invalid] 
Sent: Friday, April 11, 2008 10:30 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: RE: [hobbit] Hobbit client executing a script to be proactive
if a problem occurs?

Here is how I execute a remote "pkill" on a client.  Replace
"client_server" with your client hostname, and replace "client_userid"
with the userid (on the client) that you want to run the script (pkill)
under.  Also, set up ssh pubkey authentication between the Hobbit server
and client so that ssh does not prompt you for a password.

hobbit_alerts.cfg:

##############################
$KILL_SCRIPT=/hobbit/server/ext/kill_script
HOST=client_server SERVICE=procs COLOR=red
        SCRIPT $KILL_SCRIPT client_server,client_userid,grep_pattern
##############################

/hobbit/server/ext/kill_script:

##############################
#!/bin/perl
#$debug++;      # Comment-out this line for normal operation
($host, $userid, $pattern) = split(/,/, $ENV{RCPT}, 3); exit unless
((defined $host) && (defined $userid) && (defined $pattern)); $command =
qq!ssh $userid\@$host "pkill -f '$pattern'"!; if ($debug) {
        $command =~ s/pkill/pgrep -l/;
        print `$command`;
}
else {
        `$command`;
}
##############################

And here is how I execute an arbitrary script on the client (in this
example, "weblogics_kill_script").  It is a minor modification of the
above:

hobbit_alerts.cfg:

##############################
$REMOTE_SCRIPT=/hobbit/server/ext/remote_script
HOST=ssdpws001 SERVICE=weblogics COLOR=red
        SCRIPT $REMOTE_SCRIPT
client_server,client_userid,/export/home/hobbit/scripts/weblogics_kill_s
cript
##############################

Note: weblogics_kill_script exists on the client - I use it to force a
few thread dumps, archive logfiles, and then kill/restart weblogics

/hobbit/server/ext/remote_script:

##############################
#!/bin/perl
#$debug++;      # Comment-out this line for normal operation
($host, $userid, $script) = split(/,/, $ENV{RCPT}, 3); exit unless
((defined $host) && (defined $userid) && (defined $script)); $command =
qq!ssh $userid\@$host "$script"!; if ($debug) {
        print `$command`;
}
else {
        `$command`;
}
##############################

-----Original Message-----
From: Chris Wopat [mailto:user-8ece45634613@xymon.invalid]
Sent: Friday, April 11, 2008 7:56 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Hobbit client executing a script to be proactive if a
problem occurs?

Hello,

I've been running hobbit for a few years, after converting from Big
Brother. I actively use it to monitor hundreds of hosts, send alerts,
etc. However, one thing I have yet to do is use any custom scripts for
anything. Looking at docs, I see two ways that scripts are called:

1) an alert can call a script instead of email directly. This script is
passed variables, and just about anything can be done *on the hobbit
server* from here.

2) A script can be launched on the client, from ext/, when the hobbit
clien starts. This script is generally used to add another column to
hobbit.

What I'd like to do is execute a script *on the client* when an alert
has happened. Say, if CPU goes red on something, I'd like a script to
run on the cilent. I could force the issue with #1 above, but this seems
like it would invovle the server having the script, likely having SSH
keys setup to get into the client, then run the script.

Is there a cleaner way built in that will just say "if service FOO is
red then on client run script ext/BAR.sh"?

Thanks
--Chris

list Josh Luthman · Fri, 11 Apr 2008 12:56:21 -0400 ·

Assuming you're using SSH - what SSH client are you pasting from?  The
formating would be put in the buffer from the client, not VI at all.

I use puTTY and the formatting works perfectly for me.

On Fri, Apr 11, 2008 at 12:51 PM, Haertig, David F (Dave) <user-68874b735d77@xymon.invalid>

▸ quoted from Dave Haertig

wrote:

Sorry about the formating of the code I sent.  I don't know why I can't
successfully cut-n-paste from vi on Linux into Outlook on Windows and
not get screwed up format-wise.  I guess if you know Hobbit alert syntax
and PERL, they you can figure it out easy enough.

-----Original Message-----
From: Haertig, David F (Dave) [mailto:user-68874b735d77@xymon.invalid]
Sent: Friday, April 11, 2008 10:30 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: RE: [hobbit] Hobbit client executing a script to be proactive
if a problem occurs?

Here is how I execute a remote "pkill" on a client.  Replace
"client_server" with your client hostname, and replace "client_userid"
with the userid (on the client) that you want to run the script (pkill)
under.  Also, set up ssh pubkey authentication between the Hobbit server
and client so that ssh does not prompt you for a password.

hobbit_alerts.cfg:

##############################
$KILL_SCRIPT=/hobbit/server/ext/kill_script
HOST=client_server SERVICE=procs COLOR=red
       SCRIPT $KILL_SCRIPT client_server,client_userid,grep_pattern
##############################

/hobbit/server/ext/kill_script:

##############################
#!/bin/perl
#$debug++;      # Comment-out this line for normal operation
($host, $userid, $pattern) = split(/,/, $ENV{RCPT}, 3); exit unless
((defined $host) && (defined $userid) && (defined $pattern)); $command =
qq!ssh $userid\@$host "pkill -f '$pattern'"!; if ($debug) {
       $command =~ s/pkill/pgrep -l/;
       print `$command`;
}
else {
       `$command`;
}
##############################

And here is how I execute an arbitrary script on the client (in this
example, "weblogics_kill_script").  It is a minor modification of the
above:

hobbit_alerts.cfg:

##############################
$REMOTE_SCRIPT=/hobbit/server/ext/remote_script
HOST=ssdpws001 SERVICE=weblogics COLOR=red
       SCRIPT $REMOTE_SCRIPT
client_server,client_userid,/export/home/hobbit/scripts/weblogics_kill_s
cript
##############################

Note: weblogics_kill_script exists on the client - I use it to force a
few thread dumps, archive logfiles, and then kill/restart weblogics

/hobbit/server/ext/remote_script:

##############################
#!/bin/perl
#$debug++;      # Comment-out this line for normal operation
($host, $userid, $script) = split(/,/, $ENV{RCPT}, 3); exit unless
((defined $host) && (defined $userid) && (defined $script)); $command =
qq!ssh $userid\@$host "$script"!; if ($debug) {
       print `$command`;
}
else {
       `$command`;
}
##############################

-----Original Message-----
From: Chris Wopat [mailto:user-8ece45634613@xymon.invalid]
Sent: Friday, April 11, 2008 7:56 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Hobbit client executing a script to be proactive if a
problem occurs?

Hello,

I've been running hobbit for a few years, after converting from Big
Brother. I actively use it to monitor hundreds of hosts, send alerts,
etc. However, one thing I have yet to do is use any custom scripts for
anything. Looking at docs, I see two ways that scripts are called:

1) an alert can call a script instead of email directly. This script is
passed variables, and just about anything can be done *on the hobbit
server* from here.

2) A script can be launched on the client, from ext/, when the hobbit
clien starts. This script is generally used to add another column to
hobbit.

What I'd like to do is execute a script *on the client* when an alert
has happened. Say, if CPU goes red on something, I'd like a script to
run on the cilent. I could force the issue with #1 above, but this seems
like it would invovle the server having the script, likely having SSH
keys setup to get into the client, then run the script.

Is there a cleaner way built in that will just say "if service FOO is
red then on client run script ext/BAR.sh"?

Thanks
--Chris

--


Josh Luthman
Office: XXX-XXX-XXXX
Direct: XXX-XXX-XXXX
XXXX Wayne St
Suite XXXX
Troy, OH XXXXX

Those who don't understand UNIX are condemned to reinvent it, poorly.
--- Henry Spencer

list Dave Haertig · Fri, 11 Apr 2008 11:11:00 -0600 ·

My local computer is Windows XP, using Putty to ssh into my Linux box
(which runs Hobbit).  VI on Linux, highlight, then cut-n-paste into an
Outlook 2003 reply to the Hobbit mailing list.  I have only noticed this
formatting problem when posting code to this Hobbit list.  It's probably
some text-only email line-wrap formatting option I have set in Outlook.
It's not an issue in HTML email (which I hate, but when in Rome, do as
the Romans do, and HTML email is what most everybody in my company
uses).  I had to grudging give up on Thunderbird and move to Outlook so
I could easily deal with all the other Windows-loving coworkers and
their Outlook calendar meeting invites.
 
I'll check my Outlook text settings again.  There's probably something
in there that I missed.

▸ quoted from Josh Luthman



From: Josh Luthman [mailto:user-4c45a83f15cb@xymon.invalid] 
Sent: Friday, April 11, 2008 10:56 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Hobbit client executing a script to be proactive
if a problem occurs?


Assuming you're using SSH - what SSH client are you pasting from?  The
formating would be put in the buffer from the client, not VI at all.

I use puTTY and the formatting works perfectly for me.


On Fri, Apr 11, 2008 at 12:51 PM, Haertig, David F (Dave)
<user-68874b735d77@xymon.invalid> wrote:


	Sorry about the formating of the code I sent.  I don't know why
I can't
	successfully cut-n-paste from vi on Linux into Outlook on
Windows and
	not get screwed up format-wise.  I guess if you know Hobbit
alert syntax
	and PERL, they you can figure it out easy enough.
	

	-----Original Message-----
	From: Haertig, David F (Dave) [mailto:user-68874b735d77@xymon.invalid]
	Sent: Friday, April 11, 2008 10:30 AM
	To: user-ae9b8668bcde@xymon.invalid
	
	Subject: RE: [hobbit] Hobbit client executing a script to be
proactive
	if a problem occurs?
	
	Here is how I execute a remote "pkill" on a client.  Replace
	"client_server" with your client hostname, and replace
"client_userid"
	with the userid (on the client) that you want to run the script
(pkill)
	under.  Also, set up ssh pubkey authentication between the
Hobbit server
	and client so that ssh does not prompt you for a password.
	
	
	hobbit_alerts.cfg:
	
	##############################
	$KILL_SCRIPT=/hobbit/server/ext/kill_script
	HOST=client_server SERVICE=procs COLOR=red
	       SCRIPT $KILL_SCRIPT
client_server,client_userid,grep_pattern
	##############################
	
	
	/hobbit/server/ext/kill_script:
	
	##############################
	#!/bin/perl
	#$debug++;      # Comment-out this line for normal operation
	($host, $userid, $pattern) = split(/,/, $ENV{RCPT}, 3); exit
unless
	((defined $host) && (defined $userid) && (defined $pattern));
$command =
	qq!ssh $userid\@$host "pkill -f '$pattern'"!; if ($debug) {
	       $command =~ s/pkill/pgrep -l/;
	       print `$command`;
	}
	else {
	       `$command`;
	}
	##############################
	
	
	And here is how I execute an arbitrary script on the client (in
this
	example, "weblogics_kill_script").  It is a minor modification
of the
	above:
	
	
	hobbit_alerts.cfg:
	
	##############################
	$REMOTE_SCRIPT=/hobbit/server/ext/remote_script
	HOST=ssdpws001 SERVICE=weblogics COLOR=red
	       SCRIPT $REMOTE_SCRIPT
	
client_server,client_userid,/export/home/hobbit/scripts/weblogics_kill_s
	cript
	##############################
	
	
	Note: weblogics_kill_script exists on the client - I use it to
force a
	few thread dumps, archive logfiles, and then kill/restart
weblogics
	
	/hobbit/server/ext/remote_script:
	
	##############################
	#!/bin/perl
	#$debug++;      # Comment-out this line for normal operation
	($host, $userid, $script) = split(/,/, $ENV{RCPT}, 3); exit
unless
	((defined $host) && (defined $userid) && (defined $script));
$command =
	qq!ssh $userid\@$host "$script"!; if ($debug) {
	       print `$command`;
	}
	else {
	       `$command`;
	}
	##############################
	
	
	-----Original Message-----
	From: Chris Wopat [mailto:user-8ece45634613@xymon.invalid]
	Sent: Friday, April 11, 2008 7:56 AM
	To: user-ae9b8668bcde@xymon.invalid
	Subject: [hobbit] Hobbit client executing a script to be
proactive if a
	problem occurs?
	
	Hello,
	
	I've been running hobbit for a few years, after converting from
Big
	Brother. I actively use it to monitor hundreds of hosts, send
alerts,
	etc. However, one thing I have yet to do is use any custom
scripts for
	anything. Looking at docs, I see two ways that scripts are
called:
	
	1) an alert can call a script instead of email directly. This
script is
	passed variables, and just about anything can be done *on the
hobbit
	server* from here.
	
	2) A script can be launched on the client, from ext/, when the
hobbit
	clien starts. This script is generally used to add another
column to
	hobbit.
	
	What I'd like to do is execute a script *on the client* when an
alert
	has happened. Say, if CPU goes red on something, I'd like a
script to
	run on the cilent. I could force the issue with #1 above, but
this seems
	like it would invovle the server having the script, likely
having SSH
	keys setup to get into the client, then run the script.
	
	Is there a cleaner way built in that will just say "if service
FOO is
	red then on client run script ext/BAR.sh"?
	
	Thanks
	--Chris
	
	
-- 
Josh Luthman
Office: XXX-XXX-XXXX
Direct: XXX-XXX-XXXX
XXXX Wayne St
Suite XXXX
Troy, OH XXXXX

Those who don't understand UNIX are condemned to reinvent it, poorly.
--- Henry Spencer

list Chris Wopat · Fri, 11 Apr 2008 12:17:24 -0500 ·

▸ quoted from Phil Wild

Phil Wild wrote:

Hi Chris,

I think it really depends on what you are testing? If you are using the standard hobbit client and the standard tests, most of the client side is pretty basic, I guess you could call it a dumb client in a way as it does a simple job of pulling the data out and sends it on without any intelligent decisions being made about thresholds etc.

To do what you want, you either have to do as you say (set up keys from the server etc and have the server perform an action after a threshold breach from a script initiated/configured in the hobbit-alerts file.

Bummer, I was hoping there was perhaps a barely documented feature that would let you exec a script on the client to make my life easier.

Or, which would be much simpler, put some code in your monitoring script to take action, but then you are starting to move away from the simplicity of hobbit. It decomes even harder if you want to take action based on something picked up in the standard tests (like CPU that you mentioned in your post).

Indeed, my intent is for standard tests.

You may need to write your own test/new column that monitors the same metric but in a different light. In my view, an automated action based on a detected event probably does not belong in the monitoring system. If a failure can be expected and an automated action is known to fix the issue, perhaps that should be built into the startup process of the application (a watchdog process etc).

Indeed, a daemon shouldn't fail and it should run properly or be fixed natively. However, in the case of what I'm trying to monitor is an item that has a series of dependencies - Postfix, depending on Amavis, depending on p0f, depending on ClamAV, depending on greylist software, depending on database, etc. Under certain circumstances if one of these were to go down, it ends up snowballing to have high CPU, in my case.

The better way for me to handle this is to likely search logs for items, instead of relying on high CPU.

Hobbit can then be used to monitor the log for a restart event, or a failed restart event etc. Actually, thinking about it more, building the intelligent action into the agent is an ok idea and you also have the opportunity of capturing and transmitting additional information about why something dies if you run an action to fix and it failed etc.

For the scenario I laid out above, I intend to write a script that will restart the daemons properly in the correct order, but this is the "oh shit" script, and wouldn't be a system startup script, for example.

I am waffling... You still need extra security based configuration steps on the client with sudo or ssh anyway to get around access permission to restart something anyway as your client is running as the hobbit userid so this brings the client configuration closer to an ssh setup on the server. I don't think either way is perfect but both would do what you want...

Indeed. It sounds like generally the two scenarios I'd mentioned in my email are the way to get it to work, and whichever is most reliable/less hack-ish would be the best way to do it.

▸ quoted from Phil Wild

Perhaps this is something that belongs on a request for feature list of a future release of hobbit.

The hobbit client installation to configure sudo to allow it to run commands as other users (on admins acceptance during the installation of course).
The ability of the hobbit server to send a series of actions to the hobbit client for execution via the hobbit communication channel. Sounds like something that could have lots of uses if done well...

I think this would be the perfect scenario. Something added to the hobbit client, that would go in 'localclient.cfg'. A simple 'SCRIPT' line that could be nested under a test, that would pass along whatever meaningful variables that could be useful, such as PID.

The hobbit server *already uses* 'setuid root' for some binaries, such as 'hobbitping'. The client would simply need to call some binary who's sole purpose is to launch scripts as root so essentially anything would be possible.

I can't think of any reason that some hook would have to exist for the script to tell the hobbit client anything back, I think it can just wait for the next poll period to see if it went back to green.

--Chris

list Josh Luthman · Fri, 11 Apr 2008 13:26:57 -0400 ·

David,

Sorry for your loss =(  I know of many people that are being forced into
Outlook.

I use puTTY and Gmail exclusively.  I'm using rich text (Gmail will not send
out HTML) which VI seems to like.

Anyways, just thought I'd help you eliminate issues - I think we can all
agree Outlook is the cause of the copy/paste malfunction.

Josh

▸ quoted from Chris Wopat


On Fri, Apr 11, 2008 at 1:17 PM, Chris Wopat <user-8ece45634613@xymon.invalid> wrote:

Phil Wild wrote:

Hi Chris,

I think it really depends on what you are testing? If you are using the
standard hobbit client and the standard tests, most of the client side is
pretty basic, I guess you could call it a dumb client in a way as it does a
simple job of pulling the data out and sends it on without any intelligent
decisions being made about thresholds etc.

To do what you want, you either have to do as you say (set up keys from
the server etc and have the server perform an action after a threshold
breach from a script initiated/configured in the hobbit-alerts file.

Bummer, I was hoping there was perhaps a barely documented feature that
would let you exec a script on the client to make my life easier.


 Or, which would be much simpler, put some code in your monitoring script

to take action, but then you are starting to move away from the simplicity
of hobbit. It decomes even harder if you want to take action based on
something picked up in the standard tests (like CPU that you mentioned in
your post).

Indeed, my intent is for standard tests.


 You may need to write your own test/new column that monitors the same

metric but in a different light. In my view, an automated action based on a
detected event probably does not belong in the monitoring system. If a
failure can be expected and an automated action is known to fix the issue,
perhaps that should be built into the startup process of the application (a
watchdog process etc).

Indeed, a daemon shouldn't fail and it should run properly or be fixed
natively. However, in the case of what I'm trying to monitor is an item that
has a series of dependencies - Postfix, depending on Amavis, depending on
p0f, depending on ClamAV, depending on greylist software, depending on
database, etc. Under certain circumstances if one of these were to go down,
it ends up snowballing to have high CPU, in my case.

The better way for me to handle this is to likely search logs for items,
instead of relying on high CPU.


 Hobbit can then be used to monitor the log for a restart event, or a

failed restart event etc. Actually, thinking about it more, building the
intelligent action into the agent is an ok idea and you also have the
opportunity of capturing and transmitting additional information about why
something dies if you run an action to fix and it failed etc.

For the scenario I laid out above, I intend to write a script that will
restart the daemons properly in the correct order, but this is the "oh shit"
script, and wouldn't be a system startup script, for example.


 I am waffling... You still need extra security based configuration steps

on the client with sudo or ssh anyway to get around access permission to
restart something anyway as your client is running as the hobbit userid so
this brings the client configuration closer to an ssh setup on the server. I
don't think either way is perfect but both would do what you want...

Indeed. It sounds like generally the two scenarios I'd mentioned in my
email are the way to get it to work, and whichever is most reliable/less
hack-ish would be the best way to do it.


 Perhaps this is something that belongs on a request for feature list of a

future release of hobbit.

The hobbit client installation to configure sudo to allow it to run
commands as other users (on admins acceptance during the installation of
course).
The ability of the hobbit server to send a series of actions to the
hobbit client for execution via the hobbit communication channel. Sounds
like something that could have lots of uses if done well...

I think this would be the perfect scenario. Something added to the hobbit
client, that would go in 'localclient.cfg'. A simple 'SCRIPT' line that
could be nested under a test, that would pass along whatever meaningful
variables that could be useful, such as PID.

The hobbit server *already uses* 'setuid root' for some binaries, such as
'hobbitping'. The client would simply need to call some binary who's sole
purpose is to launch scripts as root so essentially anything would be
possible.

I can't think of any reason that some hook would have to exist for the
script to tell the hobbit client anything back, I think it can just wait for
the next poll period to see if it went back to green.


--Chris

-- 
Josh Luthman
Office: XXX-XXX-XXXX
Direct: XXX-XXX-XXXX
XXXX Wayne St
Suite XXXX
Troy, OH XXXXX

Those who don't understand UNIX are condemned to reinvent it, poorly.
--- Henry Spencer

list Chris Wopat · Fri, 11 Apr 2008 13:51:16 -0500 ·

▸ quoted from Dave Haertig

Haertig, David F (Dave) wrote:

Here is how I execute a remote "pkill" on a client.  Replace
"client_server" with your client hostname, and replace "client_userid"
with the userid (on the client) that you want to run the script (pkill)
under.  Also, set up ssh pubkey authentication between the Hobbit server
and client so that ssh does not prompt you for a password.


hobbit_alerts.cfg:

<snip>


Thanks, this is exactly what I needed to get started. I can un-wrap the lines no problem, only a few were anyway.

I'm assuming you're using ssh keys? My current hobbit server installation (from FreeBSD ports) has no home dir set, so it looks like I'll have to set one to store its side of the keys.

--Chris

list Chris Wopat · Fri, 11 Apr 2008 15:25:45 -0500 ·

▸ quoted from Chris Wopat

Chris Wopat wrote:

Haertig, David F (Dave) wrote:

Here is how I execute a remote "pkill" on a client.  Replace
"client_server" with your client hostname, and replace "client_userid"
with the userid (on the client) that you want to run the script (pkill)
under.  Also, set up ssh pubkey authentication between the Hobbit server
and client so that ssh does not prompt you for a password.


hobbit_alerts.cfg:

<snip>

Thanks, this is exactly what I needed to get started. I can un-wrap the lines no problem, only a few were anyway.

I'm assuming you're using ssh keys? My current hobbit server installation (from FreeBSD ports) has no home dir set, so it looks like I'll have to set one to store its side of the keys.

d'oh, I just realized you already wrote that in oyur original reply. I've got everything set up, now to wait for it to go red!

--Chris

list Dave Haertig · Fri, 11 Apr 2008 14:40:00 -0600 ·

Yes, all of my Hobbit clients have ssh authorized_keys setup to allow
the Hobbit server in without password.  In the case where I need to run
a script on the Hobbit client under a different userid than 'hobbit',
that other userid also has the Hobbit servers pubkey in its
authorized_keys file.  Alternately, you gould use setuid scripts on the
client (very unsecure), or use "expect" in your Hobbit server scripts
and "sudo" on the client end to gain access to the userid you need.
This second expect/sudo route is doable, but messy and requires you to
have the 'hobbit' password from the client end stored on your server
(not the most secure thing).

When coworkers ask me to use Hobbit to "fix" something on their client
end I council them that Hobbit really is an alerting system and not a
repairing system.  But if they really want me to attempt an automated
"repair", then they have to put my pubkey into their authorized_keys
files, therefore giving me full access to their userid on the client
machine.  I also mandate that if they want me to restart a process, that
I will only kill it.  The restart must be their responsibility (using a
local cronjob or whatever).  This further insulates me from any
political fallback regarding a failed automated repair attempt on an
errant process.  Further insolation for me is provided by me informing
everyone (management too) that they can shut off my automated Hobbit
repairs at any time, instantaneously, by simply removing my pubkey from
their authorized_keys file(s).  It's called "CYA" for when I am
pressured to make Hobbit do something that it rally wasn't designed for.

▸ quoted from Chris Wopat


-----Original Message-----
From: Chris Wopat [mailto:user-8ece45634613@xymon.invalid] 
Sent: Friday, April 11, 2008 12:51 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Hobbit client executing a script to be proactive
if a problem occurs?

Haertig, David F (Dave) wrote:

Here is how I execute a remote "pkill" on a client.  Replace 
"client_server" with your client hostname, and replace "client_userid"
with the userid (on the client) that you want to run the script 
(pkill) under.  Also, set up ssh pubkey authentication between the 
Hobbit server and client so that ssh does not prompt you for a
password.


hobbit_alerts.cfg:

<snip>

Thanks, this is exactly what I needed to get started. I can un-wrap the
lines no problem, only a few were anyway.

I'm assuming you're using ssh keys? My current hobbit server
installation (from FreeBSD ports) has no home dir set, so it looks like
I'll have to set one to store its side of the keys.

--Chris

list Henrik Størner · Mon, 14 Apr 2008 07:43:01 +0200 ·

▸ quoted from Chris Wopat

On Fri, Apr 11, 2008 at 08:56:02AM -0500, Chris Wopat wrote:

What I'd like to do is execute a script *on the client* when an alert has happened. Say, if CPU goes red on something, I'd like a script to run on the cilent. I could force the issue with #1 above, but this seems like it would invovle the server having the script, likely having SSH keys setup to get into the client, then run the script.

Is there a cleaner way built in that will just say "if service FOO is red then on client run script ext/BAR.sh"?

There is. You use the Hobbit "query" command to probe the status, and
then act accordingly. To use your example with the "cpu" status going
red:

   #!/bin/sh

   CPUCOLOR=`$BB $BBDISP "query $MACHINE.cpu" | awk '{print $1}'`
   if test "$CPUCOLOR" = "red"
   then
       # Kill some cpu hogging processes
       .... whatever it takes ....
   fi
   exit 0

The "query" command returns the first line of the current status, and
the first word is the current color. You probably will want to put in
some sort of sanity check so you don't trigger this too often. Or
perhaps just log the fact that it triggered.


Regards,
Henrik

list Chris Wopat · Mon, 14 Apr 2008 12:00:53 -0500 ·

▸ quoted from Henrik Størner

Henrik Stoerner wrote:

On Fri, Apr 11, 2008 at 08:56:02AM -0500, Chris Wopat wrote:

Is there a cleaner way built in that will just say "if service FOO is red then on client run script ext/BAR.sh"?

There is. You use the Hobbit "query" command to probe the status, and
then act accordingly. To use your example with the "cpu" status going
red:

   #!/bin/sh

   CPUCOLOR=`$BB $BBDISP "query $MACHINE.cpu" | awk '{print $1}'`
   if test "$CPUCOLOR" = "red"
   then
       # Kill some cpu hogging processes
       .... whatever it takes ....
   fi
   exit 0

The "query" command returns the first line of the current status, and
the first word is the current color. You probably will want to put in
some sort of sanity check so you don't trigger this too often. Or
perhaps just log the fact that it triggered.

Excellent, was unaware of this query feature. The script above looks like it would perfectly deal with my situation if I were to put it in cron on the client machine, which may be suitable.

Is there some way to tell the hobbit server to execute this script on the client, or would something in a SCRIPT line in a hobbit alert, likely using ssh keys, be the only way?

--Chris

list Phil Wild · Tue, 15 Apr 2008 10:04:18 +0800 ·

Hi Chris,

As far as I know, you would need to use a SCRIPT line in hobbit-alerts. The
example that Henrik gave is quite simple and I would be very careful about
how you use it... For example, if you run a backup, or a one off job that
does something CPU intensive that would not be considered a fault, yet it
did trigger a CPU threshold breach, you may actually triger an event that
kills and restarts a perfectly healthy process.

Your best approach is to see if there is something that is very unique about
your problem and build a test around it, for example, use ps or top to give
you the process using the most cpu, if that process is your usual suspect
and cpu usage is greater than 80% and it has been that way for five minutes,
then take an action and report up to hobbit that an action has been taken...

Cheers

Phil

2008/4/15 Chris Wopat <user-8ece45634613@xymon.invalid>:

▸ quoted from Chris Wopat

Henrik Stoerner wrote:

On Fri, Apr 11, 2008 at 08:56:02AM -0500, Chris Wopat wrote:

Is there a cleaner way built in that will just say "if service FOO is
red then on client run script ext/BAR.sh"?

There is. You use the Hobbit "query" command to probe the status, and
then act accordingly. To use your example with the "cpu" status going
red:

  #!/bin/sh

  CPUCOLOR=`$BB $BBDISP "query $MACHINE.cpu" | awk '{print $1}'`
  if test "$CPUCOLOR" = "red"
  then
      # Kill some cpu hogging processes
      .... whatever it takes ....
  fi
  exit 0

The "query" command returns the first line of the current status, and
the first word is the current color. You probably will want to put in
some sort of sanity check so you don't trigger this too often. Or
perhaps just log the fact that it triggered.

Excellent, was unaware of this query feature. The script above looks like
it would perfectly deal with my situation if I were to put it in cron on the
client machine, which may be suitable.

Is there some way to tell the hobbit server to execute this script on the
client, or would something in a SCRIPT line in a hobbit alert, likely using
ssh keys, be the only way?

--Chris

-- 
Tel: XXXX XXX XXX
Fax: XXXX XXX XXX
email: user-e365c1418192@xymon.invalid

Hobbit client executing a script to be proactive if a problem occurs? 🔗 link

Hobbit client executing a script to be proactive if a problem occurs?