execute a command from server (browser) on a client
list Deepak Deore
Is it possible to execute a command on client from xymon browser? eg. 1.If a service is in RED state, just select that service and fire restart command from browser. 2. If tomcat not responding then take thread dump from browser by selecting that particular tomcat. I can write html for gui.
list Ralph Mitchell
It could be done, but it's a bit more complicated than that. Your browser is talking to the Xymon server, not the server where the downed service should be running. Here's a couple of ways it could be done, using your tomcat example: 1) you would need a cgi script on the Xymon server capable of logging in to the remote tomcat server with enough privilege to be able to restart tomcat or to dump memory, then send back to your browser any restart messages or the memory dump as a file download; or 2) a cgi script on the xymon server that could add a variable to the client-local.cfg for the tomcat server. Next time the remote checks in with xymon it would pick up that flag and store it in $HOME/etc/local.cfg. A cron job could examine that file occasionally and if the flag shows up, restart or dump memory as appropriate. This cycle could take 10 - 15 minutes to complete. I don't know if anyone has such automation currently running. Ralph Mitchell
▸
On Thu, Jul 4, 2013 at 7:24 AM, deepak deore <user-7b03b2a1ee70@xymon.invalid>wrote:
Is it possible to execute a command on client from xymon browser? eg. 1.If a service is in RED state, just select that service and fire restart command from browser. 2. If tomcat not responding then take thread dump from browser by selecting that particular tomcat. I can write html for gui.
list Another Xymon User
Thanks, Ralph! I had case 1 working with a PKI trust relationship for root between the Xymon servers and the clients, but when the auditors made us set "NoRootLogin yes" in sshd_config everywhere, it broke and I had not had a chance to figure out how to do it with sudo. Case 2 will let me work up an alternative. On 2013-07-04 12:07, Ralph
▸
Mitchell wrote: It could be done, but it's a bit more complicated than that. Your browser is talking to the Xymon server, not the server where the downed service should be running. Here's a couple of ways it could be done, using your tomcat example: 1) you would need a cgi script on the Xymon server capable of logging in to the remote tomcat server with enough privilege to be able to restart tomcat or to dump memory, then send back to your browser any restart messages or the memory dump as a file download; or 2) a cgi script on the xymon server that could add a variable to the client-local.cfg for the tomcat server. Next time the remote checks in with xymon it would pick up that flag and store it in $HOME/etc/local.cfg. A cron job could examine that file occasionally and if the flag shows up, restart or dump memory as appropriate. This cycle could take 10 - 15 minutes to complete. I don't know if anyone has such automation currently running. Ralph
Mitchell
On Thu, Jul 4, 2013 at 7:24 AM, deepak deore
<user-7b03b2a1ee70@xymon.invalid> wrote:
Is it possible to execute a command on client from xymon browser? eg. 1.If a service is in RED state, just select that service and fire restart command from browser. 2. If tomcat not responding then take thread dump from browser by selecting that particular tomcat. I can write html for gui.
list Ralph Mitchell
You should be very careful about how you validate this kind of automation. The client should probably do some kind of verification, and use canned scripts rather than just running any command handed to it. For example: xymon adds to client-local.cfg for server1: restartapache:`date+%s` server1 xymon client saves that in $HOME/etc/local.cfg server1 cron job sees "restartapache", extracts the timestamp, checks that it didn't already do that, then posts that timestamp back to the xymon server, asking what it means; xymon server replies: restartapache As long as the reply matches, the client kicks apache Everybody writes log entries The client call-back to verify the command could be done with curl over https with a client-side SSL certificate to validate each party to the other. That may seem like overkill, but when auditors are involved, maybe not... :-) Ralph Mitchell On Thu, Jul 4, 2013 at 3:21 PM, Another Xymon User <
▸
user-b71bfe4edfbc@xymon.invalid> wrote:
** Thanks, Ralph! I had case 1 working with a PKI trust relationship for root between the Xymon servers and the clients, but when the auditors made us set "NoRootLogin yes" in sshd_config everywhere, it broke and I had not had a chance to figure out how to do it with sudo. Case 2 will let me work up an alternative. On 2013-07-04 12:07, Ralph Mitchell wrote: It could be done, but it's a bit more complicated than that. Your browser is talking to the Xymon server, not the server where the downed service should be running. Here's a couple of ways it could be done, using your tomcat example: 1) you would need a cgi script on the Xymon server capable of logging in to the remote tomcat server with enough privilege to be able to restart tomcat or to dump memory, then send back to your browser any restart messages or the memory dump as a file download; or 2) a cgi script on the xymon server that could add a variable to the client-local.cfg for the tomcat server. Next time the remote checks in with xymon it would pick up that flag and store it in $HOME/etc/local.cfg. A cron job could examine that file occasionally and if the flag shows up, restart or dump memory as appropriate. This cycle could take 10 - 15 minutes to complete. I don't know if anyone has such automation currently running. Ralph Mitchell On Thu, Jul 4, 2013 at 7:24 AM, deepak deore <user-7b03b2a1ee70@xymon.invalid>wrote:Is it possible to execute a command on client from xymon browser? eg. 1.If a service is in RED state, just select that service and fire restart command from browser. 2. If tomcat not responding then take thread dump from browser by selecting that particular tomcat. I can write html for gui.
Xymon mailing user-d459c9d661b6@xymon.invalid
list Another Xymon User
Actually, I'm thinking that since I already have the PKI stuff in place, the server just puts a private-key encrypted https URL to a script in client-local.cfg, and the client decrypts and retrieves it. One thing I've learned in 30 years in this business is that you never give the auditors everything you can do at the outset. Let them make a finding, then wrap something else around it. On 2013-07-04 17:44,
▸
Ralph Mitchell wrote: You should be very careful about how you validate this kind of automation. The client should probably do some kind of verification, and use canned scripts rather than just running any command handed to it. For example: xymon adds to client-local.cfg for server1: restartapache:`date+%s` server1 xymon client saves that in $HOME/etc/local.cfg server1 cron job sees
"restartapache", extracts the timestamp, checks that it didn't already do that, then posts that timestamp back to the xymon server, asking what it means;
xymon server replies: restartapache As long as the reply matches, the client kicks apache Everybody writes log entries
The client call-back to verify the command could be done with curl over https with a client-side SSL certificate to validate each party to the other. That may seem like overkill, but when auditors are involved, maybe not... :-)
Ralph Mitchell On Thu, Jul 4, 2013 at 3:21
PM, Another Xymon User <user-b71bfe4edfbc@xymon.invalid> wrote:
Thanks, Ralph! I had case 1 working with a PKI trust relationship for root between the Xymon servers and the clients, but when the auditors made us set "NoRootLogin yes" in sshd_config everywhere, it broke and I had not had a chance to figure out how to do it with sudo. Case 2 will let me work up an alternative.
On 2013-07-04 12:07, Ralph
Mitchell wrote:
It could be done, but it's a bit more complicated than that. Your browser is talking to the Xymon server, not the server where the downed service should be running. Here's a couple of ways it could be done, using your tomcat example: 1) you would need a cgi script on the Xymon server capable of logging in to the remote tomcat server with enough privilege to be able to restart tomcat or to dump memory, then send back to your browser any restart messages or the memory dump as a file download; or 2) a cgi script on the xymon server that could add a variable to the client-local.cfg for the tomcat server. Next time the remote checks in with xymon it would pick up that flag and store it in $HOME/etc/local.cfg. A cron job could examine that file occasionally and if the flag shows up, restart or dump memory as appropriate. This cycle could take 10 - 15 minutes to complete. I don't know if anyone has such automation currently running. Ralph Mitchell On Thu, Jul 4, 2013 at 7:24
AM, deepak deore <user-7b03b2a1ee70@xymon.invalid> wrote:
Is it possible to execute a command on client from xymon browser?
eg.
1.If a service is in RED state, just select that service and fire restart command from browser. 2. If tomcat not responding then take thread dump from browser by selecting that particular tomcat. I can write html for gui. Xymon mailing
listXymon mailing listXymon mailing list
[1]
list Jeremy Laidman
▸
On 5 July 2013 07:44, Ralph Mitchell <user-00a5e44c48c0@xymon.invalid> wrote:
You should be very careful about how you validate this kind of automation. The client should probably do some kind of verification, and use canned scripts rather than just running any command handed to it.
Yes. With ssh, authorized_keys can force a particular command to run, rather than let an attacker dictate what gets executed.
For example: xymon adds to client-local.cfg for server1: restartapache:`date+%s`
This is ingenious. However it should be noted that it can take quite some time for changes in client-local.cfg to be actioned on the client. Expect a 10-15 minute delay between the change to client-local.cfg and when the cron task picks it up - and that's if it runs every minute. It would be more responsive for the client cron task to just query the status from the server every minute, and act accordingly. Something like: #!/usr/lib/xymon/client/bin/xymoncmd /bin/sh $XYMON $XYMSRV "query $MACHINE.http" | grep ^green >/dev/null || sudo service httpd restart $XYMON $XYMSRV "query $MACHINE.smtp" | grep ^green >/dev/null || sudo service postfix restart This kind of automated restart can get you into a bit of trouble, so should be used with caution. The last thing you want is a stack of run-away tasks that end up consuming all memory and CPU resources. A change to make this a manually-triggered process might be for the client cron task to look for a disabled/acknowledged status using xymondboard, and detect when a special keyword (eg "!!RESTART!!") is entered into the disable/ack message. Something like this: #!/usr/lib/xymon/client/bin/xymoncmd /bin/sh RESPONSE=`$XYMON $XYMSRV "xymondboard host=$MACHINE test=http color=blue fields=lastchange,dismsg" | grep '!!RESTART!!' | cut -d"|" -f1` [ `expr 0$RESPONSE + 60` -gt `date +%s` ] && sudo service httpd restart This gets the status time for the host and test, but only if it's blue (disabled) and only if it contains the '!!RESTART!!' keyword. The status time is compared to the current time and if within 60 seconds, runs the restart command. J
list Ralph Mitchell
▸
On Jul 4, 2013 10:58 PM, "Jeremy Laidman" <user-71895fb2e44c@xymon.invalid> wrote:
On 5 July 2013 07:44, Ralph Mitchell <user-00a5e44c48c0@xymon.invalid> wrote:You should be very careful about how you validate this kind of automation. The client should probably do some kind of verification, and use canned scripts rather than just running any command handed to it.Yes. With ssh, authorized_keys can force a particular command to run, rather than let an attacker dictate what gets executed.
I was thinking more of having restart scripts that vary from one machine to another. E.g. on one server just restart apache, on another perhaps clear cache and tmp files first. Xymon doesn't need to care about that.
▸
For example: xymon adds to client-local.cfg for server1: restartapache:`date+%s`This is ingenious. However it should be noted that it can take quite some time for changes in client-local.cfg to be actioned on the client.
Expect a 10-15 minute delay between the change to client-local.cfg and when the cron task picks it up - and that's if it runs every minute.
I already noted the delay. The action check script doesn't need to run from
cron every minute. It can be hung on the end of the xymonclient.sh, to
execute right after the local.cfg has been received when the client data
message has been sent.
▸
It would be more responsive for the client cron task to just query the status from the server every minute, and act accordingly. Something like: #!/usr/lib/xymon/client/bin/xymoncmd /bin/sh $XYMON $XYMSRV "query $MACHINE.http" | grep ^green >/dev/null || sudo service httpd restart $XYMON $XYMSRV "query $MACHINE.smtp" | grep ^green >/dev/null || sudo service postfix restart
I've swapped out the client's xymon binary with a script that delivers the status report via secure https using curl posting to xymoncgimsg. I get back the local.cfg, but I haven't tried a query like that, though.
▸
This kind of automated restart can get you into a bit of trouble, so should be used with caution. The last thing you want is a stack of run-away tasks that end up consuming all memory and CPU resources. A change to make this a manually-triggered process might be for the client cron task to look for a disabled/acknowledged status using xymondboard, and detect when a special keyword (eg "!!RESTART!!") is entered into the disable/ack message. Something like this:
The original poster asked about getting the Xymon gui to initiate restarts. I threw the in the timestamp as a way to limit multiple retries. When the client sees the restart message it can query a CGI on the Xymon server to see if the action still needs to be taken The CGI can look in Xymon and see the current color. If its green, then the answer if no. If red, do the restart, if blue, answer no because maybe the service is supposed to be down. I think there would be less traffic that way.
▸
#!/usr/lib/xymon/client/bin/xymoncmd /bin/sh RESPONSE=`$XYMON $XYMSRV "xymondboard host=$MACHINE test=http color=blue fields=lastchange,dismsg" | grep '!!RESTART!!' | cut -d"|" -f1` [ `expr 0$RESPONSE + 60` -gt `date +%s` ] && sudo service httpd restart This gets the status time for the host and test, but only if it's blue
(disabled) and only if it contains the '!!RESTART!!' keyword. The status time is compared to the current time and if within 60 seconds, runs the restart command.
J
Ralph Mitchell
list Deepak Deore
Thanks for the suggestions. We are already using the below approach. I thought there may be more proper way to do this like Nagios has in-built option to send restart command remotely from browser.
▸
1) you would need a cgi script on the Xymon server capable of logging in to
the remote tomcat server with enough privilege to be able to restart tomcat
or to dump memory, then send back to your browser any restart messages or
the memory dump as a file download.