hobbit.sh not stopping server correctly
list E-mail j.sansford
Hi, I think I have an issue with 4.2.3 where the hobbit.sh is not stopping the server correctly. Running "./hobbit.sh stop" doesn't stop all the hobbit processes and so I can't install this as a service currently. Could anyone help me to debug why this is happening? I'm not really sure where to go from here. Many thanks James
list Buchan Milne
▸
On Monday, 17 August 2009 11:04:36 user-c15424b7e83a@xymon.invalid wrote:
Hi, I think I have an issue with 4.2.3 where the hobbit.sh is not stopping the server correctly. Running "./hobbit.sh stop" doesn't stop all the hobbit processes
Which processes remain? A vmstat running as the hobbit / xymon user?
▸
and so I can't install this as a service currently. Could anyone help me to debug why this is happening? I'm not really sure where to go from here.
It's a bit difficult with no information. A stray vmstat process after stopping xymon is expected, and using the example init scripts provided in the debian or rpm directories works on many distros ... Regards, Buchan
list E-mail j.sansford
Hi Buchan, Apologies, I'll explain it in more detail. We are running on Solaris 10 and trying to configure Xymon as an SMF service (which essentially uses stop/start scripts). The stop script is basically: kill -TERM `cat /export/home/hobbit/logs/hobbitlaunch.pid` Where hobbitlaunch.pid is the PID of the hobbitlaunch process. When running this command we are left with many stray processes, not just a vmstat. For example: /usr/local/bin/ssh -l root 10.6.6.6 sysstat -m -c1 290 sh -c (/usr/local/bin/ssh -l root 10.6.6.5 sysstat -x -c1 290 >/export/home/hob Doing a bit more digging it seems these are related to an ext/netapp monitor perl script. Taking this out leaves less processes (although I don't want to leave it out forever so I'll have to do a bit of digging around to find out how to kill these gracefully) however because the iostats do not finish the SMF reports a failure rather than a disable when I try to stop the monitor. This means the SMF can't restart the bbserver if it ever crashes.
▸
----- Original Message -----
From: "Buchan Milne" <user-9b139aff4dec@xymon.invalid>
To: <user-ae9b8668bcde@xymon.invalid>
Cc: <user-c15424b7e83a@xymon.invalid>
Sent: Monday, August 17, 2009 3:52 PM
Subject: Re: [hobbit] hobbit.sh not stopping server correctly
On Monday, 17 August 2009 11:04:36 user-c15424b7e83a@xymon.invalid wrote:Hi, I think I have an issue with 4.2.3 where the hobbit.sh is not stopping the server correctly. Running "./hobbit.sh stop" doesn't stop all the hobbit processesWhich processes remain? A vmstat running as the hobbit / xymon user?and so I can't install this as a service currently. Could anyone help me to debug why this is happening? I'm not really sure where to go from here.It's a bit difficult with no information. A stray vmstat process after stopping xymon is expected, and using the example init scripts provided in the debian or rpm directories works on many distros ... Regards, Buchan
list Jef Jagers
Hi Buchan, Just a stuppid question. But have you stopped and the hobbit server and the hobbit client on that machine or only the server? Regards, Jef Jagers Systems Engineer Thomson CompuMark Thomson Reuters T +32 3 220 76 02
▸
-----Original Message-----
From: user-c15424b7e83a@xymon.invalid [mailto:user-c15424b7e83a@xymon.invalid]
Sent: dinsdag, augustus 18, 2009 13:25
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] hobbit.sh not stopping server correctly
Hi Buchan,
Apologies, I'll explain it in more detail. We are running on Solaris 10 and trying to configure Xymon as an SMF service (which essentially uses stop/start scripts). The stop script is basically:
kill -TERM `cat /export/home/hobbit/logs/hobbitlaunch.pid`
Where hobbitlaunch.pid is the PID of the hobbitlaunch process. When running this command we are left with many stray processes, not just a vmstat. For example:
/usr/local/bin/ssh -l root 10.6.6.6 sysstat -m -c1 290
sh -c (/usr/local/bin/ssh -l root 10.6.6.5 sysstat -x -c1 290 >/export/home/hob
Doing a bit more digging it seems these are related to an ext/netapp monitor perl script. Taking this out leaves less processes (although I don't want to leave it out forever so I'll have to do a bit of digging around to find out how to kill these gracefully) however because the iostats do not finish the SMF reports a failure rather than a disable when I try to stop the monitor. This means the SMF can't restart the bbserver if it ever crashes.
----- Original Message -----
From: "Buchan Milne" <user-9b139aff4dec@xymon.invalid>
To: <user-ae9b8668bcde@xymon.invalid>
Cc: <user-c15424b7e83a@xymon.invalid>
Sent: Monday, August 17, 2009 3:52 PM
Subject: Re: [hobbit] hobbit.sh not stopping server correctly
On Monday, 17 August 2009 11:04:36 user-c15424b7e83a@xymon.invalid wrote:Hi, I think I have an issue with 4.2.3 where the hobbit.sh is not stopping the server correctly. Running "./hobbit.sh stop" doesn't stop all the hobbit processesWhich processes remain? A vmstat running as the hobbit / xymon user?and so I can't install this as a service currently. Could anyone help me to debug why this is happening? I'm not really sure where to go from here.It's a bit difficult with no information. A stray vmstat process after stopping xymon is expected, and using the example init scripts provided in the debian or rpm directories works on many distros ... Regards, Buchan
list E-mail j.sansford
Hi there Jef, I assume that question was aimed at me? The stop script stops "hobbitlaunch" and not a client - however the client is owned by hobbitlaunch as it is configured within hobbitlaunch.cfg - we don't run a separately owned client on the system.
▸
---- user-a78daed26301@xymon.invalid wrote: Hi Buchan, Just a stuppid question. But have you stopped and the hobbit server and the hobbit client on that machine or only the server? Regards, Jef Jagers Systems Engineer Thomson CompuMark Thomson Reuters T +32 3 220 76 02 -----Original Message----- From: user-c15424b7e83a@xymon.invalid [mailto:user-c15424b7e83a@xymon.invalid] Sent: dinsdag, augustus 18, 2009 13:25 To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] hobbit.sh not stopping server correctly Hi Buchan, Apologies, I'll explain it in more detail. We are running on Solaris 10 and trying to configure Xymon as an SMF service (which essentially uses stop/start scripts). The stop script is basically: kill -TERM `cat /export/home/hobbit/logs/hobbitlaunch.pid` Where hobbitlaunch.pid is the PID of the hobbitlaunch process. When running this command we are left with many stray processes, not just a vmstat. For example: /usr/local/bin/ssh -l root 10.6.6.6 sysstat -m -c1 290 sh -c (/usr/local/bin/ssh -l root 10.6.6.5 sysstat -x -c1 290 >/export/home/hob Doing a bit more digging it seems these are related to an ext/netapp monitor perl script. Taking this out leaves less processes (although I don't want to leave it out forever so I'll have to do a bit of digging around to find out how to kill these gracefully) however because the iostats do not finish the SMF reports a failure rather than a disable when I try to stop the monitor. This means the SMF can't restart the bbserver if it ever crashes. ----- Original Message ----- From: "Buchan Milne" <user-9b139aff4dec@xymon.invalid> To: <user-ae9b8668bcde@xymon.invalid> Cc: <user-c15424b7e83a@xymon.invalid> Sent: Monday, August 17, 2009 3:52 PM Subject: Re: [hobbit] hobbit.sh not stopping server correctlyOn Monday, 17 August 2009 11:04:36 user-c15424b7e83a@xymon.invalid wrote:Hi, I think I have an issue with 4.2.3 where the hobbit.sh is not stopping the server correctly. Running "./hobbit.sh stop" doesn't stop all the hobbit processesWhich processes remain? A vmstat running as the hobbit / xymon user?and so I can't install this as a service currently. Could anyone help me to debug why this is happening? I'm not really sure where to go from here.It's a bit difficult with no information. A stray vmstat process after stopping xymon is expected, and using the example init scripts provided in the debian or rpm directories works on many distros ... Regards, Buchan
list Greg Hubbard
How long are you waiting after you kill the main Hobbit process for all the other processes to die off? Sometimes it can take several minutes for all the iostats and vmstats and such to complete -- they must be independent of their parent. Add-ons, especially from deadcat (written for BB, not Hobbit/Xymon) must be considered separately... GLH
▸
On 8/18/09, user-c15424b7e83a@xymon.invalid <user-c15424b7e83a@xymon.invalid> wrote:Hi there Jef, I assume that question was aimed at me? The stop script stops "hobbitlaunch" and not a client - however the client is owned by hobbitlaunch as it is configured within hobbitlaunch.cfg - we don't run a separately owned client on the system. ---- user-a78daed26301@xymon.invalid wrote:Hi Buchan, Just a stuppid question. But have you stopped and the hobbit server and the hobbit client on that machine or only the server? Regards, Jef Jagers Systems Engineer Thomson CompuMark Thomson Reuters T +32 3 220 76 02 -----Original Message----- From: user-c15424b7e83a@xymon.invalid [mailto:user-c15424b7e83a@xymon.invalid] Sent: dinsdag, augustus 18, 2009 13:25 To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] hobbit.sh not stopping server correctly Hi Buchan, Apologies, I'll explain it in more detail. We are running on Solaris 10 and trying to configure Xymon as an SMF service (which essentially uses stop/start scripts). The stop script is basically: kill -TERM `cat /export/home/hobbit/logs/hobbitlaunch.pid` Where hobbitlaunch.pid is the PID of the hobbitlaunch process. When running this command we are left with many stray processes, not just a vmstat. For example: /usr/local/bin/ssh -l root 10.6.6.6 sysstat -m -c1 290 sh -c (/usr/local/bin/ssh -l root 10.6.6.5 sysstat -x -c1 290 /export/home/hob Doing a bit more digging it seems these are related to an ext/netapp monitor perl script. Taking this out leaves less processes (although I don't want to leave it out forever so I'll have to do a bit of digging around to find out how to kill these gracefully) however because the iostats do not finish the SMF reports a failure rather than a disable when I try to stop the monitor. This means the SMF can't restart the bbserver if it ever crashes. ----- Original Message ----- From: "Buchan Milne" <user-9b139aff4dec@xymon.invalid> To: <user-ae9b8668bcde@xymon.invalid> Cc: <user-c15424b7e83a@xymon.invalid> Sent: Monday, August 17, 2009 3:52 PM Subject: Re: [hobbit] hobbit.sh not stopping server correctlyOn Monday, 17 August 2009 11:04:36 user-c15424b7e83a@xymon.invalid wrote:Hi, I think I have an issue with 4.2.3 where the hobbit.sh is not stopping the server correctly. Running "./hobbit.sh stop" doesn't stop all the hobbit processesWhich processes remain? A vmstat running as the hobbit / xymon user?and so I can't install this as a service currently. Could anyone help me to debug why this is happening? I'm not really sure where to go from here.It's a bit difficult with no information. A stray vmstat process after stopping xymon is expected, and using the example init scripts provided in the debian or rpm directories works on many distros ... Regards, Buchan
list E-mail j.sansford
Hi Greg, I've tried waiting over 5 minutes. I'll try waiting a little longer...:)
▸
---- Greg Hubbard <user-435e16ecfd6a@xymon.invalid> wrote: How long are you waiting after you kill the main Hobbit process for all the other processes to die off? Sometimes it can take several minutes for all the iostats and vmstats and such to complete -- they must be independent of their parent. Add-ons, especially from deadcat (written for BB, not Hobbit/Xymon) must be considered separately... GLH On 8/18/09, user-c15424b7e83a@xymon.invalid <user-c15424b7e83a@xymon.invalid> wrote:Hi there Jef, I assume that question was aimed at me? The stop script stops "hobbitlaunch" and not a client - however the client is owned by hobbitlaunch as it is configured within hobbitlaunch.cfg - we don't run a separately owned client on the system. ---- user-a78daed26301@xymon.invalid wrote:Hi Buchan, Just a stuppid question. But have you stopped and the hobbit server and the hobbit client on that machine or only the server? Regards, Jef Jagers Systems Engineer Thomson CompuMark Thomson Reuters T +32 3 220 76 02 -----Original Message----- From: user-c15424b7e83a@xymon.invalid [mailto:user-c15424b7e83a@xymon.invalid] Sent: dinsdag, augustus 18, 2009 13:25 To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] hobbit.sh not stopping server correctly Hi Buchan, Apologies, I'll explain it in more detail. We are running on Solaris 10 and trying to configure Xymon as an SMF service (which essentially uses stop/start scripts). The stop script is basically: kill -TERM `cat /export/home/hobbit/logs/hobbitlaunch.pid` Where hobbitlaunch.pid is the PID of the hobbitlaunch process. When running this command we are left with many stray processes, not just a vmstat. For example: /usr/local/bin/ssh -l root 10.6.6.6 sysstat -m -c1 290 sh -c (/usr/local/bin/ssh -l root 10.6.6.5 sysstat -x -c1 290 /export/home/hob Doing a bit more digging it seems these are related to an ext/netapp monitor perl script. Taking this out leaves less processes (although I don't want to leave it out forever so I'll have to do a bit of digging around to find out how to kill these gracefully) however because the iostats do not finish the SMF reports a failure rather than a disable when I try to stop the monitor. This means the SMF can't restart the bbserver if it ever crashes. ----- Original Message ----- From: "Buchan Milne" <user-9b139aff4dec@xymon.invalid> To: <user-ae9b8668bcde@xymon.invalid> Cc: <user-c15424b7e83a@xymon.invalid> Sent: Monday, August 17, 2009 3:52 PM Subject: Re: [hobbit] hobbit.sh not stopping server correctlyOn Monday, 17 August 2009 11:04:36 user-c15424b7e83a@xymon.invalid wrote:Hi, I think I have an issue with 4.2.3 where the hobbit.sh is not stopping the server correctly. Running "./hobbit.sh stop" doesn't stop all the hobbit processesWhich processes remain? A vmstat running as the hobbit / xymon user?and so I can't install this as a service currently. Could anyone help me to debug why this is happening? I'm not really sure where to go from here.It's a bit difficult with no information. A stray vmstat process after stopping xymon is expected, and using the example init scripts provided in the debian or rpm directories works on many distros ... Regards, Buchan
list E-mail j.sansford
Well whadaya know...set the service timeout to around 7-8 minutes and it works. It may be slow...but at least it now has automatic resiliance. Many thanks all. James.
▸
---- user-c15424b7e83a@xymon.invalid wrote: Hi Greg, I've tried waiting over 5 minutes. I'll try waiting a little longer...:) ---- Greg Hubbard <user-435e16ecfd6a@xymon.invalid> wrote:How long are you waiting after you kill the main Hobbit process for all the other processes to die off? Sometimes it can take several minutes for all the iostats and vmstats and such to complete -- they must be independent of their parent. Add-ons, especially from deadcat (written for BB, not Hobbit/Xymon) must be considered separately... GLH On 8/18/09, user-c15424b7e83a@xymon.invalid <user-c15424b7e83a@xymon.invalid> wrote:Hi there Jef, I assume that question was aimed at me? The stop script stops "hobbitlaunch" and not a client - however the client is owned by hobbitlaunch as it is configured within hobbitlaunch.cfg - we don't run a separately owned client on the system. ---- user-a78daed26301@xymon.invalid wrote:Hi Buchan, Just a stuppid question. But have you stopped and the hobbit server and the hobbit client on that machine or only the server? Regards, Jef Jagers Systems Engineer Thomson CompuMark Thomson Reuters T +32 3 220 76 02 -----Original Message----- From: user-c15424b7e83a@xymon.invalid [mailto:user-c15424b7e83a@xymon.invalid] Sent: dinsdag, augustus 18, 2009 13:25 To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] hobbit.sh not stopping server correctly Hi Buchan, Apologies, I'll explain it in more detail. We are running on Solaris 10 and trying to configure Xymon as an SMF service (which essentially uses stop/start scripts). The stop script is basically: kill -TERM `cat /export/home/hobbit/logs/hobbitlaunch.pid` Where hobbitlaunch.pid is the PID of the hobbitlaunch process. When running this command we are left with many stray processes, not just a vmstat. For example: /usr/local/bin/ssh -l root 10.6.6.6 sysstat -m -c1 290 sh -c (/usr/local/bin/ssh -l root 10.6.6.5 sysstat -x -c1 290 /export/home/hob Doing a bit more digging it seems these are related to an ext/netapp monitor perl script. Taking this out leaves less processes (although I don't want to leave it out forever so I'll have to do a bit of digging around to find out how to kill these gracefully) however because the iostats do not finish the SMF reports a failure rather than a disable when I try to stop the monitor. This means the SMF can't restart the bbserver if it ever crashes. ----- Original Message ----- From: "Buchan Milne" <user-9b139aff4dec@xymon.invalid> To: <user-ae9b8668bcde@xymon.invalid> Cc: <user-c15424b7e83a@xymon.invalid> Sent: Monday, August 17, 2009 3:52 PM Subject: Re: [hobbit] hobbit.sh not stopping server correctlyOn Monday, 17 August 2009 11:04:36 user-c15424b7e83a@xymon.invalid wrote:Hi, I think I have an issue with 4.2.3 where the hobbit.sh is not stopping the server correctly. Running "./hobbit.sh stop" doesn't stop all the hobbit processesWhich processes remain? A vmstat running as the hobbit / xymon user?and so I can't install this as a service currently. Could anyone help me to debug why this is happening? I'm not really sure where to go from here.It's a bit difficult with no information. A stray vmstat process after stopping xymon is expected, and using the example init scripts provided in the debian or rpm directories works on many distros ... Regards, Buchan
list Buchan Milne
▸
On Tuesday, 18 August 2009 12:25:26 user-c15424b7e83a@xymon.invalid wrote:
Hi Buchan, Apologies, I'll explain it in more detail. We are running on Solaris 10 and trying to configure Xymon as an SMF service (which essentially uses stop/start scripts). The stop script is basically: kill -TERM `cat /export/home/hobbit/logs/hobbitlaunch.pid` Where hobbitlaunch.pid is the PID of the hobbitlaunch process. When running this command we are left with many stray processes, not just a vmstat. For example: /usr/local/bin/ssh -l root 10.6.6.6 sysstat -m -c1 290 sh -c (/usr/local/bin/ssh -l root 10.6.6.5 sysstat -x -c1 290/export/home/hobDoing a bit more digging it seems these are related to an ext/netapp monitor perl script.
I haven't (yet) monitored any netapp devices with devmon (most likely I will soon) ... but there is a template, which has tests named 'cluster', 'if_load','netapptemp','quota','status' and 'volume'. You may want to consider trying it instead of this script. If the teplate is missing something which is available by SNMP, we can add support for it. BTW, are you running this out of cron, or from hobbitlaunch?
▸
Taking this out leaves less processes (although I don't want to leave it out forever so I'll have to do a bit of digging around to find out how to kill these gracefully) however because the iostats do not finish the SMF reports a failure rather than a disable when I try to stop the monitor.
Well, then it seems SMF is a bit brain-dead ... however, you could run something equivalent to: pkill -u xymon vmstat as well, which on my test box results in no processes running as the xymon user (after stopping the service via the init script on a Linux box).
▸
This means the SMF can't restart the bbserver if it ever crashes.
I don't think I've ever seen hobbitlaunch crash ... Regards, Buchan