I'm starting to look at setting up 'propr' monitoring for my production SAP environment, and wonder if anyone else is playing with HACMP clusters
list Tom Kauffman
I need to split the monitoring for SAP and Oracle from the monitoring of the host it runs on -- because, in an IBM HACMP environment, the workload can move. So the oracle tests, *some* of the disk space tests, and *some* of the proc tests move from one box to another. So does the primary IP address. I need to be able to track 'colorado' the box (the hostname does NOT move) as well as 'colorado' the IP address. 'Colorado' the address could actually be pointing to the 'thames' or 'yukon' boxes. At which time, 'colorado' the box will be responding to the 'colorado-bt' adapter address (if 'colorado' the box is up at all). I can get some of what I want by running two clients, one for 'colorado' the box, and one for 'SAP' or some other such synthetic name. The biggest two challenges I see are splitting the OS filesystems from the application filesystems for freespace reporting, and the shifting IP address. I may have an 'out' on the IP address in the future, if I can migrate to what IBM calls 'aliasing' -- where they add the running address as a second address on the adapter and respond to both. And to make this a bit more fun, I have THREE of these silly environments to work with. Henrik, is there any way to hang two different IP addresses on the same bb-host entry such that the network connectivity test will be happy if either one responds? They are always on the same IP network, just different host addresses (our convention is to add 100 to the last octet to get the boot-time address for the matching run-time address). Is anyone else doing something silly like this? TIA Tom Kauffman NIBCO, Inc CONFIDENTIALITY NOTICE: This email and any attachments are for the exclusive and confidential use of the intended recipient. If you are not the intended recipient, please do not read, distribute or take action in reliance upon this message. If you have received this in error, please notify us immediately by return email and promptly delete this message and its attachments from your computer system. We do not waive attorney-client or work product privilege by the transmission of this message.
list Thomas Pedersen
We do have the same setup in my company. Here I am running a cluster agent and 2 "node" agents. The cluster agent then checks only the cluster services and reports with the cluster name. Is also located on a clsuter filesystem so that it is handles by HACMP to stop/start with the ressource group. in the bb-hosts file if you add conn=worst,1.2.3.4,2.3.4.5 both ips will be pinged and alerts will go red if any of the ips are unavailable.
▸
Kauffman, Tom wrote:I need to split the monitoring for SAP and Oracle from the monitoring of the host it runs on -- because, in an IBM HACMP environment, the workload can move. So the oracle tests, *some* of the disk space tests, and *some* of the proc tests move from one box to another. So does the primary IP address. I need to be able to track 'colorado' the box (the hostname does NOT move) as well as 'colorado' the IP address. 'Colorado' the address could actually be pointing to the 'thames' or 'yukon' boxes. At which time, 'colorado' the box will be responding to the 'colorado-bt' adapter address (if 'colorado' the box is up at all). I can get some of what I want by running two clients, one for 'colorado' the box, and one for 'SAP' or some other such synthetic name. The biggest two challenges I see are splitting the OS filesystems from the application filesystems for freespace reporting, and the shifting IP address. I may have an 'out' on the IP address in the future, if I can migrate to what IBM calls 'aliasing' -- where they add the running address as a second address on the adapter and respond to both. And to make this a bit more fun, I have THREE of these silly environments to work with. Henrik, is there any way to hang two different IP addresses on the same bb-host entry such that the network connectivity test will be happy if either one responds? They are always on the same IP network, just different host addresses (our convention is to add 100 to the last octet to get the boot-time address for the matching run-time address). Is anyone else doing something silly like this? TIA Tom Kauffman NIBCO, Inc CONFIDENTIALITY NOTICE: This email and any attachments are for the exclusive and confidential use of the intended recipient. If you are not the intended recipient, please do not read, distribute or take action in reliance upon this message. If you have received this in error, please notify us immediately by return email and promptly delete this message and its attachments from your computer system. We do not waive attorney-client or work product privilege by the transmission of this message.
list Rolf Schrittenlocher
Hi Tom,
▸
I need to split the monitoring for SAP and Oracle from the monitoring of the host it runs on -- because, in an IBM HACMP environment, the workload can move. So the oracle tests, *some* of the disk space tests, and *some* of the proc tests move from one box to another. So does the primary IP address. I need to be able to track 'colorado' the box (the hostname does NOT move) as well as 'colorado' the IP address. 'Colorado' the address could actually be pointing to the 'thames' or 'yukon' boxes. At which time, 'colorado' the box will be responding to the 'colorado-bt' adapter address (if 'colorado' the box is up at all). I can get some of what I want by running two clients, one for 'colorado' the box, and one for 'SAP' or some other such synthetic name. The biggest two challenges I see are splitting the OS filesystems from the application filesystems for freespace reporting, and the shifting IP address. I may have an 'out' on the IP address in the future, if I can migrate to what IBM calls 'aliasing' -- where they add the running address as a second address on the adapter and respond to both. And to make this a bit more fun, I have THREE of these silly environments to work with. Henrik, is there any way to hang two different IP addresses on the same bb-host entry such that the network connectivity test will be happy if either one responds? They are always on the same IP network, just different host addresses (our convention is to add 100 to the last octet to get the boot-time address for the matching run-time address). Is anyone else doing something silly like this?
Not the same but similar. We have 1-3 virtual hosts on each machine. There is only one hobbit client installation but one instance running for each host on all machines. Custom scripts differ for these hosts. From serverside we treat them as different machines so e.g. disk is surveyed completely for all hosts. But you could easily disable the normal disk test and replace it by a custom script which greps for the interesting partitions. kind regards Rolf -- Mit freundlichen Gruessen Rolf Schrittenlocher HRZ/BDV, Senckenberganlage 31, 60054 Frankfurt Tel: (XX) XX - XXX XXXXX Fax: (XX) XX - XXX XXXXX LBS: user-1e39a1813094@xymon.invalid Persoenlich: user-6ea8e907e200@xymon.invalid
list Timothy Myers
If I understand what you are getting at, you will want to setup where the resource groups with the resource groups ID as a server. Then you set up a persistent IP - ( if your running HACMP 5.x this is standard) that you use for each LPAR. Then in you resource startup you enable reporting ( you will need to spoof you hostname on the reports) and in you resource shutdown scripts disable the reporting for this resource although you may want to send a report that you are going down to Hobbit this could be a custom report of status on your screen. The node up and down could spawn updates along with the hacmp.out log file or a subset. You could also make your reporting HACMP aware and they could run all the time but only report if they own the resource group. You also have advance event monitoring within HACMP that could spawn a report based on an event in real time.
▸
On 4/27/06, Kauffman, Tom <user-3feba9e60a8b@xymon.invalid> wrote:I need to split the monitoring for SAP and Oracle from the monitoring of the host it runs on -- because, in an IBM HACMP environment, the workload can move. So the oracle tests, *some* of the disk space tests, and *some* of the proc tests move from one box to another. So does the primary IP address. I need to be able to track 'colorado' the box (the hostname does NOT move) as well as 'colorado' the IP address. 'Colorado' the address could actually be pointing to the 'thames' or 'yukon' boxes. At which time, 'colorado' the box will be responding to the 'colorado-bt' adapter address (if 'colorado' the box is up at all). I can get some of what I want by running two clients, one for 'colorado' the box, and one for 'SAP' or some other such synthetic name. The biggest two challenges I see are splitting the OS filesystems from the application filesystems for freespace reporting, and the shifting IP address. I may have an 'out' on the IP address in the future, if I can migrate to what IBM calls 'aliasing' -- where they add the running address as a second address on the adapter and respond to both. And to make this a bit more fun, I have THREE of these silly environments to work with. Henrik, is there any way to hang two different IP addresses on the same bb-host entry such that the network connectivity test will be happy if either one responds? They are always on the same IP network, just different host addresses (our convention is to add 100 to the last octet to get the boot-time address for the matching run-time address). Is anyone else doing something silly like this? TIA Tom Kauffman NIBCO, Inc CONFIDENTIALITY NOTICE: This email and any attachments are for the exclusive and confidential use of the intended recipient. If you are not the intended recipient, please do not read, distribute or take action in reliance upon this message. If you have received this in error, please notify us immediately by return email and promptly delete this message and its attachments from your computer system. We do not waive attorney-client or work product privilege by the transmission of this message.
list David Gore
I am trying to get msgs and files to work and made the configuration changes below: client-local.cfg: [myhost] file:/opt/app/apache/logs/access.log log:/opt/app/apache/logs/error-test.log:10240 hobbit-clients.cfg: LOG /opt/app/apache/logs/error-test.log MATCH="%Segmentation Fault" COLOR=red FILE /opt/app/apache/logs/access.log size>2000 COLOR=red Then I realized there are two possible reasons this may be failing. The first being that the above configuration is INCORRECT. The second being perhaps I need updated client binaries and configuration files because my old version of the client is NOT SUPPORTED for these new columns? Anybody out there have it working? Did you use the current client binaries and config files? Server version: *Hobbit Monitor 4.2-alfa-20060416 <http://hobbitmon.sourceforge.net/>* Server OS: Fedora Core 5 Client version: built August 2005 Client OS: Solaris 8 • <http://hobbitmon.sourceforge.net/>* ~David
list David Gore
I am trying to get msgs and files to work and made the configuration changes below: client-local.cfg: [myhost] file:/opt/app/apache/logs/access.log log:/opt/app/apache/logs/error-test.log:10240 hobbit-clients.cfg: LOG /opt/app/apache/logs/error-test.log MATCH="%Segmentation Fault" COLOR=red FILE /opt/app/apache/logs/access.log size>2000 COLOR=red Then I realized there are two possible reasons this may be failing. The first being that the above configuration is INCORRECT. The second being perhaps I need updated client binaries and configuration files because my old version of the client is NOT SUPPORTED for these new columns? Anybody out there have it working? Did you use the current client binaries and config files? Server version: *Hobbit Monitor 4.2-alfa-20060416 <http://hobbitmon.sourceforge.net/>* Server OS: Fedora Core 5 Client version: built August 2005 Client OS: Solaris 8 • <http://hobbitmon.sourceforge.net/>* ~David
list Tom Kauffman
That's pretty much it - except that this started as an HACMP 2.1 cluster and keeps getting upgraded as it moves to new hardware - so we're still doing IPAT and don't have either a persistent IP address or IP aliasing to keep the boot address around. I get more time to look at this next week - right now I've got a ton and a half (7.6 TB) of disk data to schlep around; I also wear the storage admin (and, for that matter, the TSM admin) hat here and we've a model 800 ESS being replaced by a DS-8100 over the next 5 days. I just wanted to kick the question off and start some ideas floating around internally. Thanks! Tom From: timothy Myers [mailto:user-62852960a8b8@xymon.invalid] Sent: Thursday, April 27, 2006 1:39 PM To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] I'm starting to look at setting up 'propr' monitoring for my production SAP environment, and wonder if anyone else is playing with HACMP clusters
▸
If I understand what you are getting at, you will want to setup where
the resource groups with the resource groups ID as a server. Then you
set up a persistent IP - ( if your running HACMP 5.x this is standard)
that you use for each LPAR. Then in you resource startup you enable
reporting ( you will need to spoof you hostname on the reports) and in
you resource shutdown scripts disable the reporting for this resource
although you may want to send a report that you are going down to Hobbit
this could be a custom report of status on your screen. The node up and
down could spawn updates along with the hacmp.out log file or a subset.
You could also make your reporting HACMP aware and they could run all
the time but only report if they own the resource group.
You also have advance event monitoring within HACMP that could spawn a
report based on an event in real time.
On 4/27/06, Kauffman, Tom <user-3feba9e60a8b@xymon.invalid> wrote:
I need to split the monitoring for SAP and Oracle from the monitoring of
the host it runs on -- because, in an IBM HACMP environment, the
workload can move. So the oracle tests, *some* of the disk space tests,
and *some* of the proc tests move from one box to another.
So does the primary IP address.
I need to be able to track 'colorado' the box (the hostname does NOT
move) as well as 'colorado' the IP address. 'Colorado' the address could
actually be pointing to the 'thames' or 'yukon' boxes. At which time,
'colorado' the box will be responding to the 'colorado-bt' adapter
address (if 'colorado' the box is up at all).
I can get some of what I want by running two clients, one for 'colorado'
the box, and one for 'SAP' or some other such synthetic name. The
biggest two challenges I see are splitting the OS filesystems from the
application filesystems for freespace reporting, and the shifting IP
address. I may have an 'out' on the IP address in the future, if I can
migrate to what IBM calls 'aliasing' -- where they add the running
address as a second address on the adapter and respond to both.
And to make this a bit more fun, I have THREE of these silly
environments to work with.
Henrik, is there any way to hang two different IP addresses on the same
bb-host entry such that the network connectivity test will be happy if
either one responds? They are always on the same IP network, just
different host addresses (our convention is to add 100 to the last octet
to get the boot-time address for the matching run-time address).
Is anyone else doing something silly like this?
TIA
Tom Kauffman
NIBCO, Inc
CONFIDENTIALITY NOTICE: This email and any attachments are for the
exclusive and confidential use of the intended recipient. If you are
not
the intended recipient, please do not read, distribute or take action in
reliance upon this message. If you have received this in error, please
notify us immediately by return email and promptly delete this message
and its attachments from your computer system. We do not waive
attorney-client or work product privilege by the transmission of this
message.
CONFIDENTIALITY NOTICE: This email and any attachments are for the
exclusive and confidential use of the intended recipient. If you are not
the intended recipient, please do not read, distribute or take action in
reliance upon this message. If you have received this in error, please
notify us immediately by return email and promptly delete this message
and its attachments from your computer system. We do not waive
attorney-client or work product privilege by the transmission of this
message.
list Henrik Størner
▸
On Thu, Apr 27, 2006 at 08:12:34PM +0000, David Gore wrote:
Then I realized there are two possible reasons this may be failing. The first being that the above configuration is INCORRECT. The second being perhaps I need updated client binaries and configuration files because my old version of the client is NOT SUPPORTED for these new columns?
You definitely need to update the client. I'm really sorry about that, but back in August last year I had no idea how to do the log-file stuff, so the old client simply doesn't provide the data needed for the log- and file-monitoring. Regards, Henrik
list David Gore
▸
Henrik Stoerner wrote:
On Thu, Apr 27, 2006 at 08:12:34PM +0000, David Gore wrote:Then I realized there are two possible reasons this may be failing. The first being that the above configuration is INCORRECT. The second being perhaps I need updated client binaries and configuration files because my old version of the client is NOT SUPPORTED for these new columns?You definitely need to update the client. I'm really sorry about that, but back in August last year I had no idea how to do the log-file stuff, so the old client simply doesn't provide the data needed for the log- and file-monitoring. Regards, Henrik
Ok, is there some minimum number of files I can update like only the binaries bb and hobbitlaunch? ~David
list Henrik Størner
▸
On Thu, Apr 27, 2006 at 10:36:37PM +0000, David Gore wrote:
You definitely need to update the client. I'm really sorry about that, but back in August last year I had no idea how to do the log-file stuff, so the old client simply doesn't provide the data needed for the log- and file-monitoring.Ok, is there some minimum number of files I can update like only the binaries bb and hobbitlaunch?
~hobbit/client/bin/* and ~hobbit/client/runclient.sh - i.e. all of the programs. Henrik
list David Gore
David Gore (v965-3670) Enhanced Technology Support (ETS) Network Management Systems (NMS) IMPACT Transport Team Lead - SCSA, SCNA Page: 1-800-PAG-eMCI pin 1406090 Vnet: 965-3676 *e-mail via SUSE Linux 9.3 and other open source tools.
▸
Henrik Stoerner wrote:On Thu, Apr 27, 2006 at 10:36:37PM +0000, David Gore wrote:You definitely need to update the client. I'm really sorry about that, but back in August last year I had no idea how to do the log-file stuff, so the old client simply doesn't provide the data needed for the log- and file-monitoring.Ok, is there some minimum number of files I can update like only the binaries bb and hobbitlaunch?~hobbit/client/bin/* and ~hobbit/client/runclient.sh - i.e. all of the programs. Henrik
Well, it is going to be very painful to update hundreds of hosts. Regardless, I have created a new instance and just modified the new instance with our previous changes. I still cannot get LOGS to work FILES. I can get to work, but perhaps I found a bug? Here is the entry: client-local.cfg:file:/opt/app/apache/logs/access-test.log hobbit-clients.cfg: FILE /opt/app/apache/logs/access-test.log size<2000000 COLOR=red Here is the result when alarming: /opt/app/apache/logs/access-test.log File is missing And when you click the link: ERROR: Value too large for defined data type I am trying to test for a file size greater than 2G. It would be very nice if the size could be represented in kilobytes, megabytes and gigabytes. K, M, and G. Here is a LOG entry that fails (none work): client-local.cfg:log:/opt/app/apache/logs/error-test.log:10240 hobbit-clients.cfg: LOG /opt/app/apache/logs/error-test.log MATCH="%Segmentation Fault" COLOR=red The documentation appears out of date from your e-mail on the April 24th 21:09 GMT?
list Henrik Størner
▸
On Fri, Apr 28, 2006 at 06:26:15PM +0000, David Gore wrote:
Henrik Stoerner wrote:You definitely need to update the client. I'm really sorry about that, but back in August last year I had no idea how to do the log-file stuff, so the old client simply doesn't provide the data needed for the log- and file-monitoring.Well, it is going to be very painful to update hundreds of hosts.
Absolutely ... I haven't deployed the Hobbit client extensively yet, but I still have some 30-40 hosts I need to update. The good news is that the client now has the ability to update itself from a central filestore on the Hobbit server, so hopefully it should be a lot easier the next time. (Note - this is NOT included in the current alfa-release). I'll respond to your other questions in another e-mail. Henrik
list Henrik Størner
▸
On Fri, Apr 28, 2006 at 06:26:15PM +0000, David Gore wrote:
instance with our previous changes. I still cannot get LOGS to work FILES. I can get to work, but perhaps I found a bug? Here is the entry: client-local.cfg:file:/opt/app/apache/logs/access-test.log hobbit-clients.cfg: FILE /opt/app/apache/logs/access-test.log size<2000000 COLOR=red Here is the result when alarming: /opt/app/apache/logs/access-test.log File is missing And when you click the link: ERROR: Value too large for defined data type I am trying to test for a file size greater than 2G.
Large file support is missing in the client utility. I took this as an opportunity to vet the code for places where this might be an issue, so the next trial-version (or tomorrows snapshot) should work better.
It would be very nice if the size could be represented in kilobytes, megabytes and gigabytes. K, M, and G.
Yep ... done.
▸
Here is a LOG entry that fails (none work): client-local.cfg:log:/opt/app/apache/logs/error-test.log:10240 hobbit-clients.cfg: LOG /opt/app/apache/logs/error-test.log MATCH="%Segmentation Fault" COLOR=red
Please try again with the current snapshot. I don't see why it should not work, except if the client-side tool (logfetch) failed to process the logfile due to it being larger than 2 GB.
The documentation appears out of date from your e-mail on the April 24th 21:09 GMT?
Docs should be up-to-date in the current snapshot. Henrik