Hobbit not recognizing certain tests?
list Eric Jacobs
Using Andy Farrior's method, I set up hobbit (version 4.2, allinone patch installed) to receive SNMP traps from some of our F5 devices. Works fine except for one of the devices. Checked all the logs and made sure that the various corollary programs are getting the traps correctly. Finally figured out that Hobbit (or, more specifically, hobbitboard. The trap.pl uses hobbitboard to find all the hosts that have the "trap" test defined. I ran the hobbitboard command myself and saw that it did not include the particular device in question) was not recognizing the trap test for that device. I've tried re-writing the bb-hosts file entry and moving things around (I'm also running Devmon against that device which sort of works, though the templates probably need some work) and moved the "trap" designation behind the devmon designation. No luck. For what it is worth, on the info page for the device in question, "trap" does not show up in the status summary, though it does show up in the "Other tags" section. For the devices for which the trap works, "trap" shows up in both the status summary and the "Other tags" section. Eric Jacobs
list Henrik Størner
▸
On Mon, Jul 16, 2007 at 12:43:55PM -0400, Eric Jacobs wrote:
The trap.pl uses hobbitboard to find all the hosts that have the "trap" test defined.
Bad idea - there's a chicken-and-egg problem here. Columns do not appear in the hobbitdboard output until they exist, i.e. a status has been reported to Hobbit. So if *something* must report a "trap" status to Hobbit before the trap.pl script sees that it should do the "trap" thing ... well, I guess you can tell where I'm going. Scripts that implement custom tests should use the "bbhostgrep" utility to scan the bb-hosts file for the hosts that have a specific tag defined. There is actually another way it can be done, but it's not yet documented: You can send a "hostinfo" command to hobbitd, and it will return a pre-parsed version of the bb-hosts file - one line per hosts, with the fields delimited by '|' characters, eg: mail.hswn.dk|172.16.10.2|NET:intern|COMMENT:Internal mail server|smtp|pop3|imap Regards, Henrik
list Eric ThomasTech Jacobs
Sorry, my mistake. The part of trap.pl that uses hobbitboard is checking on the status of existing traps. The subroutine that sends the traps doesn't. Back to the drawing board. Eric
▸
-----Original Message----- From: user-ce4a2c883f75@xymon.invalid [mailto:user-ce4a2c883f75@xymon.invalid] Sent: Monday, July 16, 2007 4:41 PM To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] Hobbit not recognizing certain tests? On Mon, Jul 16, 2007 at 12:43:55PM -0400, Eric Jacobs wrote:The trap.pl uses hobbitboard to find all the hosts that have the > "trap" test defined.Bad idea - there's a chicken-and-egg problem here. Columns do not appear in the hobbitdboard output until they exist, i.e. a status has been reported to Hobbit. So if *something* must report a "trap" status to Hobbit before the trap.pl script sees that it should do the "trap" thing ... well, I guess you can tell where I'm going. Scripts that implement custom tests should use the "bbhostgrep" utility to scan the bb-hosts file for the hosts that have a specific tag defined. There is actually another way it can be done, but it's not yet documented: You can send a "hostinfo" command to hobbitd, and it will return a pre-parsed version of the bb-hosts file - one line per hosts, with the fields delimited by '|' characters, eg: mail.hswn.dk|172.16.10.2|NET:intern|COMMENT:Internal mail server|smtp|pop3|imap Regards, Henrik
list Andy Farrior
I haven't modified trap.pl in a very *long* time. Hmm... (I'm still using it.) Going back over the code, I'm running hobbitdboard so I can get the validtime field for the test: "$BB $BBDISP \"hobbitdboard test=trap fields=hostname,validtime,color\" " In case there's a host that doesn't send a trap that often, I wanted to change the color to green instead of having it go purple and generate an alert. So to get the time left for a test, I needed the hobbitdboard results. A side-effect is that the column won't appear until the device sends it's first trap. For the problem device, is the log from SEC showing a hostname or IP address for the device sending the trap? There may be a problem if it's sending a status message to Hobbit with an IP address instead of a hostname. I don't think trap.pl does a good job (or at all) of trying to resolve an IP address to a hostname that's in bb-hosts. Hope that helps, Andy
▸
-----Original Message-----
From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: Monday, July 16, 2007 3:41 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Hobbit not recognizing certain tests?
On Mon, Jul 16, 2007 at 12:43:55PM -0400, Eric Jacobs wrote:The trap.pl uses hobbitboard to find all the hosts that have the "trap" test defined.
Bad idea - there's a chicken-and-egg problem here. Columns do not appear in the hobbitdboard output until they exist, i.e. a status has been reported to Hobbit. So if *something* must report a "trap" status to Hobbit before the trap.pl script sees that it should do the "trap" thing ... well, I guess you can tell where I'm going. Scripts that implement custom tests should use the "bbhostgrep" utility to scan the bb-hosts file for the hosts that have a specific tag defined. There is actually another way it can be done, but it's not yet documented: You can send a "hostinfo" command to hobbitd, and it will return a pre-parsed version of the bb-hosts file - one line per hosts, with the fields delimited by '|' characters, eg: mail.hswn.dk|172.16.10.2|NET:intern|COMMENT:Internal mail server|smtp|pop3|imap Regards, Henrik
list Eric ThomasTech Jacobs
-----Original Message----- From: FARRIOR, Andy [mailto:user-ca324d8ab782@xymon.invalid] Sent: Monday, July 16, 2007 5:30 PM To: user-ae9b8668bcde@xymon.invalid Subject: RE: [hobbit] Hobbit not recognizing certain tests?
Andy, wonderful to hear from you. And thanks for your method of getting snmp traps into Hobbit. It is a bit of a feather in my cap that I can do so. My boss is anxious to get rid of our highly priced NetCool monitoring package, but we needed to be able to monitor traps, at least from our F5 devices.
▸
I haven't modified trap.pl in a very *long* time. Hmm... (I'm still using it.) Going back over the code, I'm running hobbitdboard so I can get the validtime field for the test: "$BB $BBDISP \"hobbitdboard test=trap fields=hostname,validtime,color\" " In case there's a host that doesn't send a trap that often, I wanted to change the color to green instead of having it go purple and generate an alert. So to get the time left for a test, I needed the hobbitdboard results. A side-effect is that the column won't appear until the device sends it's first trap.
Yes, I looked again at the code and realized that I was mistaken, and have already apologized to Henrik for having the temerity to suggest that there might be a bug in his code.
▸
For the problem device, is the log from SEC showing a hostname or IP address for the device sending the trap?
Yes, but that was sometime ago. Maybe before I made some other corrections to get things working. I'm tempted to just down a server to cause a trap.
▸
There may be a problem if it's sending a status message to Hobbit with an IP address instead of a hostname. I don't think trap.pl does a good job (or at all) of trying to resolve an IP address to a hostname that's in bb-hosts.
Yes, I've seen that. In order to get things to work, I've had to make /etc/host file entries and make sure that the sending devices are configured to send their traps via certain interfaces so that the IP address will correspond to the host file address.
▸
Hope that helps, Andy -----Original Message----- From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid] Sent: Monday, July 16, 2007 3:41 PM To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] Hobbit not recognizing certain tests? On Mon, Jul 16, 2007 at 12:43:55PM -0400, Eric Jacobs wrote:The trap.pl uses hobbitboard to find all the hosts that have the > "trap" test defined.Bad idea - there's a chicken-and-egg problem here. Columns do not appear in the hobbitdboard output until they exist, i.e. a status has been reported to Hobbit. So if *something* must report a "trap" status to Hobbit before the trap.pl script sees that it should do the "trap" thing ... well, I guess you can tell where I'm going. Scripts that implement custom tests should use the "bbhostgrep" utility to scan the bb-hosts file for the hosts that have a specific tag defined. There is actually another way it can be done, but it's not yet documented: You can send a "hostinfo" command to hobbitd, and it will return a pre-parsed version of the bb-hosts file - one line per hosts, with the fields delimited by '|' characters, eg: mail.hswn.dk|172.16.10.2|NET:intern|COMMENT:Internal mail server|smtp|pop3|imap Regards, Henrik
list Greg L Hubbard
Being familiar with both Netcool and Hobbit, I would never consider replacing Netcool trap management with Hobbit. Netcool processes traps in "near real time" and scales quite large. Hobbit does not provide real time monitoring (default is 5 minute samples, with 1 minute screen updates). The only scenario where Hobbit might successfully replace a Netcool system is if you only want to know that *something bad* happened in the last five minutes, and you don't mind drilling down into a Hobbit screen to find out what *something bad* is. You will have to be careful about how you construct your trap test so it will handle alarm clearing and won't forget problems. I am not throwing any rocks at Hobbit -- it was not designed to handle real-time alarm management. I *think* Henrik would agree... With all that said, it is pretty easy to forward Hobbit alarms to Netcool, and I don't think it would be very hard to forward Netcool alarms to a Hobbit test -- if you are concerned about a dashboard, etc. GLH -----Original Message----- From: Jacobs, Eric (ThomasTech) [mailto:user-a7d0190671dd@xymon.invalid] Sent: Monday, July 16, 2007 5:05 PM
▸
To: 'user-ae9b8668bcde@xymon.invalid'
Subject: RE: [hobbit] Hobbit not recognizing certain tests?
-----Original Message----- From: FARRIOR, Andy [mailto:user-ca324d8ab782@xymon.invalid] Sent: Monday, July 16, 2007 5:30 PM To: user-ae9b8668bcde@xymon.invalid Subject: RE: [hobbit] Hobbit not recognizing certain tests?
Andy, wonderful to hear from you. And thanks for your method of getting snmp traps into Hobbit. It is a bit of a feather in my cap that I can do so. My boss is anxious to get rid of our highly priced NetCool monitoring package, but we needed to be able to monitor traps, at least from our F5 devices.
I haven't modified trap.pl in a very *long* time. Hmm... (I'm still using it.) Going back over the code, I'm running hobbitdboard so I can get the validtime field for the test: "$BB $BBDISP \"hobbitdboard test=trap fields=hostname,validtime,color\" " In case there's a host that doesn't send a trap that often, I wanted to change the color to green instead of having it go purple and generate an alert. So to get the time left for a test, I needed the hobbitdboard results. A side-effect is that the column won't appear until the device sends it's first trap.
Yes, I looked again at the code and realized that I was mistaken, and have already apologized to Henrik for having the temerity to suggest that there might be a bug in his code.
For the problem device, is the log from SEC showing a hostname or IP address for the device sending the trap?
Yes, but that was sometime ago. Maybe before I made some other corrections to get things working. I'm tempted to just down a server to cause a trap.
There may be a problem if it's sending a status message to Hobbit with
an IP address instead of a hostname. I don't think trap.pl does a good job (or at all) of trying to resolve an IP address to a hostname that's in bb-hosts.
Yes, I've seen that. In order to get things to work, I've had to make /etc/host file entries and make sure that the sending devices are configured to send their traps via certain interfaces so that the IP address will correspond to the host file address.
Hope that helps, Andy -----Original Message----- From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid] Sent: Monday, July 16, 2007 3:41 PM To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] Hobbit not recognizing certain tests? On Mon, Jul 16, 2007 at 12:43:55PM -0400, Eric Jacobs wrote:The trap.pl uses hobbitboard to find all the hosts that have the > "trap" test defined.Bad idea - there's a chicken-and-egg problem here. Columns do not appear in the hobbitdboard output until they exist, i.e. a status has been reported to Hobbit. So if *something* must report a "trap" status to Hobbit before the trap.pl script sees that it should do the "trap" thing ... well, I guess you can tell where I'm
going. Scripts that implement custom tests should use the "bbhostgrep" utility to scan the bb-hosts file for the hosts that have a specific tag defined. There is actually another way it can be done, but it's not yet documented: You can send a "hostinfo" command to hobbitd, and it will return a pre-parsed version of the bb-hosts file - one line per hosts,
with the fields delimited by '|' characters, eg: mail.hswn.dk|172.16.10.2|NET:intern|COMMENT:Internal mail server|smtp|pop3|imap Regards, Henrik
list Henrik Størner
▸
On Tue, Jul 17, 2007 at 08:07:31AM -0500, Hubbard, Greg L wrote:
Being familiar with both Netcool and Hobbit, I would never consider replacing Netcool trap management with Hobbit. Netcool processes traps in "near real time" and scales quite large. Hobbit does not provide real time monitoring (default is 5 minute samples, with 1 minute screen updates).
There is nothing inherent in Hobbit that prevents it from doing real-time handling of events. Hobbit processes events as soon as it is told about them; the fact that some types of information is only checked once every 5 minutes is not something that necessarily applies to everything Hobbit monitors. I havent looked at Andy's trap script, but if I were to implement SNMP trap handling in Hobbit, I'd start off with snmptrapd from the Net-SNMP tools - this receives snmp traps, and can be configured to do "something" when a trap arrives. That "something" would then be a script/utility that grabs the hostname and trap type from the trap information, and feeds that into Hobbit as a status update. That will give you an immediate alert, and a status change in the Hobbit display if you use the "Critical systems" view which is dynamically generated. I'm not throwing rocks at Netcool <grin> but I just want to make it clear that Hobbit can be as real-time as you want it to - it's only a matter of feeding it data as quickly as you possible. Regards, Henrik
list Greg L Hubbard
We are on the same page then. I don't think I would ask Hobbit to do this for very many devices. I handle 100 traps per second in Netcool, and I am just getting started. But I love what Hobbit provides for monitoring the tool servers themselves -- without relying on anything on those same servers in order to function. GLH
▸
-----Original Message-----
From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: Tuesday, July 17, 2007 8:33 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Hobbit not recognizing certain tests?
On Tue, Jul 17, 2007 at 08:07:31AM -0500, Hubbard, Greg L wrote:Being familiar with both Netcool and Hobbit, I would never consider replacing Netcool trap management with Hobbit. Netcool processes traps in "near real time" and scales quite large. Hobbit does not provide real time monitoring (default is 5 minute samples, with 1 minute screen updates).
There is nothing inherent in Hobbit that prevents it from doing real-time handling of events. Hobbit processes events as soon as it is told about them; the fact that some types of information is only checked once every 5 minutes is not something that necessarily applies to everything Hobbit monitors. I havent looked at Andy's trap script, but if I were to implement SNMP trap handling in Hobbit, I'd start off with snmptrapd from the Net-SNMP tools - this receives snmp traps, and can be configured to do "something" when a trap arrives. That "something" would then be a script/utility that grabs the hostname and trap type from the trap information, and feeds that into Hobbit as a status update. That will give you an immediate alert, and a status change in the Hobbit display if you use the "Critical systems" view which is dynamically generated. I'm not throwing rocks at Netcool <grin> but I just want to make it clear that Hobbit can be as real-time as you want it to - it's only a matter of feeding it data as quickly as you possible. Regards, Henrik
list Asif Iqbal
▸
On 7/17/07, Henrik Stoerner <user-ce4a2c883f75@xymon.invalid> wrote:
On Tue, Jul 17, 2007 at 08:07:31AM -0500, Hubbard, Greg L wrote:Being familiar with both Netcool and Hobbit, I would never consider replacing Netcool trap management with Hobbit. Netcool processes traps in "near real time" and scales quite large. Hobbit does not provide real time monitoring (default is 5 minute samples, with 1 minute screen updates).There is nothing inherent in Hobbit that prevents it from doing real-time handling of events. Hobbit processes events as soon as it is told about them; the fact that some types of information is only checked once every 5 minutes is not something that necessarily applies to everything Hobbit monitors. I havent looked at Andy's trap script, but if I were to implement SNMP trap handling in Hobbit, I'd start off with snmptrapd from the
I wish you do :-). I have lots of customers who like to see that feature in
hobbit.
May be then I can convince my IT department to get rid of Netcool slowly :P
▸
Net-SNMP tools - this receives snmp traps, and can be configured todo "something" when a trap arrives. That "something" would then be a script/utility that grabs the hostname and trap type from the trap information, and feeds that into Hobbit as a status update. That will give you an immediate alert, and a status change in the Hobbit display if you use the "Critical systems" view which is dynamically generated. I'm not throwing rocks at Netcool <grin> but I just want to make it clear that Hobbit can be as real-time as you want it to - it's only a matter of feeding it data as quickly as you possible. Regards, Henrik
--
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
list Asif Iqbal
▸
On 7/16/07, Henrik Stoerner <user-ce4a2c883f75@xymon.invalid> wrote:
On Mon, Jul 16, 2007 at 12:43:55PM -0400, Eric Jacobs wrote:The trap.pl uses hobbitboard to find all the hosts that have the "trap" test defined.Bad idea - there's a chicken-and-egg problem here. Columns do not appear in the hobbitdboard output until they exist, i.e. a status has been reported to Hobbit. So if *something* must report a "trap" status to Hobbit before the trap.pl script sees that it should do the "trap" thing ... well, I guess you can tell where I'm going. Scripts that implement custom tests should use the "bbhostgrep" utility to scan the bb-hosts file for the hosts that have a specific tag defined. There is actually another way it can be done, but it's not yet documented: You can send a "hostinfo" command to hobbitd, and it will return a pre-parsed version of the bb-hosts file - one line per hosts, with the fields delimited by '|' characters, eg: mail.hswn.dk|172.16.10.2|NET:intern|COMMENT:Internal mail server|smtp|pop3|imap
What's the syntax to send a "hostinfo" command to "hobbitd"?
Regards,Henrik
-- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
list Andy Farrior
Here's a quick overview of how I handle SNMP traps with Hobbit: 1 - Snmptrapd is configured to feed SNMPTT the OID and hostname of the sending SNMP agent. 2 - SNMPTT then translates the OID into a text message based on the MIB description for that OID and logs the message in /var/log/messages (or where ever you put it). You can also configure SNMPTT to log to a MySQL database simultaneously. 3 - SEC monitors the /var/log/messages file for entries from SNMPTT. Since some equipment can send the same trap multiple times in quick sucession, SEC is configured to ignore duplicate messages for a second or two 4 - SEC then launches a wrapper script that sends Hobbit a message using Hobbit's BB client program. Hobbit will send an alert if its status is yellow/red. 5 - A script is run by Hobbit every 5 minutes to prevent any trap message columns from turning purple. (I don't want my screen turning purple if I don't get a trap inside of 30min or whatever the no response timeout period is for Hobbit.) In the event you get a rapid sequence of a "CRITICAL" trap and then a "Normal" trap, you'll get a Hobbit alert, but when you view the web page, it'll be green. You have to rely on the trap history to see all of the traps that SNMPTT recorded. The real trick (pain) is defining what traps you want to be CRITICAL and what not. Andy
▸
From: Asif Iqbal [mailto:user-6f4b51ac2a40@xymon.invalid]
Sent: Tuesday, July 17, 2007 9:31 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Hobbit not recognizing certain tests?
On 7/17/07, Henrik Stoerner <user-ce4a2c883f75@xymon.invalid> wrote:
On Tue, Jul 17, 2007 at 08:07:31AM -0500, Hubbard, Greg L wrote:Being familiar with both Netcool and Hobbit, I would never consider replacing Netcool trap management with Hobbit. Netcool processes traps in "near real time" and scales quite large. Hobbit does not provide real time monitoring (default is 5 minute samples, with 1 minute screen updates).
There is nothing inherent in Hobbit that prevents it from doing real-time handling of events. Hobbit processes events as soon as it is told about them; the fact that some types of information is only checked once every 5 minutes is not something that necessarily applies to everything Hobbit monitors. I havent looked at Andy's trap script, but if I were to implement SNMP trap handling in Hobbit, I'd start off with snmptrapd from the I wish you do :-). I have lots of customers who like to see that feature in hobbit. May be then I can convince my IT department to get rid of Netcool slowly :P Net-SNMP tools - this receives snmp traps, and can be configured to do "something" when a trap arrives. That "something" would then be a script/utility that grabs the hostname and trap type from the trap information, and feeds that into Hobbit as a status update. That will give you an immediate alert, and a status change in the Hobbit display if you use the "Critical systems" view which is dynamically generated. I'm not throwing rocks at Netcool <grin> but I just want to make it clear that Hobbit can be as real-time as you want it to - it's only a matter of feeding it data as quickly as you possible. Regards, Henrik -- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
list Henrik Størner
▸
On Tue, Jul 17, 2007 at 10:34:45AM -0400, Asif Iqbal wrote:
What's the syntax to send a "hostinfo" command to "hobbitd"?
This gives you all hosts: bb 127.0.0.1 hostinfo If you want to filter them, you can use the same filters (pagename, hostname) as the "hobbitdboard" command, e.g. bb 127.0.0.1 "hostinfo host=host[1-9].foo.com" would get host1.foo.com, host2.foo.com etc. Henrik