Devmon : a bbxsnmp.pl & routermon replacement
list Eric E *hs Schwimmer
Hi everybody, I've been working on a little project for the past 6 months or so, one that is designed to replace bbxsnmp.pl, routermon, and other similar SNMP data collectors for the Hobbit and BigBrother monitoring systems. In my (extremely biased) opinion, it is one of the better SNMP data collectors out there. Just a few key highlights: . Fast: Uses psuedo non-blocking SNMP queries (basically forked snmp) . Scalable: If you want to poll many remote devices very quickly, you can configure Devmon to work in a clustered configuration across multiple machines. . Easily managed: After the initial configuration, all host management is done by editing your display server's bb-hosts file. . Very customizable: Via the Devmon template system, a savvy sysadmin can use Devmon to monitor virtually any type of SNMP capable device. The project went up on SourceForge as of about 30 minutes ago; it's web page is at: http://devmon.sf.net You can download it form there, leave a message in the forums, join the mailing list, etc. I'm hoping that this will be the only Devmon message that goes out to the Hobbit list; if you have any questions or comments for me, please please please either send them to me personally, or, preferably, join the Devmon mailing list and post your questions there. And last, but not least, I'd like to thank Henrik once again for his excellent work on Hobbit, as Devmon wouldn't be very useful without it! :) -Eric Schwimmer Network Engineer UVA HSCS Network Engineering
list Henrik Størner
▸
On Thu, Feb 23, 2006 at 01:58:33PM -0500, Schwimmer, Eric E *HS wrote:
I've been working on a little project for the past 6 months or so, one that is designed to replace bbxsnmp.pl, routermon, and other similar SNMP data collectors for the Hobbit and BigBrother monitoring systems.
Nice! I've given it a quick look-over, and even with my less-than-stellar Perl knowledge, I can see that this covers a lot of the features that people have requested from an SNMP data collector for Hobbit. The idea about "templates" looks very promising - it will be interesting to try it out on some of the devices I have access to. Having good, well-documented and maintained SNMP support for Hobbit has been on my wish-list for quite some time.
▸
And last, but not least, I'd like to thank Henrik once again for his excellent work on Hobbit, as Devmon wouldn't be very useful without it!
:)I'm flattered that you made Hobbit the default in your config :-) And truly delighted that my little project is so useful, that You would spend time improving on it. Thanks! Henrik
list Dan Vande More
Amazing. I was just ruminating this morning about the sad shape of bb-xsnmp and how it wasn't integrated very well, didn't thread, and ill-maintaned. I especially like the idea of clusting it, though I'm not prepared to do that just yet. I do however, have tens of thousands of devices to monitor, so the potential exists for me to use it. Anyway, I'm diving into it as we speak, unfortunately, you don't have templates for any of my devices (Foundry serverirons, cisco 7609s, Ironport C600s, etc) so it's gonna take some work:) Thanks for this promising add-on. -Dan p.s. It didn't make the /var/run/devmon/ dir by default.
▸
On 2/23/06, Schwimmer, Eric E *HS <user-1e1008b069d5@xymon.invalid> wrote:Hi everybody, I've been working on a little project for the past 6 months or so, one that is designed to replace bbxsnmp.pl, routermon, and other similar SNMP data collectors for the Hobbit and BigBrother monitoring systems. In my (extremely biased) opinion, it is one of the better SNMP data collectors out there. Just a few key highlights: . Fast: Uses psuedo non-blocking SNMP queries (basically forked snmp) . Scalable: If you want to poll many remote devices very quickly, you can configure Devmon to work in a clustered configuration across multiple machines. . Easily managed: After the initial configuration, all host management is done by editing your display server's bb-hosts file. . Very customizable: Via the Devmon template system, a savvy sysadmin can use Devmon to monitor virtually any type of SNMP capable device. The project went up on SourceForge as of about 30 minutes ago; it's web page is at: http://devmon.sf.net You can download it form there, leave a message in the forums, join the mailing list, etc. I'm hoping that this will be the only Devmon message that goes out to the Hobbit list; if you have any questions or comments for me, please please please either send them to me personally, or, preferably, join the Devmon mailing list and post your questions there. And last, but not least, I'd like to thank Henrik once again for his excellent work on Hobbit, as Devmon wouldn't be very useful without it!
:)
-Eric Schwimmer
Network Engineer
UVA HSCS Network Engineering
list Steve Aiello
Trying to play with it too, I am sure I can figure out where to pop SNMP_Session to get things to work. Just anyone know the 'proper', method ? Does this deal with devices sending SNMO traps to, or just polling ?
▸
-----Original Message----- From: Dan Vande More [mailto:user-f3c4c62d9d50@xymon.invalid] Sent: Thursday, February 23, 2006 4:38 PM To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] Devmon : a bbxsnmp.pl & routermon replacement Amazing. I was just ruminating this morning about the sad shape of bb-xsnmp and how it wasn't integrated very well, didn't thread, and ill-maintaned. I especially like the idea of clusting it, though I'm not prepared to do that just yet. I do however, have tens of thousands of devices to monitor, so the potential exists for me to use it. Anyway, I'm diving into it as we speak, unfortunately, you don't have templates for any of my devices (Foundry serverirons, cisco 7609s, Ironport C600s, etc) so it's gonna take some work:) Thanks for this promising add-on. -Dan p.s. It didn't make the /var/run/devmon/ dir by default. On 2/23/06, Schwimmer, Eric E *HS <user-1e1008b069d5@xymon.invalid> wrote:Hi everybody, I've been working on a little project for the past 6 months or so, one > that is designed to replace bbxsnmp.pl, routermon, and other similar > SNMP data collectors for the Hobbit and BigBrother monitoring systems. In my (extremely biased) opinion, it is one of the better SNMP data > collectors out there. Just a few key highlights: . Fast: Uses psuedo non-blocking SNMP queries (basically forked snmp) . Scalable: If you want to poll many remote devices very quickly, you can configure Devmon to work in a clustered configuration across multiple machines. . Easily managed: After the initial configuration, all host management is done by editing your display server's bb-hosts file. . Very customizable: Via the Devmon template system, a savvy sysadmin can use Devmon to monitor virtually any type of SNMP capable > device. The project went up on SourceForge as of about 30 minutes ago; it's > web page is at: http://devmon.sf.net You can download it form there, leave a message in the forums, join > the mailing list, etc. I'm hoping that this will be the only Devmon > message that goes out to the Hobbit list; if you have any questions > or comments for me, please please please either send them to me > personally, or, preferably, join the Devmon mailing list and post your > questions there. And last, but not least, I'd like to thank Henrik once again for his > excellent work on Hobbit, as Devmon wouldn't be very useful without > it!
:) -Eric Schwimmer Network Engineer UVA HSCS Network Engineering
list Eric E *hs Schwimmer
Well, to be honest, the code is still far from what I would consider "good" :) The template code, which is run once for every device during every polling cycle, is far from optimal (by far the slowest part of Devmon thus far; if there are any Elder Perl Gods out there that might want to throw some Perl wisdom my way, please feel free). Multi-node IPC is also kind of a hack, too, since its all done through the MySQL server (if someone wants to port Devmon to Postgres or some other DB, or even take a shot at optimizing the current MySQL tables, please do so. I am not a DB guru). That being said, it runs pretty solid. Starting up a multi-node cluster takes some patience and little TLC, but once it's running it works fairly well. As to making Hobbit the default config: choosing between Hobbit and BB was a no-brainer for us here. Hobbit is "Just Better". -Eric
▸
-----Original Message-----
From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: Thu 2/23/2006 4:26 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Devmon : a bbxsnmp.pl & routermon replacement
On Thu, Feb 23, 2006 at 01:58:33PM -0500, Schwimmer, Eric E *HS wrote:I've been working on a little project for the past 6 months or so, one that is designed to replace bbxsnmp.pl, routermon, and other similar SNMP data collectors for the Hobbit and BigBrother monitoring systems.
Nice! I've given it a quick look-over, and even with my less-than-stellar Perl knowledge, I can see that this covers a lot of the features that people have requested from an SNMP data collector for Hobbit. The idea about "templates" looks very promising - it will be interesting to try it out on some of the devices I have access to. Having good, well-documented and maintained SNMP support for Hobbit has been on my wish-list for quite some time.
And last, but not least, I'd like to thank Henrik once again for his excellent work on Hobbit, as Devmon wouldn't be very useful without it!
:)▸
I'm flattered that you made Hobbit the default in your config :-) And truly delighted that my little project is so useful, that You would spend time improving on it. Thanks! Henrik
list Eric E *hs Schwimmer
Tens... of thousands... wow. That's alot. Alot alot. That will definitely push Devmon to limits that we haven't even dreamed of yet. Right now we are polling ~800 devices with Devmon, and we are running at about 75% load (for a 60 second poll period) using 7 polling nodes. I can only imagine how manning polling nodes it will take you to achive the same polling interval (if you figure about 100 devices per node, thats 100 nodes for 10k devices. Yikes!) Most of the load (80-90% of it) is spent running template calculations, which right now is done on a single thread. I thought, earlier on in the design, that most time-intensive portion of the polling cycle was going to be the SNMP data collection portion. It looks like I'm going to have to farm the template code out to the forked processes, if I want to get any more speed out of this puppy. I've had the idea rolling around in the back of my head for a while now, but I've been very good at ignoring it, as it will involve rewriting a large chunk of code. (Not again!) Let me know how the template building goes. I did all of the template documentation in a single night, so it might be a bit lacking :) (Suggestions for change are welcome). Once you get your templates done, send 'em my way too and I'll roll them into the distro. -Eric
▸
-----Original Message-----
From: Dan Vande More [mailto:user-f3c4c62d9d50@xymon.invalid]
Sent: Thu 2/23/2006 4:38 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Devmon : a bbxsnmp.pl & routermon replacement
Amazing. I was just ruminating this morning about the sad shape of
bb-xsnmp and how it wasn't integrated very well, didn't thread, and
ill-maintaned. I especially like the idea of clusting it, though I'm
not prepared to do that just yet. I do however, have tens of thousands
of devices to monitor, so the potential exists for me to use it.
Anyway, I'm diving into it as we speak, unfortunately, you don't have
templates for any of my devices (Foundry serverirons, cisco 7609s,
Ironport C600s, etc) so it's gonna take some work:)
Thanks for this promising add-on.
-Dan
p.s. It didn't make the /var/run/devmon/ dir by default.
On 2/23/06, Schwimmer, Eric E *HS <user-1e1008b069d5@xymon.invalid> wrote:Hi everybody, I've been working on a little project for the past 6 months or so, one that is designed to replace bbxsnmp.pl, routermon, and other similar SNMP data collectors for the Hobbit and BigBrother monitoring systems. In my (extremely biased) opinion, it is one of the better SNMP data collectors out there. Just a few key highlights: . Fast: Uses psuedo non-blocking SNMP queries (basically forked snmp) . Scalable: If you want to poll many remote devices very quickly, you can configure Devmon to work in a clustered configuration across multiple machines. . Easily managed: After the initial configuration, all host management is done by editing your display server's bb-hosts file. . Very customizable: Via the Devmon template system, a savvy sysadmin can use Devmon to monitor virtually any type of SNMP capable device. The project went up on SourceForge as of about 30 minutes ago; it's web page is at: http://devmon.sf.net You can download it form there, leave a message in the forums, join the mailing list, etc. I'm hoping that this will be the only Devmon message that goes out to the Hobbit list; if you have any questions or comments for me, please please please either send them to me personally, or, preferably, join the Devmon mailing list and post your questions there. And last, but not least, I'd like to thank Henrik once again for his excellent work on Hobbit, as Devmon wouldn't be very useful without it!
:)
-Eric Schwimmer
Network Engineer
UVA HSCS Network Engineering
list Eric E *hs Schwimmer
For me, installing SNMP_Session was as easy as: myhost% wget http://www.switch.ch/misc/leinen/snmp/perl/dist/SNMP_Session-1.08.tar.gz myhost% tar zxf SNMP_Session-1.08.tar.gz myhost% cd SNMP_Session-1.08/ myhost% perl Makefile.PL myhost% make install Also, I hate to be a nag, but please try to refrain from sending support questions to the Hobbit mailing list. Knowing my code, I'm sure that there are going to be enough of them that people who aren't interested in Devmon will start to get annoyed at the traffic levels :) You can subscribe to the Devmon mailing list here: -Eric
▸
-----Original Message-----
From: Aiello, Steve (GE, Corporate, consultant) [mailto:user-49fae449733a@xymon.invalid]
Sent: Thu 2/23/2006 4:45 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: RE: [hobbit] Devmon : a bbxsnmp.pl & routermon replacement
Trying to play with it too, I am sure I can figure out where to pop
SNMP_Session to get things to work. Just anyone know the 'proper',
method ?
Does this deal with devices sending SNMO traps to, or just polling ?
-----Original Message----- From: Dan Vande More [mailto:user-f3c4c62d9d50@xymon.invalid] Sent: Thursday, February 23, 2006 4:38 PM To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] Devmon : a bbxsnmp.pl & routermon replacement Amazing. I was just ruminating this morning about the sad shape of bb-xsnmp and how it wasn't integrated very well, didn't thread, and ill-maintaned. I especially like the idea of clusting it, though I'm not prepared to do that just yet. I do however, have tens of thousands of devices to monitor, so the potential exists for me to use it. Anyway, I'm diving into it as we speak, unfortunately, you don't have templates for any of my devices (Foundry serverirons, cisco 7609s, Ironport C600s, etc) so it's gonna take some work:) Thanks for this promising add-on. -Dan p.s. It didn't make the /var/run/devmon/ dir by default. On 2/23/06, Schwimmer, Eric E *HS <user-1e1008b069d5@xymon.invalid> wrote:Hi everybody, I've been working on a little project for the past 6 months or so, one > that is designed to replace bbxsnmp.pl, routermon, and other similar > SNMP data collectors for the Hobbit and BigBrother monitoring systems. In my (extremely biased) opinion, it is one of the better SNMP data > collectors out there. Just a few key highlights: . Fast: Uses psuedo non-blocking SNMP queries (basically forked snmp) . Scalable: If you want to poll many remote devices very quickly, you can configure Devmon to work in a clustered configuration across multiple machines. . Easily managed: After the initial configuration, all host management is done by editing your display server's bb-hosts file. . Very customizable: Via the Devmon template system, a savvy sysadmin can use Devmon to monitor virtually any type of SNMP capable > device. The project went up on SourceForge as of about 30 minutes ago; it's > web page is at: http://devmon.sf.net You can download it form there, leave a message in the forums, join > the mailing list, etc. I'm hoping that this will be the only Devmon > message that goes out to the Hobbit list; if you have any questions > or comments for me, please please please either send them to me > personally, or, preferably, join the Devmon mailing list and post your > questions there. And last, but not least, I'd like to thank Henrik once again for his > excellent work on Hobbit, as Devmon wouldn't be very useful without > it!
:) -Eric Schwimmer Network Engineer UVA HSCS Network Engineering
list Henrik Størner
▸
On Thu, Feb 23, 2006 at 11:14:52PM -0500, Schwimmer, Eric E *HS wrote:
Right now we are polling ~800 devices with Devmon, and we are running at about 75% load (for a 60 second poll period) using 7 polling nodes.
[snip]
Most of the load (80-90% of it) is spent running template calculations, which right now is done on a single thread. I thought, earlier on in the design, that most time-intensive portion of the polling cycle was going to be the SNMP data collection portion.
The "template calculations" are where you compare the SNMP data you've collected against the various limits you've setup for each host, I suppose ? To me, it sounds as if this part of the code ought to be mostly CPU-bound. You're not loading a lot of data from disk. Have you tried looking at some vmstat data while this runs ?
▸
It looks like I'm going to have to farm the template code out to the forked processes, if I want to get any more speed out of this puppy. I've had the idea rolling around in the back of my head for a while now, but I've been very good at ignoring it, as it will involve rewriting a large chunk of code. (Not again!)
If - as I suspect - it is CPU-bound, then splitting up the task on multiple processes running on a single node won't give you any improvement. If you split this onto multiple nodes, it's a different story, of course. Just some input from a guy who's spent too much time watching how BB spent its time running all of the shell scripts :-) I have a couple of comments about how Devmon does some things, but I'll take those over to the Devmon list. Regards, Henrik
list Eric E *hs Schwimmer
▸
The "template calculations" are where you compare the SNMP data you've collected against the various limits you've setup for each host, I suppose ? To me, it sounds as if this part of the code ought to be mostly CPU-bound. You're not loading a lot of data from disk. Have you tried looking at some vmstat data while this runs ?
Yep, the template calculations involve taking the SNMP data and putting it through various transform functions (converting numeric values to string values, converting bits to bytes, doing regexp substitutions, etc) and then comparing them to the threshold data. The lions share of the time is spent on the interface tests (if_stat, if_load, if_err, etc), as some switches have SNMP repeater oids with 50+ leaves, and each leaf gets run through the gamut of all these subroutines. This will be true for any type of test (whether it be for layer2 devices or not) that you want to test large repeater-type OIDs on.
▸
If - as I suspect - it is CPU-bound, then splitting up the task on multiple processes running on a single node won't give you any improvement. If you split this onto multiple nodes, it's a different story, of course. Just some input from a guy who's spent too much time watching how BB spent its time running all of the shell scripts :-)
Arggh. I just realized that you are totally correct. Indeed, when the SNMP data collection process ends and the template calculations begin, the CPU usage of the master devmon process jumps to 99%. Sigh. And here I was looking to take the "easy" way out :) -Eric