System Log Monitoring
list Tom Callahan
Morning all,
Is there a way to monitor for certain key words that show up in the
system logs? ie. I would like for hobbit to page me "iff" the words
"FAILOVER", "LOOP DEAD", and "LOOP DOWN" are detected in the
/var/logs/messages file on our Linux machines.
Let me know if this is possible.
Thanks,
Tom Callahan
list Larry Barber
Enter the phrases you want to look for in the bb-msgtab file in your BB client's etc directory. Thanks, Larry Barber
▸
On Wed, 2005-06-22 at 09:26 -0500, user-16f19114071e@xymon.invalid wrote:Morning all, Is there a way to monitor for certain key words that show up in the system logs? ie. I would like for hobbit to page me "iff" the words "FAILOVER", "LOOP DEAD", and "LOOP DOWN" are detected in the /var/logs/messages file on our Linux machines. Let me know if this is possible. Thanks, Tom Callahan
list Bob Gordon
Hello Henrick - Wondering if you had any kind of time frame for adding system log monitoring into the client? (/var/log/syslog, /var/adm/messages and possibly a user configurable one).... Thanks! -- --==[ Bob Gordon ]==--
list Henrik Størner
▸
On Thu, Aug 11, 2005 at 11:21:48PM -0700, Bob Gordon wrote:
Hello Henrick - Wondering if you had any kind of time frame for adding system log monitoring into the client? (/var/log/syslog, /var/adm/messages and possibly a user configurable one)....
The thing is I don't really know what's the best way to do that. I guess the log monitoring will need some local configuration, although I really hate that. But sending entire logs across the wire every 5 minutes won't work. Also, I'd really like to have some mechanism for alerting different people depending on what log-entry triggers an alert. (Same goes for the "procs" and perhaps "disk" olumns, by the way - but I know how to implement that). So ideas and suggestions are welcome. There *is* a hook in the current client, so that if the client sends - as part of the client message - a section headed "[msgs]", then this will be used for the "msgs" column instead of the default "clear" status. Any "&red" or "&yellow" identifiers will trigger a change of the color. There's also a deadcat add-on for log monitoring, that can be run as a client extension script. You'll need to disable the default Hobbit client reporting of the "msgs" column, which is not possible in 4.1.1, but has been added in the current snapshots and upcoming 4.1.2. Regards, Henrik
list Bob Gordon
▸
On 8/12/05, Henrik Stoerner <user-ce4a2c883f75@xymon.invalid> wrote:
I guess the log monitoring will need some local configuration, although I really hate that. But sending entire logs across the wire every 5 minutes won't work.
True.. That can get quite intensive...
▸
Also, I'd really like to have some mechanism for alerting different people depending on what log-entry triggers an alert. (Same goes for the "procs" and perhaps "disk" olumns, by the way - but I know how to implement that). So ideas and suggestions are welcome.
Hmmm... How do you feel about pushing configs to clients? (or pulling from).. The configuration could be maintained on the server and every 8 hours or so (configurable) the client requests a configuration update. You can probably use the same alerting mechanism from the procs and disks columns.
▸
There *is* a hook in the current client, so that if the client sends - as part of the client message - a section headed "[msgs]", then this will be used for the "msgs" column instead of the default "clear" status. Any "&red" or "&yellow" identifiers will trigger a change of the color.
I haven't had a chance to look at the client that much (only installed it last night).. Where do we add something for this? clientlaunch.cfg?
▸
There's also a deadcat add-on for log monitoring, that can be run as a client extension script. You'll need to disable the default Hobbit client reporting of the "msgs" column, which is not possible in 4.1.1, but has been added in the current snapshots and upcoming 4.1.2.
Adding the scripts is a piece of cake in this client (Thanks!!).. For right now where would we disable the client msgs? (I am running the 4.1.2 snapshot).. -- --==[ Bob Gordon ]==--
list Jason K. Kruse
-----Original Message----- From: user-ce4a2c883f75@xymon.invalid [mailto:user-ce4a2c883f75@xymon.invalid]
▸
Also, I'd really like to have some mechanism for alerting different
people depending on what log-entry triggers an alert. (Same goes for
the "procs" and perhaps "disk" olumns, by the way - but I know how
to implement that).
So ideas and suggestions are welcome.I hacked up big brother to do this and was going to try to get this for hobbit when I had free time. I'd post code but I'm not sure if I'm legally allowed. I created a new config file I called the twitchtab. Each entry was: Hostname ; service ; specific item ; people to page ; people to not page I have a wrapper script that parses each service and the warnrules to create the people to page/ignore entry to put in the file. This needs to be consistent with warnrules since I trigger the check after a match of a notification being sent. Messages have a specific item of 'file : " string you matched on "'. I use perl so some of this might be harder in C. Basically I read until '&pagelevel /filename (scanned at ...)', parse the filename and set a hash with the list of recipients. I then read until the next &color /filename, checking each line to see if our string is found. Each line is treated as having it's own list of recipients with a tally being done at the end to find who really gets notified. At the end I walk the hash and create the final list of recipients. Usually it's the same list as the beginning since it's hard to account for all items that may be put in the log. Anyhow, I'm really looking forward to this since it would be one less thing I need to support myself. Jason
list Asif Iqbal
▸
On Fri, Aug 12, 2005 at 09:14:24AM, Henrik Stoerner wrote:
On Thu, Aug 11, 2005 at 11:21:48PM -0700, Bob Gordon wrote:Hello Henrick - Wondering if you had any kind of time frame for adding system log monitoring into the client? (/var/log/syslog, /var/adm/messages and possibly a user configurable one)....The thing is I don't really know what's the best way to do that. I guess the log monitoring will need some local configuration, although I really hate that. But sending entire logs across the wire every 5 minutes won't work. Also, I'd really like to have some mechanism for alerting different people depending on what log-entry triggers an alert. (Same goes for the "procs" and perhaps "disk" olumns, by the way - but I know how to implement that). So ideas and suggestions are welcome.
How about using something like logtail--on client--run every 5 mins and alert if matches a string. As far as who should get be paged for the alert can still be managed by SCRIPT. This way no same string of unique time stamp wont be seen twice and hence won't have to rotate the messages file manually to stop the alert. I am sure you already thought of something better. Just making my "voice" heard ;-) -- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu "..there are two kinds of people: those who work and those who take the credit...try to be in the first group;...less competition there." - Indira Gandhi