hobbitclient + msgs test sugesstion
list Peter Welter
Hello Henrik + fellow hobbiters, Since the msgs-check is not available yet in the Hobbit display, I want to make a suggestion to have it enabled relatively easy. I think of two methods: -1- A client must have read access to the file, lets say /var/adm/messages, but that has a major drawback: you can only monitor one [predefined] file. Anyway, you can pass on the major BB functionality. But any other logfiles, as (is/was) possible with bb-msgtab is discarded. -2- Your Hobbit server must _also_ be a central loghost. In this way, all configuration remains on the Hobbit server (a GREAT welcome Hobbit-feature). You, Henrik, can even define the format that you want the central loghost being set up. Fi. seperate .info, .debug -> green, .kern -> yellow, etc. Each host can use up a history of one week message-files and serious trouble can be set up/defined within, let's say, /etc/hobbit-client-messages.cfg. Now it is even possible to have this setup work for another logfile, let's say, /var/log/mail? Perhaps (I am googling rigth now) this is useful: http://freshmeat.net/projects/syslog-ng/ but I guess this paints the picture. Well, that's just my 2 cents. Regards, Peter
list Henrik Størner
Hi Peter (and anyone else interested),
▸
On Tue, Nov 29, 2005 at 08:26:14PM +0100, Peter Welter wrote:Since the msgs-check is not available yet in the Hobbit display, I want to make a suggestion to have it enabled relatively easy. I think of two methods:
-1- A client must have read access to the file [client picks out the
"interesting" bits]
-2- Your Hobbit server must _also_ be a central loghost. [allows
centralized configuration of how to monitor the logs]I'm not really thrilled with either of these - sorry! Each of them have some drawbacks: The first one moves the configuration of what logs to monitor away from the central hobbit server, and the second one only works for logs that go through the syslog interface. If I want to monitor e.g. an Apache webserver error-log, or the custom logs from a BEA server, solution 2) won't work. I dont see how it can work with logs from a Windows server either. Plus it adds load to the central Hobbit server to deal with all of the logfiles. So - I think some other solution is needed, and I've been thinking about how to do it. So far it's just ideas - no code. But I believe the log checking has to happen on each client, but controlled by a central configuration. So what I plan to implement is something like this: * The configuration of what logs to monitor and what strings to look for is maintained on the central Hobbit server, either as an addition to the hobbit-clients.cfg file, or in a separate file - that isn't really important. * When a client connects and sends in a client-side message, the Hobbit server accepts the client message, but also sends back the current log-check configuration info. By re-using the client connection, the overhead involved in pushing the configuration to each client becomes almost nil. * When the client has a log-check configuration, it knows what logs to check for what strings, and can include that information in the normal client message it sends back to the Hobbit server. That means the client will need a tool to do the logfile checking; probably using a client-side regular-expression matching tool like "grep". It can either be built into the Hobbit client, or it could just rely on the existing "grep" utility found on the system - this would probably be the simplest to implement. Regards, Henrik
list Rich Smrcina
Doesn't the current bb implementation only monitor log file messages issued during a time period just past (like 30 minutes)? I like the expression matching idea, but repetitively walking through the whole thing could introduce alot of overhead. It would be beneficial walk backward from the end of the file to a pre-configured point in the past, then scan the messages. Sorry if this is obvious... :)
▸
Henrik Stoerner wrote:Hi Peter (and anyone else interested), On Tue, Nov 29, 2005 at 08:26:14PM +0100, Peter Welter wrote:Since the msgs-check is not available yet in the Hobbit display, I want to make a suggestion to have it enabled relatively easy. I think of two methods: -1- A client must have read access to the file [client picks out the "interesting" bits] -2- Your Hobbit server must _also_ be a central loghost. [allows centralized configuration of how to monitor the logs]I'm not really thrilled with either of these - sorry! Each of them have some drawbacks: The first one moves the configuration of what logs to monitor away from the central hobbit server, and the second one only works for logs that go through the syslog interface. If I want to monitor e.g. an Apache webserver error-log, or the custom logs from a BEA server, solution 2) won't work. I dont see how it can work with logs from a Windows server either. Plus it adds load to the central Hobbit server to deal with all of the logfiles. So - I think some other solution is needed, and I've been thinking about how to do it. So far it's just ideas - no code. But I believe the log checking has to happen on each client, but controlled by a central configuration. So what I plan to implement is something like this: * The configuration of what logs to monitor and what strings to look for is maintained on the central Hobbit server, either as an addition to the hobbit-clients.cfg file, or in a separate file - that isn't really important. * When a client connects and sends in a client-side message, the Hobbit server accepts the client message, but also sends back the current log-check configuration info. By re-using the client connection, the overhead involved in pushing the configuration to each client becomes almost nil. * When the client has a log-check configuration, it knows what logs to check for what strings, and can include that information in the normal client message it sends back to the Hobbit server. That means the client will need a tool to do the logfile checking; probably using a client-side regular-expression matching tool like "grep". It can either be built into the Hobbit client, or it could just rely on the existing "grep" utility found on the system - this would probably be the simplest to implement. Regards, Henrik
--
Rich Smrcina VM Assist, Inc. Main: (262)392-2026 Cell: (XXX)XXX-XXXX Ans Service: (360)715-2467 user-61add9955ef9@xymon.invalid Catch the WAVV! http://www.wavv.org WAVV 2006 - Chattanooga, TN - April 7-11, 2006
list Steve Aiello
▸
So - I think some other solution is needed, and I've been thinking about how to do it. So far it's just ideas - no code. But I believe the log checking has to happen on each client, but controlled by a central configuration. So what I plan to implement is something like this: * The configuration of what logs to monitor and what strings to look for is maintained on the central Hobbit server, either as an addition to the hobbit-clients.cfg file, or in a separate file - that isn't really important. * When a client connects and sends in a client-side message, the Hobbit server accepts the client message, but also sends back the current log-check configuration info. By re-using the client connection, the overhead involved in pushing the configuration to each client becomes almost nil. * When the client has a log-check configuration, it knows what logs to check for what strings, and can include that information in the normal client message it sends back to the Hobbit server. That means the client will need a tool to do the logfile checking; probably using a client-side regular-expression matching tool like "grep". It can either be built into the Hobbit client, or it could just rely on the existing "grep" utility found on the system - this would probably be the simplest to implement.
I had come to the same conclusion when trying to think how to implement msgs tests in a 'hobbit' way. So it seems that hobbit needs some manner of 'push' mechanism. Of course this is just pushing configuration, but since you are talking about a 'push' mechanism what about pushing files too ? Though this option may be cosidered a security hole. But this would be nice for pushing out updates. Whether it is a full Hobbit client update, or if a particular script/test needed to be pushed out and updated. Pushing out files could be a configureable option that can be enable/disabled.
list Asif Iqbal
▸
On Tue, Nov 29, 2005 at 10:09:48PM, Henrik Stoerner wrote:
Hi Peter (and anyone else interested), On Tue, Nov 29, 2005 at 08:26:14PM +0100, Peter Welter wrote:Since the msgs-check is not available yet in the Hobbit display, I want to make a suggestion to have it enabled relatively easy. I think of two methods: -1- A client must have read access to the file [client picks out the "interesting" bits] -2- Your Hobbit server must _also_ be a central loghost. [allows centralized configuration of how to monitor the logs]I'm not really thrilled with either of these - sorry! Each of them have some drawbacks: The first one moves the configuration of what logs to monitor away from the central hobbit server, and the second one only works for logs that go through the syslog interface. If I want to monitor e.g. an Apache webserver error-log, or the custom logs from a BEA server, solution 2) won't work. I dont see how it can work with logs from a Windows server either. Plus it adds load to the central Hobbit server to deal with all of the logfiles. So - I think some other solution is needed, and I've been thinking about how to do it. So far it's just ideas - no code. But I believe the log checking has to happen on each client, but controlled by a central configuration. So what I plan to implement is something like this: * The configuration of what logs to monitor and what strings to look for is maintained on the central Hobbit server, either as an addition to the hobbit-clients.cfg file, or in a separate file - that isn't really important. * When a client connects and sends in a client-side message, the Hobbit server accepts the client message, but also sends back the current log-check configuration info. By re-using the client connection, the overhead involved in pushing the configuration to each client becomes almost nil. * When the client has a log-check configuration, it knows what logs to check for what strings, and can include that information in the normal client message it sends back to the Hobbit server. That means the client will need a tool to do the logfile checking; probably using a client-side regular-expression matching tool like "grep". It can either be built into the Hobbit client, or it could just rely on the existing "grep" utility found on the system - this would probably be the simplest to implement.
Would it be possible to create a new hobbitd channel that will get install with hobbit client. Then add that channel to the syslog.conf which is kind a work like a pipe. So when syslog say related to /var/adm/messages file get send to the hobbitd channel (or pipe) it will scan right away against strings that needs to get alerted about. Also it won't store anything in the channel. So there is no chance to scan the same string on the same timestamp twice. Also if it is not receiving any alert for say 5 mins it will check if syslogd is actually running by sending a 'logger' output to the channel. Sorry if I talking 'no sense' but throwing anything here while the idea is still cooking :-) Thanks
Regards, Henrik
-- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu "..there are two kinds of people: those who work and those who take the credit...try to be in the first group;...less competition there." - Indira Gandhi
list Manu
Hi, Why should the log-entries, hobbit-msg-monitor should look after be maintained centrally on the hobbit server? Are important log-entries essential on a sql-server as well in every case as on a firewall? I think that there are different things on different servers you want hobbit to take care about... Maybe I missed the point... To reduce overhead, you can use a similar mechanism as logtail does. Storing the file offset in conjunction with the inode-id would grant you never check an entry twice. Maybe, having a closer look at logsentry from the sentry-tools (http://sourceforge.net/projects/sentrytools) would help finding an appropriate way of realizing this. Kind regards, Manuel ----- Message from user-c8222abeff59@xymon.invalid ---------
▸
Date: Tue, 29 Nov 2005 16:42:21 -0500
From: Asif Iqbal <user-c8222abeff59@xymon.invalid>
Reply-To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] hobbitclient + msgs test sugesstion
To: user-ae9b8668bcde@xymon.invalid
On Tue, Nov 29, 2005 at 10:09:48PM, Henrik Stoerner wrote:Hi Peter (and anyone else interested), On Tue, Nov 29, 2005 at 08:26:14PM +0100, Peter Welter wrote:Since the msgs-check is not available yet in the Hobbit display, I want to make a suggestion to have it enabled relatively easy. I think of two methods: -1- A client must have read access to the file [client picks out the "interesting" bits] -2- Your Hobbit server must _also_ be a central loghost. [allows centralized configuration of how to monitor the logs]I'm not really thrilled with either of these - sorry! Each of them have some drawbacks: The first one moves the configuration of what logs to monitor away from the central hobbit server, and the second one only works for logs that go through the syslog interface. If I want to monitor e.g. an Apache webserver error-log, or the custom logs from a BEA server, solution 2) won't work. I dont see how it can work with logs from a Windows server either. Plus it adds load to the central Hobbit server to deal with all of the logfiles. So - I think some other solution is needed, and I've been thinking about how to do it. So far it's just ideas - no code. But I believe the log checking has to happen on each client, but controlled by a central configuration. So what I plan to implement is something like this: * The configuration of what logs to monitor and what strings to look for is maintained on the central Hobbit server, either as an addition to the hobbit-clients.cfg file, or in a separate file - that isn't really important. * When a client connects and sends in a client-side message, the Hobbit server accepts the client message, but also sends back the current log-check configuration info. By re-using the client connection, the overhead involved in pushing the configuration to each client becomes almost nil. * When the client has a log-check configuration, it knows what logs to check for what strings, and can include that information in the normal client message it sends back to the Hobbit server. That means the client will need a tool to do the logfile checking; probably using a client-side regular-expression matching tool like "grep". It can either be built into the Hobbit client, or it could just rely on the existing "grep" utility found on the system - this would probably be the simplest to implement.Would it be possible to create a new hobbitd channel that will get install with hobbit client. Then add that channel to the syslog.conf which is kind a work like a pipe. So when syslog say related to /var/adm/messages file get send to the hobbitd channel (or pipe) it will scan right away against strings that needs to get alerted about. Also it won't store anything in the channel. So there is no chance to scan the same string on the same timestamp twice. Also if it is not receiving any alert for say 5 mins it will check if syslogd is actually running by sending a 'logger' output to the channel. Sorry if I talking 'no sense' but throwing anything here while the idea is still cooking :-) ThanksRegards, Henrik-- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu "..there are two kinds of people: those who work and those who take the credit...try to be in the first group;...less competition there." - Indira Gandhi
----- End message from user-c8222abeff59@xymon.invalid -----
list Allan Marillier
A log file monitor would also need to consider what happens when logrotate (or other switching mechanism) switches logs, as the inode and file offset may (or will) change. I know some people have a cron job which copies the log, then truncates the original log file. In this case the inode won't change, but the file contents / offset will. There are many things to consider, I'm sure Henrik will come up with a good solution. Manu <user-ba950eaddb92@xymon.invalid> 11/29/2005 05:38 PM Please respond to user-ae9b8668bcde@xymon.invalid To user-ae9b8668bcde@xymon.invalid cc
▸
Subject Re: [hobbit] hobbitclient + msgs test sugesstion Hi, Why should the log-entries, hobbit-msg-monitor should look after be maintained centrally on the hobbit server? Are important log-entries essential on a sql-server as well in every case as on a firewall? I think that there are different things on different servers you want hobbit to take care about... Maybe I missed the point... To reduce overhead, you can use a similar mechanism as logtail does. Storing the file offset in conjunction with the inode-id would grant you never check an entry twice. Maybe, having a closer look at logsentry from the sentry-tools (http://sourceforge.net/projects/sentrytools) would help finding an appropriate way of realizing this. Kind regards, Manuel ----- Message from user-c8222abeff59@xymon.invalid --------- Date: Tue, 29 Nov 2005 16:42:21 -0500 From: Asif Iqbal <user-c8222abeff59@xymon.invalid> Reply-To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] hobbitclient + msgs test sugesstion To: user-ae9b8668bcde@xymon.invalid
On Tue, Nov 29, 2005 at 10:09:48PM, Henrik Stoerner wrote:Hi Peter (and anyone else interested), On Tue, Nov 29, 2005 at 08:26:14PM +0100, Peter Welter wrote:Since the msgs-check is not available yet in the Hobbit display, I want to make a suggestion to have it enabled relatively easy. I think of two methods: -1- A client must have read access to the file [client picks out the "interesting" bits] -2- Your Hobbit server must _also_ be a central loghost. [allows centralized configuration of how to monitor the logs]I'm not really thrilled with either of these - sorry! Each of them have some drawbacks: The first one moves the configuration of what logs to monitor away from the central hobbit server, and the second one only works for logs that go through the syslog interface. If I want to monitor e.g. an Apache webserver error-log, or the custom logs from a BEA server, solution 2) won't work. I dont see how it can work with logs from a Windows server either. Plus it adds load to the central Hobbit server to deal with all of the logfiles. So - I think some other solution is needed, and I've been thinking about how to do it. So far it's just ideas - no code. But I believe the log checking has to happen on each client, but controlled by a central configuration. So what I plan to implement is something like this: * The configuration of what logs to monitor and what strings to look for is maintained on the central Hobbit server, either as an addition to the hobbit-clients.cfg file, or in a separate file - that isn't really important. * When a client connects and sends in a client-side message, the Hobbit server accepts the client message, but also sends back the current log-check configuration info. By re-using the client connection, the overhead involved in pushing the configuration to each client becomes almost nil. * When the client has a log-check configuration, it knows what logs to check for what strings, and can include that information in the normal client message it sends back to the Hobbit server. That means the client will need a tool to do the logfile checking; probably using a client-side regular-expression matching tool like "grep". It can either be built into the Hobbit client, or it could just rely on the existing "grep" utility found on the system - this would probably be the simplest to implement.Would it be possible to create a new hobbitd channel that will get install with hobbit client. Then add that channel to the syslog.conf which is kind a work like a pipe. So when syslog say related to /var/adm/messages file get send to the hobbitd channel (or pipe) it will scan right away against strings that needs to get alerted about. Also it won't store anything in the channel. So there is no chance to scan the same string on the same timestamp twice. Also if it is not receiving any alert for say 5 mins it will check if syslogd is actually running by sending a 'logger' output to the channel. Sorry if I talking 'no sense' but throwing anything here while the idea is still cooking :-) ThanksRegards, Henrik-- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu "..there are two kinds of people: those who work and those who take the credit...try to be in the first group;...less competition there." - Indira Gandhi
----- End message from user-c8222abeff59@xymon.invalid -----