Help with very large log file - not getting the right lines
list Elizabeth Schwartz
I've got to monitor some very large log files. They're up to a couple gigs a day and individual lines can be 30800 characters or more , including HTML. (changing the log file format is a project for another day) So my last half hour of one of these files chosen at random is 21,000 lines, 47G. I want to look at all the lines that start with 2011-11-22 4:15:31 ERROR servicename LotsOfText I want to ignore lines that start 2011-11-22 17:13:39 LOG NNNNN servicename LotsOfHTML Ignoring all of those lines would bring it to a manageable size (this particular file is 41 lines, 23k data) I've been playing around with rules in client-local.cfg like: [mmw2.example.com] log:/var/log/mmb1/MMRequest.log:10240 trigger ERROR ignore LOG but I'm just not getting the ERROR lines in the log. Is this file just too large and too full of HTML to parse? Any suggestions? (we can write a custom script, of course, and I'm thinking of bringing in SEC. But it sure would be handy to be able to do this with out of the box xymon)
list Elizabeth Schwartz
TYPO there, 47M not 47G. The files are about 2G/day On Tue, Nov 22, 2011 at 6:06 PM, Elizabeth Schwartz
▸
<user-c61747246f66@xymon.invalid> wrote:I've got to monitor some very large log files. They're up to a couple gigs a day and individual lines can be 30800 characters or more , including HTML. (changing the log file format is a project for another day) So my last half hour of one of these files chosen at random is 21,000 lines, 47G. I want to look at all the lines that start with 2011-11-22 4:15:31 ERROR servicename LotsOfText I want to ignore lines that start 2011-11-22 17:13:39 LOG NNNNN servicename LotsOfHTML Ignoring all of those lines would bring it to a manageable size (this particular file is 41 lines, 23k data) I've been playing around with rules in client-local.cfg like: [mmw2.example.com] log:/var/log/mmb1/MMRequest.log:10240 trigger ERROR ignore LOG but I'm just not getting the ERROR lines in the log. Is this file just too large and too full of HTML to parse? Any suggestions? (we can write a custom script, of course, and I'm thinking of bringing in SEC. But it sure would be handy to be able to do this with out of the box xymon)
list Ralph Mitchell
First of many "quick fixes": could you set up an auto-restarting script to do "tail -f logfile | grep ERROR > errorlog"?? Then watch the aeroflot file. Ralph Mitchell On Nov 22, 2011 6:07 PM, "Elizabeth Schwartz" <user-c61747246f66@xymon.invalid>
▸
wrote:
I've got to monitor some very large log files. They're up to a couple gigs a day and individual lines can be 30800 characters or more , including HTML. (changing the log file format is a project for another day) So my last half hour of one of these files chosen at random is 21,000 lines, 47G. I want to look at all the lines that start with 2011-11-22 4:15:31 ERROR servicename LotsOfText I want to ignore lines that start 2011-11-22 17:13:39 LOG NNNNN servicename LotsOfHTML Ignoring all of those lines would bring it to a manageable size (this particular file is 41 lines, 23k data) I've been playing around with rules in client-local.cfg like: [mmw2.example.com] log:/var/log/mmb1/MMRequest.log:10240 trigger ERROR ignore LOG but I'm just not getting the ERROR lines in the log. Is this file just too large and too full of HTML to parse? Any suggestions? (we can write a custom script, of course, and I'm thinking of bringing in SEC. But it sure would be handy to be able to do this with out of the box xymon)
list Xymon User in Richmond
My first thought, also. I think you can also train syslog-ng to do that, if you're using syslog-ng.
▸
On Tue, November 22, 2011 18:34, Ralph Mitchell wrote:First of many "quick fixes": could you set up an auto-restarting script to do "tail -f logfile | grep ERROR > errorlog"?? Then watch the aeroflot file. Ralph Mitchell On Nov 22, 2011 6:07 PM, "Elizabeth Schwartz" <user-c61747246f66@xymon.invalid> wrote:I've got to monitor some very large log files. They're up to a couple gigs a day and individual lines can be 30800 characters or more , including HTML. (changing the log file format is a project for another day) So my last half hour of one of these files chosen at random is 21,000 lines, 47G. I want to look at all the lines that start with 2011-11-22 4:15:31 ERROR servicename LotsOfText I want to ignore lines that start 2011-11-22 17:13:39 LOG NNNNN servicename LotsOfHTML Ignoring all of those lines would bring it to a manageable size (this particular file is 41 lines, 23k data) I've been playing around with rules in client-local.cfg like: [mmw2.example.com] log:/var/log/mmb1/MMRequest.log:10240 trigger ERROR ignore LOG but I'm just not getting the ERROR lines in the log. Is this file just too large and too full of HTML to parse? Any suggestions? (we can write a custom script, of course, and I'm thinking of bringing in SEC. But it sure would be handy to be able to do this with out of
the box xymon) _______________________________________________ Xymon mailing list Xymon at xymon.com
list Xymon User in Richmond
Actually, you can do the "tail" with the default rsyslogd imfile module. http://www.rsyslog.com/doc/imfile.html http://www.rsyslog.com/doc/multi_ruleset.html This was pointed out to me a while back by Irithori on linuxforums.org, and I had forgotten it.
▸
On Tue, November 22, 2011 20:20, Xymon User in Richmond wrote:My first thought, also. I think you can also train syslog-ng to do that, if you're using syslog-ng. On Tue, November 22, 2011 18:34, Ralph Mitchell wrote:First of many "quick fixes": could you set up an auto-restarting script to do "tail -f logfile | grep ERROR > errorlog"?? Then watch the aeroflot file. Ralph Mitchell On Nov 22, 2011 6:07 PM, "Elizabeth Schwartz" <user-c61747246f66@xymon.invalid> wrote:I've got to monitor some very large log files. They're up to a couple gigs a day and individual lines can be 30800 characters or more , including HTML. (changing the log file format is a project for another day) So my last half hour of one of these files chosen at random is 21,000 lines, 47G. I want to look at all the lines that start with 2011-11-22 4:15:31 ERROR servicename LotsOfText I want to ignore lines that start 2011-11-22 17:13:39 LOG NNNNN servicename LotsOfHTML Ignoring all of those lines would bring it to a manageable size (this particular file is 41 lines, 23k data) I've been playing around with rules in client-local.cfg like: [mmw2.example.com] log:/var/log/mmb1/MMRequest.log:10240 trigger ERROR ignore LOG but I'm just not getting the ERROR lines in the log. Is this file just too large and too full of HTML to parse? Any suggestions? (we can write a custom script, of course, and I'm thinking of bringing in SEC. But it sure would be handy to be able to do this with out of the box xymon) _______________________________________________ Xymon mailing list Xymon at xymon.com
list Jeremy Laidman
On Wed, Nov 23, 2011 at 10:06 AM, Elizabeth Schwartz
▸
<user-c61747246f66@xymon.invalid> wrote:I've got to monitor some very large log files. They're up to a couple
You can use the GNU tail "--retry" and "--follow=name" options to tail a file even through log rotation. Then pipe it into grep (set bufferring to per-line if more than one grep is used) and send the output via "logger" to syslog. If you simply redirect to another file rather than logger/syslog, you have problems when you want to zero/rotate the output file, so using logger/syslog is easier that way. For example, add this into /etc/inittab: tail --follow=MMRequest.log | grep "ERROR servicename LotsOfText" | grep --line-buffered -v "LOG NNNNN servicename LotsOfHTML" | logger -p user2.info -t logwatch Note that some versions of init can't handle inittab lines that are too long, so you'd put it into a script.
▸
but I'm just not getting the ERROR lines in the log. Is this file just too large and too full of HTML to parse? Any suggestions?
Yes, could be. I've had problems using the "log:" stuff with fast-moving logfiles. Also, I've found bugs in the "logfetch" program in some cases of "trigger" and "ignore". Cheers Jeremy