Xymon Mailing List Archive search

Help with very large log file - not getting the right lines

list Xymon User in Richmond
Thu, 24 Nov 2011 07:53:20 -0500
Message-Id: <a74b3eb515831727de73b53373f7d6c8.squirrel@server>

Actually, you can do the "tail" with the default rsyslogd imfile module.
http://www.rsyslog.com/doc/imfile.html
http://www.rsyslog.com/doc/multi_ruleset.html

This was pointed out to me a while back by Irithori on linuxforums.org,
and I had forgotten it.

On Tue, November 22, 2011 20:20, Xymon User in Richmond wrote:
My first thought, also.  I think you can also train syslog-ng to do that,
if you're using syslog-ng.

On Tue, November 22, 2011 18:34, Ralph Mitchell wrote:
First of many "quick fixes": could you set up an auto-restarting script
to do "tail -f logfile | grep ERROR > errorlog"??  Then watch the
aeroflot file.

Ralph Mitchell On Nov 22, 2011 6:07 PM, "Elizabeth Schwartz"
<user-c61747246f66@xymon.invalid> wrote:
I've got to monitor some very large log files. They're up to a couple
gigs a day and individual lines can be 30800 characters or more ,
including HTML. (changing the log file format is a project for another
day)   So my last half hour of one of these files chosen at random is
21,000 lines, 47G.

I want to look at all the lines that start with

2011-11-22 4:15:31 ERROR        servicename LotsOfText

I want to ignore lines that start 2011-11-22 17:13:39 LOG NNNNN
servicename LotsOfHTML

Ignoring all of those lines would  bring it to a manageable size (this
particular file is 41 lines, 23k data)

I've been playing around with rules in client-local.cfg like:
[mmw2.example.com] log:/var/log/mmb1/MMRequest.log:10240 trigger ERROR
ignore LOG

but I'm just not getting the ERROR lines in the log. Is this file just
too large and too full of HTML to parse? Any suggestions?

(we can write a custom script, of course, and I'm thinking of bringing
in SEC. But it sure would be handy to be able to do this with out of
the box xymon) _______________________________________________ Xymon
mailing list Xymon at xymon.com