trimhistory problems 🔗 link

Hello,

I am trying to do some hobbit house keeping by using the trimhistory
tool, however it is not working as expected.

I have run the following command and expected no action to be taken as
the server was only built in November 2007

[root at xxx01x $] /var/hobbit/server/bin/trimhistory --debug
--env=/var/hobbit/server/etchobbitserver.cfg --cutoff=`date +%s
--date="1 Oct 2004"`

However it looks to be trimming logs from many hosts:
2008-01-09 15:30:31 Processing xxx04x.conn
2008-01-09 15:30:31 Processing xxx01x.conn
2008-01-09 15:30:31 Processing xxx01x.conn
2008-01-09 15:30:31 Processing xxx01x.conn
2008-01-09 15:30:31 Processing xxx03x.conn
..........

When i then look at the conn tests on the above servers, i have
different lengths of history left, days or hours, so not consistent for
each one.

It is as if the --cutoff option is not working, does this have to be the
1st command after trimhistory?

Server in question has correct date set
[root at xxx01x etc]# date
Wed Jan  9 17:03:50 GMT 2008

looking at another server that it has yet to get too shows that there
are no logs older than Nov 2007.

Any pointers would be appreciated running the latest stable version of
hobbit 4.2.0.

Thanks

Dominic


 Protected by Websense Messaging Security ? www.websense.com

Henrik,

Looking at trimhistory.c in the trim_history function then the ALLEVENTS
case section, i am confused as to why you use column3 and not 2, and
think that this may be a mistake, but not sure, do you mind clarifying.

trimhistory.c line 97:
 switch (ftype) {
                          case F_HOSTHISTORY:
                                copying = (!cols[1] || (atoi(cols[1]) >=
cutoff));
                                break;

                          case F_SERVICEHISTORY:
                                copying = (!cols[6] || (atoi(cols[6]) >=
cutoff));
                                break;

                          case F_ALLEVENTS:
                                copying = (!cols[3] || (atoi(cols[3]) >=
cutoff));
                                break;

                          case F_DROPIT:
                          case F_PURGELOGS:
                                /* Cannot happen */
                                errprintf("Impossible -
F_DROPIT/F_PURGELOGS in trim_history\n");
                                return;


when processing the ALLEVENTS file you reference column3 which is an
epoch time stamp, 
The ALLEVENTS file looks to be sorted by column 2 and not 3, however the
logic in the above case statement means that when processing the
ALLEVENTS file you may delete too much/little data as the file is not
sorted by column 3.

Should the above ALLEVENTS case statement not reference column2 as
below?

case F_ALLEVENTS:
	copying = (!cols[2] || (atoi(cols[2]) >= cutoff));
break;

see below extract from ALLEVENTS file:

xxx01e procs 1195058658 1195058435 223 gr cl 1
yyy01e hobbitd 1195058721 1195058721 0 gr - -1
zzz01e procs 1195058736 1195058658 78 cl gr 2
ffff-m conn 1195058821 1195058447 374 gr re 1

Thanks

Regards

Dominic

On Wed, 2008-01-09 at 17:10 +0000, Dominic Young wrote:

 TO REPORT THIS AS SPAM, PLEASE CLICK THE FOLLOWING LINK: 
https://www.mailcontrol.com/sr/uUL+OVS8ZIUjSBxHehtsm8Zce65ll4171PONrb+jPjSXBzIi4wtGOZm9H!6!Zn!Ty2wHP1XV4nYsJVkF713lqa+p9h3Y2GqJUkdWTp!VIyuuYiC1uet413O7Vh!noKs7GHqkM8DzHcy2fn4tJVDr2o6S0IpDD2tcDw2xApNJ61d0t7DnahD6SKdSydLTt!HN+5MoUj!2fLcnMG!PXock1E9u!n4UIOoj  

-- 
Dominic Young
Infrastructure Operations Engineer

WEBSENSE, INC.
+44 (0)XXX XXX XXXX Support
+44 (0)XXX XXX XXXX Fax
www.websense.com

INTELLIGENT CONTENT PROTECTION
WHEN AND WHERE YOU NEED IT

Henrik,

Have been looking at trimhistory.c, and a bit confused about the
ALLEVENTS case statement in trim_history (line 121 ish).

case F_ALLEVENTS:
	copying = (!cols[3] || (atoi(cols[3]) >= cutoff));
break;

The ALLEVENTS file is ordered by column 2 (epoch time), however this
function goes through the file and as soon as epoch time in clolumn3 is

= to cutoff the copying beings, and all entries before it are not
copied.

However as the file is not ordered by column3 but column2, the trim that
is carried out is incorrect. It may trim more or less than expected.
Maybe my allevents file is corrupted or not correctly built?

my example ALLEVETNS file:

a1e procs 1195058658 1195058435 223 gr cl 1
b1e hobbitd 1195058721 1195058721 0 gr - -1
c1e procs 1195058736 1195058658 78 cl gr 2
d-m conn 1195058821 1195058447 374 gr re 1
e-m smtp 1195058821 1195058447 374 ye cl 2
f1e msgs 1195058957 1195058658 299 ye gr 2
g1e procs 1195058957 1195058736 221 gr cl 1
h1e procs 1195059037 1195058957 80 cl gr 2

so if cutofftime was 1195058447, all the lines below line 3 above would
stay, but you loose the 1st 3 lines, however lines 2 and 3 are older
than 4 and should stay.

I think that the case statement should ref column 2, assuming my
ALLEVENTS file is correct.

case F_ALLEVENTS:
	copying = (!cols[2] || (atoi(cols[2]) >= cutoff));
break;

Do you mind letting me know if this is true or if i have gone mad, or
missed something.

Thanks.

Regards

Dominic

-- 
Dominic Young
Infrastructure Operations Engineer

WEBSENSE, INC.
+44 (0)XXX XXX XXXX Support
+44 (0)XXX XXX XXXX Fax
www.websense.com

INTELLIGENT CONTENT PROTECTION
WHEN AND WHERE YOU NEED IT