out-of-inodes hobbit conn error for all 300 systems 🔗 link

Hi All

I am running solaris 9. My hobbit is running off of /export/home/hobbit
dir. The `df -i` shows

(root)@hobbit:/export/home/hobbit/data$ /usr/local/bin/df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/md/dsk/d0        494592    7665  486927    2% /
/dev/md/dsk/d40       504000   63200  440800   13% /usr
/dev/md/dsk/d20       494592    8052  486540    2% /var
swap                   98024     112   97912    1% /var/run
swap                   98024     112   97912    1% /tmp
/dev/md/dsk/d50       504000   16322  487678    4% /opt
/dev/md/dsk/d30      1005696 1005005     691  100% /export/home

But still have enough disk space per `df -k`

(root)@bigbrother:/export/home/hobbit/data$ /usr/local/bin/df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/md/dsk/d0          986131    114371    812593  13% /
/dev/md/dsk/d40        2033103   1281660    690450  65% /usr
/dev/md/dsk/d20         986131    153015    773949  17% /var
swap                   2705592        24   2705568   1% /var/run
swap                   2706504       936   2705568   1% /tmp
/dev/md/dsk/d50        2033103    850773   1121337  44% /opt
/dev/md/dsk/d30        8261393   4437625   3741155  55% /export/home

Wish I had the filesystem built optimized for lots of inodes with small
files instead. But it is too late. As far as resources on this system I
also have 674M actual and 2642M swap memory left.

Either I think of using these resources, like may be creating some RAM
disks (Henrik had a tip on that, cant find it) or get rid of `hist'
folder which has about 4000 files in it. I really like to keep the
`hist' files for report and going back to check for old events.

Anyone can think of any suggestion to overcome this predicament?

Thanks


-- 
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
"..there are two kinds of people: those who work and those who take the credit...try
 to be in the first group;...less competition there."  - Indira Gandhi

On Fri, Dec 16, 2005 at 10:11:18PM -0500, Asif Iqbal wrote:

[snip]

Keep the "hist" folder - there's only one file per status you have in
Hobbit, and there won't be more.

What you should have a look at is the "histlogs" folder. This holds a
copy of each status log whenever a status change occurred - and it gets
*really* large over time. These historical status logs are only used
when you look at the history of a status, and click on the red button
off the "Jun 16 2003 14:21:58" event to see the detailed status log.

What I do on my production systems is to archive the history logs once
a month, and then deleting those history logs that are older than 3 
months. So you can get the full, detailed status of any event that
happened within the past 3 months; if you need to go further back than
that, it is still visible in the history view, but you'll need to
extract the detailed status log from the tar file to see all of the
details.

Feel free to use this snippet from my monthly cron job - it requires
the GNU date utility. Right now, you will of course need to clean up
your histlogs directory by hand, the snippet below only processes
the histlog-files one month at a time.


cd /var/lib/hobbit/data

# Archive histlogs generated in the past month
FNPATTERN=`date --date="today -1 month" +"???_%b_*_%Y"`
ARCFN=`date --date="today -1 month" +"histlogs_%Y_%m.cpio.gz"`
find histlogs -name "$FNPATTERN" | cpio -o --format=newc | gzip >archive/$ARCFN

# Delete histlogs from 4 months ago (so we always have the last 3 months online)
FNPATTERN=`date --date="today -4 months" +"???_%b_*_%Y"`
find histlogs -name "$FNPATTERN" | xargs rm


Regards,
Henrik

On Sat, Dec 17, 2005 at 08:43:31AM, Henrik Stoerner wrote:

Excellent. I am slowly moving old logs. I wonder if that will skew my
monthly and yearly reports.

snippet of the yearly report that gets called by bbcmd.

YEAR=$1
DATE=/usr/local/bin/date
MKDIR=/usr/local/bin/mkdir
YEAR=${YEAR:=`$DATE +%Y`}
START=`$DATE +%s --date="01 Jan $YEAR 00:00:00"`
END=`$DATE +%s --date="31 Dec $YEAR 23:59:59"`
$MKDIR -p /usr/local/apache/htdocs/reports/yearly/$YEAR
BBWEB=/reports/yearly/$YEAR \
$BBHOME/bin/bbgen --reportopts=$START:$END:1:crit \
        --ignorecolumns=ssh,bbgen,bbtest,hobbitd,info,trends,msgs,scanmsg,http
--subpagecolumns=2 \
        /usr/local/apache/htdocs/reports/yearly/$YEAR


I noticed I have about 70000 files under histlogs. Wow!!. I guess would
be nice to use database instead to store histlogs and make report still
work. I know there are suggestion out there regarding mysql use to store
logs. 

(hobbit)@bigbrother:~/data$ find histlogs -name "???_Nov_*_2005" | wc -l
   67912

Do you have any plan to have a DB plugin for that in the future? Or may
be a suggestion, on storing log in database, that you think best work
with hobbit?

Thanks

Regards,
Henrik

-- 
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
"..there are two kinds of people: those who work and those who take the credit...try
 to be in the first group;...less competition there."  - Indira Gandhi

On Sat, Dec 17, 2005 at 08:43:31AM, Henrik Stoerner wrote:

On Fri, Dec 16, 2005 at 10:11:18PM -0500, Asif Iqbal wrote:

Hi All

I am running solaris 9. My hobbit is running off of /export/home/hobbit
dir. The `df -i` shows
/dev/md/dsk/d30      1005696 1005005     691  100% /export/home

[snip]

Either I think of using these resources, like may be creating some RAM
disks (Henrik had a tip on that, cant find it) or get rid of `hist'
folder which has about 4000 files in it. I really like to keep the
`hist' files for report and going back to check for old events.

Keep the "hist" folder - there's only one file per status you have in
Hobbit, and there won't be more.

[...]

What I do on my production systems is to archive the history logs once
a month, and then deleting those history logs that are older than 3 
months. So you can get the full, detailed status of any event that
happened within the past 3 months; if you need to go further back than
that, it is still visible in the history view, but you'll need to
extract the detailed status log from the tar file to see all of the
details.

Feel free to use this snippet from my monthly cron job - it requires
the GNU date utility. Right now, you will of course need to clean up
your histlogs directory by hand, the snippet below only processes
the histlog-files one month at a time.


cd /var/lib/hobbit/data

# Archive histlogs generated in the past month
FNPATTERN=`date --date="today -1 month" +"???_%b_*_%Y"`
ARCFN=`date --date="today -1 month" +"histlogs_%Y_%m.cpio.gz"`
find histlogs -name "$FNPATTERN" | cpio -o --format=newc | gzip >archive/$ARCFN

# Delete histlogs from 4 months ago (so we always have the last 3 months online)
FNPATTERN=`date --date="today -4 months" +"???_%b_*_%Y"`
find histlogs -name "$FNPATTERN" | xargs rm

Excellent. I am slowly moving old logs. I wonder if that will skew my
monthly and yearly reports.

snippet of the yearly report that gets called by bbcmd.

YEAR=$1
DATE=/usr/local/bin/date
MKDIR=/usr/local/bin/mkdir
YEAR=${YEAR:=`$DATE +%Y`}
START=`$DATE +%s --date="01 Jan $YEAR 00:00:00"`
END=`$DATE +%s --date="31 Dec $YEAR 23:59:59"`
$MKDIR -p /usr/local/apache/htdocs/reports/yearly/$YEAR
BBWEB=/reports/yearly/$YEAR \
$BBHOME/bin/bbgen --reportopts=$START:$END:1:crit \
        --ignorecolumns=ssh,bbgen,bbtest,hobbitd,info,trends,msgs,scanmsg,http
--subpagecolumns=2 \
        /usr/local/apache/htdocs/reports/yearly/$YEAR


I noticed I have about 70000 files under histlogs. Wow!!. I guess would
be nice to use database instead to store histlogs and make report still
work. I know there are suggestion out there regarding mysql use to store
logs.

(hobbit)@bigbrother:~/data$ find histlogs -name "???_Nov_*_2005" | wc -l
   67912

Do you have any plan to have a DB plugin for that in the future? Or may
be a suggestion, on storing log in database, that you think best work
with hobbit?


-- 
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
"..there are two kinds of people: those who work and those who take the credit...try
 to be in the first group;...less competition there."  - Indira Gandhi

On Sat, Dec 17, 2005 at 11:31:42AM -0500, Asif Iqbal wrote:

It should not affect those. The reports and their percentages are
generated exclusively from the contents of the hist/* files, and just 
links to the histlogs files for the detailed status.

I dont have any immediate plans for this, but it would be a nice little
project if someone wants to dip into the Hobbit code (hint!). You could
easily turn off saving the file-based status logs (there's a
hobbitserver.cfg setting for that), and a module for storing the status
logs in a DB would be fairly trivial to implement as a Hobbit worker
module hanging off the "stachg" (status change) channel. Apart from
that, there's a simple change to pick the status log from the DB when
viewing it, and that's all.


Regards,
Henrik

Ahhh the BB/Hobbit, "How about a DB backend question ;)"

I've always been amazed that BB and Hobbit could use a filesystem as  a database and not kill a server much *sooner*.  hobbits use of  memory for bbvar/logs is a huge win . . . .

Ideally the backend for most, if not all, of the state/history  information for events would be a database.  Problem is, that  introduces a dependency of the product (hobbit) an another tool  (Database).  This can very easily become a nightmare from a support/ integration point of view.  Not to mention you raise the bar for  entry level admins ability to 'tinker' with the product.

Unless the entire hobbit tool were re-architected to have DB back- end, I don't think it would be worth the overall effort for keeping  the histlogs area small.

That's *not* to say it isn't a good idea or it *wouldn't* be a great  coding project for someone.

But I think it would be a better use of resources and benefit more of  the user community if hobbit could internally, archive and retrieve  this data.  The history data is *extremely* useful and having it  around is a wonderful asset for problem history/determination.

Henrik, I was thinking along the lines of every month or so archive  the histlogs into a single large file, or one file per host, and then  introduce logic into the show history that could determine if it  needed to expand the archive to get it.

scott

On Sun, Dec 18, 2005 at 10:28:17AM -0500, Scott Walters wrote:

That in itself would not gain you much, I agree. There has to be some
other benefit besides the space-win to justify moving to a DB backend.
E.g. if there were some fancy backend pulling data from Hobbit which
could benefit from having the Hobbit data in a DB.

That's *not* to say it isn't a good idea or it *wouldn't* be a great  coding project for someone.

The nice thing about it is that this can be implemented as a separate
module, without affecting the core Hobbit programs at all. So if someone
desperately needs a DB backend for the Hobbit history-data, I'm sure
that they will figure out how to write one and offer me the sources for
inclusion :-)

That idea had crossed my mind. The archiving bit is something I already
do with a fairly simple shell script once a month; this can easily be
turned into a general solution. Adding a bit of code to try and extract
the history logfile from an archive if the file is not immediately
available is also rather trivial. So maybe I'll try and do it.


Henrik

On Mon, 2005-12-19 at 00:08 +0100, Henrik Stoerner wrote:

Well, the other option is to simply use a filesystem that doesn't incur
a penalty for having large numbers of files in a directory, or on the
filesystem. My favourite is of course ReiserFS, where multiple files can
share the same block (no space wastage from allocating whole blocks to a
single file), as well as no limited inodes for the filesystem etc...

On the other hand, for the interested individual, I've been told that
there is some way to basically embed mysql into a product, thereby
allowing you to 'contain' the entire system, rather than relying on the
administrator to configure mysql/etc properly. In fact, this might be
helpful initially, and for the larger sites, then they would simply
reconfig hobbit to point to a external mysql server that they can
customise themselves.

Regards,
Adam

On Sat, Dec 17, 2005 at 08:43:31AM, Henrik Stoerner wrote:

My manager wants to see some old data for one of the host. What is the
best way to restore it? I tried to restore one of the archive file and
it does not expand the same way as active histlogs dir and hence finding
data about a host seems difficult.

Any suggestion on this?

Thanks

-- 
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
"..there are two kinds of people: those who work and those who take the credit...try
 to be in the first group;...less competition there."  - Indira Gandhi