OK... I enabled the modules one at a time until I reached
'larrdstatus'. When I enabled that module, hobbitd began crashing
again. Here is the contents of the hobbitlaunch.cfg.
#
# The hobbittasks.cfg file is loaded by "hobbitlaunch".
# It controls which of the Hobbit modules to run, how often, and #
with which parameters, options and environment variables. #
# This is the main Hobbit daemon. You cannot live without this one. [hobbitd]
HEARTBEAT
ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
CMD hobbitd --debug --restart=$BBTMP/hobbitd.chk
--checkpoint-file=$BBTMP/hobbitd.chk --checkpoint-interval=600
--log=$BBSERVERLOGS/hobbitd.log --admin-senders=127.0.0.1,$BBSERVERIP
# "bbstatus" saves status-logs in text- and html-format, like the old
Big Brother
# daemon does. Unless you are using add-ons that directly access the
log-files, you
# will not need to run this module, and it is recommended that you
keep it disabled # since storing the raw logs on disk can cause a
significant load on your server.
[bbstatus]
DISABLED
ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
NEEDS hobbitd
CMD hobbitd_channel --channel=status --log=$BBSERVERLOGS/status.log
hobbitd_filestore --status --html
# "bbhistory" keeps track of the status changes that happen, in a
manner that is # compatible with the Big Brother history logs. You
probably do want to run this.
[bbhistory]
ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
NEEDS hobbitd
CMD hobbitd_channel --channel=stachg --log=$BBSERVERLOGS/history.log
hobbitd_history
# "bbdata" saves information sent using the BB "data" protocol, like
the old Big Brother
# daemon does. Unless you are using add-ons that directly access the
data-files, you
# will not need to run this module, and it is recommended that you
keep it disabled # since storing the raw data on disk can cause a
significant load on your BB server. # LARRD uses data-files, but since
LARRD is handled by the hobbitd_larrd module, you do # not need to run
the "bbdata" module to get LARRD graphs.
[bbdata]
DISABLED
ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
NEEDS hobbitd
CMD hobbitd_channel --channel=data --log=$BBSERVERLOGS/data.log
hobbitd_filestore --data
# "bbnotes" saves web note-files, that are sent using the BB "notes"
protocol. This is # disabled by default; if you use the BB "notes"
protocol, then you should enable this.
[bbnotes]
DISABLED
ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
NEEDS hobbitd
CMD hobbitd_channel --channel=notes --log=$BBSERVERLOGS/notes.log
hobbitd_filestore --notes
# "bbenadis" updates the files used to indicate that a host or test
has been enabled or disabled. # These files are used by bbgen and the
"maint.pl" script to determine what is currently enabled # and
disabled, so you probably want to run this module.
[bbenadis]
ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
NEEDS hobbitd
CMD hobbitd_channel --channel=enadis --log=$BBSERVERLOGS/enadis.log
hobbitd_filestore --enadis
# "bbpage" sends out alerts. Note that this module is NOT compatible
with the old Big Brother # system - it uses a different configuration
file to determine how alerts get sent. If you want # alerts to go out
via pager, e-mail or some other means, then you must run this module.
[bbpage]
ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
NEEDS hobbitd
CMD hobbitd_channel --channel=page --log=$BBSERVERLOGS/page.log hobbitd_alert
# "larrdstatus" updates RRD files with information that arrives as
"status" messages. # If you want RRD graphs of your monitoring data,
then you want to run this.
[larrdstatus]
ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
NEEDS hobbitd
CMD hobbitd_channel --channel=status
--log=$BBSERVERLOGS/larrd-status.log hobbitd_larrd --rrddir=$BBVAR/rrd
# "larrddata" updates RRD files with information that arrives as
"data" messages. # If you want RRD graphs of your monitoring BB data,
then you want to run this.
[larrddata]
DISABLED
ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
NEEDS hobbitd
CMD hobbitd_channel --channel=data
--log=$BBSERVERLOGS/larrd-data.log hobbitd_larrd --rrddir=$BBVAR/rrd
# "bbdisplay" runs the bbgen tool to generate the Hobbit webpages from
the status information that # has been received. Big Brother updated
the webpages once every 5 minutes. The default here is to # run it
every minute for faster updates, but you can change it if you have a
highly loaded server # and dont need updates that often.
[bbdisplay]
ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
NEEDS hobbitd
GROUP generators
CMD bb-display.sh
LOGFILE $BBSERVERLOGS/bb-display.log
INTERVAL 1m
# "larrdcolumn" is responsible for updating the contents of the LARRD
overview page, found on the # "trends" column for each host. Since the
set of graphs does not change very often, we run this # less
frequently than the normal webpage updates. We also make sure (with
the "GROUP" setting) # that they don't run simultaneously with the
infocolumnn and bbdisplay tasks. [larrdcolumn]
DISABLED
ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
NEEDS hobbitd
GROUP generators
CMD bb-larrdcolumn --hobbitd --rrddir=$BBVAR/rrd --column=trends
LOGFILE $BBSERVERLOGS/bb-display.log
INTERVAL 15m
# "infocolumn" is responsible for updating the contents of the INFO
pages, found on the # "info" column for each host. Since the content
does not change unless there is a
# configuration change, we update these less frequently than the
normal webpage updates. # We also make sure (with the "GROUP" setting)
that they don't run simultaneously with
# the larrdcolumnn and bbdisplay tasks.
[infocolumn]
DISABLED
ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
NEEDS hobbitd
GROUP generators
CMD bb-infocolumn --hobbitd --column=info
LOGFILE $BBSERVERLOGS/bb-display.log
INTERVAL 15m
# "bbnet" runs the bbtest-net tool to perform the network based tests
- i.e. http, smtp, ssh, dns and # all of the various network protocols
we need to test.
[bbnet]
ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
NEEDS hobbitd
CMD bbtest-net --report --ping --checkresponse
LOGFILE $BBSERVERLOGS/bb-network.log
INTERVAL 5m
# "bbretest" picks up the tests that the normal network test consider
"failed", and re-does those # tests more often. This enables Big
Brother to pick up a recovered network service faster than # if it
were tested only by the "bbnet" task (which only runs every 5
minutes). So if you have # servers with very high availability
guarantees, running this task will make your availability # reports
look much better.
[bbretest]
ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
NEEDS hobbitd
CMD $BBHOME/ext/bbretest-net.sh
LOGFILE $BBSERVERLOGS/bb-retest.log
INTERVAL 1m
On Tue, 1 Mar 2005 10:10:12 -0800, Brian Lynch <user-0420823115a8@xymon.invalid> wrote:
Henrik,
Tried the patch and hobbitd still crashed.
Tried the disable module test. Disabled all the modules you mentioned
above and hobbitd has not crashed since 8:30 PST this morning. I'll
try adding back in modules until it crashes.
- Brian
On Tue, 1 Mar 2005 09:33:51 +0100, Henrik Stoerner <user-ce4a2c883f75@xymon.invalid> wrote:
I've found something that might explain it, but I am certainly not
sure if it's the cause of your problems. This bug would only trigger
if some client of yours was sending in very large "data" messages,
like 100 KB or more.
Could you try the attached patch and let me know if it solves your
problem?
If not, I would like you to try and disable all of the hobbitd worker
modules in hobbitlaunch.cfg - i.e. bbstatus, bbhistory, bbdata,
bbnotes, bbenadies, bbpage, larrdstatus and larrddata. Restart hobbit
and see if it still crashes. If it doesn't then we've narrowed down
the problem - if it does, then the problem is somewhere else than
where I've been looking so far.
Thanks,
Henrik