Hobbit setup after Catastrophic failure
list David Gilmore
I recently had a catastrophic failure of my hobbit box. I have rebuilt the box, installed FC6, installed hobbit, and have restored hobbit files from backup. I have started the hobbit script and restarted it after fixing some permissions issues caused by the restore, mostly missing rwx on subfolders and files. Currently Apache is serving the pages, but the data is stale and does not appear to be updating. Any suggestions? I have been looking at this for about two hours and my brain is numb. Thank you, David This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.
list Rob MacGregor
▸
On 10/1/07, David Gilmore <user-70507ff7198d@xymon.invalid> wrote:
I recently had a catastrophic failure of my hobbit box. I have rebuilt the box, installed FC6, installed hobbit, and have restored hobbit files from backup. I have started the hobbit script and restarted it after fixing some permissions issues caused by the restore, mostly missing rwx on subfolders and files. Currently Apache is serving the pages, but the data is stale and does not appear to be updating. Any suggestions? I have been looking at this for about two hours and my brain is numb.
Off the top of my head:
1) Hobbit really is running
2) Apache is configured with an appropriate redirect for the location
of the hobbit pages
3) The hobbit user can write to the location of the hobbit pages
4) What do the hobbit logs say?
--
Please keep list traffic on the list.
Rob MacGregor
Whoever fights monsters should see to it that in the process he
doesn't become a monster. Friedrich Nietzsche
list David Gilmore
Rob,
▸
I recently had a catastrophic failure of my hobbit box. I have rebuilt the box, installed FC6, installed hobbit, and have restored hobbit files from backup. I have started the hobbit script and restarted it after fixing some permissions issues caused by the restore, mostly missing rwx on subfolders and files. Currently Apache is serving the pages, but the data is stale and does not appear to be updating. Any suggestions? I have been looking at this for about two hours and my brain is numb.
Off the top of my head: 1) Hobbit really is running
Apparently not. Hobbitlaunch is listed as a sleeping process, but no hobbit is listed.
▸
2) Apache is configured with an appropriate redirect for the location of the hobbit pages
I am pretty sure I have this part correct.
3) The hobbit user can write to the location of the hobbit pages
I believe I made those changes. From what I can see the hobbit user has rwx on all files including those in the log files.
4) What do the hobbit logs say?
None of the log files are being updated.
▸
--
Please keep list traffic on the list.
Rob MacGregor
Whoever fights monsters should see to it that in the process he
doesn't become a monster. Friedrich Nietzsche
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.
list Trent Melcher
▸
-----Original Message----- From: David Gilmore [mailto:user-70507ff7198d@xymon.invalid] Sent: Monday, October 01, 2007 9:42 AM To: user-ae9b8668bcde@xymon.invalid Subject: RE: [hobbit] Hobbit setup after Catastrophic failure Rob,
I recently had a catastrophic failure of my hobbit box. I have rebuilt the box, installed FC6, installed hobbit, and have restored hobbit files from backup. I have started the hobbit script and restarted it after fixing some permissions issues caused by the restore, mostly missing rwx on subfolders and files. Currently Apache is serving the pages, but the data is stale and does not appear to be updating. Any suggestions? I have been looking at this for about two hours and my brain is numb.
Off the top of my head: 1) Hobbit really is running
Apparently not. Hobbitlaunch is listed as a sleeping process, but no hobbit is listed.
2) Apache is configured with an appropriate redirect for the location of the hobbit pages
I am pretty sure I have this part correct.
3) The hobbit user can write to the location of the hobbit pages
I believe I made those changes. From what I can see the hobbit user has rwx on all files including those in the log files.
Verify that after your restore that the index.html pages are actually
linked to the true files and not files themselves. I have had this
happen and the real files were being updated but since index.html were
also files my data looked stale.
Trent
▸
4) What do the hobbit logs say?
None of the log files are being updated.
--
Please keep list traffic on the list.
Rob MacGregor
Whoever fights monsters should see to it that in the process he
doesn't become a monster. Friedrich Nietzsche
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this email in error please notify the
system manager. This message contains confidential information and is
intended only for the individual named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail.
list Rob MacGregor
▸
On 10/1/07, David Gilmore <user-70507ff7198d@xymon.invalid> wrote:
1) Hobbit really is runningApparently not. Hobbitlaunch is listed as a sleeping process, but no hobbit is listed.
Well, that's probably a good starting point then :) Can you manually start hobbit (using hobbit.sh)?
▸
--
Please keep list traffic on the list.
Rob MacGregor
Whoever fights monsters should see to it that in the process he
doesn't become a monster. Friedrich Nietzsche
list David Gilmore
That is how I have been launching it, using hobbit.sh, which happily replies that it stops, starts, restarts, etc... I think at this point I am going to redo the configure/make/make install just to get a working installation again. Then I will restore the config one piece at a time as my bulk restore seems to have had issues. David
▸
-----Original Message-----
From: Rob MacGregor [mailto:user-07c9d92ae079@xymon.invalid]
Sent: Monday, October 01, 2007 11:02 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Hobbit setup after Catastrophic failure
On 10/1/07, David Gilmore <user-70507ff7198d@xymon.invalid> wrote:1) Hobbit really is runningApparently not. Hobbitlaunch is listed as a sleeping process, but no hobbit is listed.
Well, that's probably a good starting point then :)
Can you manually start hobbit (using hobbit.sh)?
--
Please keep list traffic on the list.
Rob MacGregor
Whoever fights monsters should see to it that in the process he
doesn't become a monster. Friedrich Nietzsche
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.
list Henrik Størner
▸
On Mon, Oct 01, 2007 at 10:05:54AM -0400, David Gilmore wrote:
I recently had a catastrophic failure of my hobbit box. I have rebuilt the box, installed FC6, installed hobbit, and have restored hobbit files from backup. I have started the hobbit script and restarted it after fixing some permissions issues caused by the restore, mostly missing rwx on subfolders and files. Currently Apache is serving the pages, but the data is stale and does not appear to be updating. Any suggestions?
Try this first (while logged in as the Hobbit user): $ bbcmd 2007-10-01 17:13:09 Using default environment file /usr/lib/hobbit/server/etc/hobbitserver.cfg $ echo $BBHOME /usr/lib/hobbit/server $ touch $BBTMP/test.tmp $ touch $BBSERVERLOGS/test.log $ touch $BBWWW/test.html This uses "bbcmd" to check that your hobbitserver.cfg settings can be loaded, and also that you have write access to a couple of directories. There should be no errors. Next, try to start hobbitd: $ ./server/bin/hobbitd 2007-10-01 17:15:14 Loading hostnames 2007-10-01 17:15:14 Setting up network listener on 0.0.0.0:1984 2007-10-01 17:15:14 Setting up local listener 2007-10-01 17:15:14 Setting up signal handlers 2007-10-01 17:15:14 Setting up hobbitd channels 2007-10-01 17:15:14 Setting up logfiles 2007-10-01 17:15:14 Setup complete This is the normal startup log from hobbitd. Again, there should be no errors. If this succeeds, then you can just ctrl-c out of it. Next, try to start hobbitd through hobbitlaunch: $ ./server/bin/hobbitlaunch --no-daemon --config=/etc/hobbit/hobbitlaunch.cfg 2007-10-01 17:18:50 hobbitlaunch starting 2007-10-01 17:18:50 Loading tasklist configuration from /etc/hobbit/hobbitlaunch.cfg 2007-10-01 17:18:50 Loading hostnames 2007-10-01 17:18:50 Loading saved state 2007-10-01 17:18:50 Setting up network listener on 172.16.10.100:1984 2007-10-01 17:18:50 Setting up local listener 2007-10-01 17:18:50 Setting up signal handlers 2007-10-01 17:18:50 Setting up hobbitd channels 2007-10-01 17:18:50 Setting up logfiles If you get this far, then Hobbit is basically running, and any problems should be reported in the logfiles. You can try running the command "bb 127.0.0.1 hobbitdboard" to check that the Hobbit daemon is responding to queries, it should dump the full state of everything monitored. Regards, Henrik
list David Gilmore
Henrik, Thank you, I was just about to Trash the install. See below for results
▸
Try this first (while logged in as the Hobbit user):
$ bbcmd
2007-10-01 17:13:09 Using default environment file /usr/lib/hobbit/server/etc/hobbitserver.cfg
no errors, but output states '/home/hobbit/server/etc/hobbitserver.cfg'
$ echo $BBHOME /usr/lib/hobbit/server
got the above response, which is apparently my problem. It should read '/home/hobbit/server'. I found the line and corrected in hobbitsever.cfg
Next, try to start hobbitd:
hobbit starts normal
Next, try to start hobbitd through hobbitlaunch:
fails to start with error that it cannot open the configuration file /etc/hobbitlaunch.cfg: no such file or directory.
I'll try finding this one while I wait to hear from the board.
Thank you, Dave
▸
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.
list David Gilmore
Update to last post. After reading Henrik's response again I realized I attempted hobbitlaunch without the specified commands so of course it failed. When I did enter it in exactly it still failed, so I changed to: ./hobbitlaunch --no-daemon --config=/home/hobbit/server/etc/hobbitlaunch.cfg It start to launch, and then I get 'Aborted (core dumped)' 'Could not connect to bbd at 10.254.254.12:1984 - Connection refused' 'Whoops ! bb failed to send message - Connection failed' David
▸
-----Original Message-----
From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: Monday, October 01, 2007 11:22 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Hobbit setup after Catastrophic failure
On Mon, Oct 01, 2007 at 10:05:54AM -0400, David Gilmore wrote:I recently had a catastrophic failure of my hobbit box. I have rebuilt the box, installed FC6, installed hobbit, and have restored hobbit files from backup. I have started the hobbit script and restarted it after fixing some permissions issues caused by the restore, mostly missing rwx on subfolders and files. Currently Apache is serving the pages, but the data is stale and does not appear to be updating. Any suggestions?
Try this first (while logged in as the Hobbit user): $ bbcmd 2007-10-01 17:13:09 Using default environment file /usr/lib/hobbit/server/etc/hobbitserver.cfg $ echo $BBHOME /usr/lib/hobbit/server $ touch $BBTMP/test.tmp $ touch $BBSERVERLOGS/test.log $ touch $BBWWW/test.html This uses "bbcmd" to check that your hobbitserver.cfg settings can be loaded, and also that you have write access to a couple of directories. There should be no errors. Next, try to start hobbitd: $ ./server/bin/hobbitd 2007-10-01 17:15:14 Loading hostnames 2007-10-01 17:15:14 Setting up network listener on 0.0.0.0:1984 2007-10-01 17:15:14 Setting up local listener 2007-10-01 17:15:14 Setting up signal handlers 2007-10-01 17:15:14 Setting up hobbitd channels 2007-10-01 17:15:14 Setting up logfiles 2007-10-01 17:15:14 Setup complete This is the normal startup log from hobbitd. Again, there should be no errors. If this succeeds, then you can just ctrl-c out of it. Next, try to start hobbitd through hobbitlaunch: $ ./server/bin/hobbitlaunch --no-daemon --config=/etc/hobbit/hobbitlaunch.cfg 2007-10-01 17:18:50 hobbitlaunch starting 2007-10-01 17:18:50 Loading tasklist configuration from /etc/hobbit/hobbitlaunch.cfg 2007-10-01 17:18:50 Loading hostnames 2007-10-01 17:18:50 Loading saved state 2007-10-01 17:18:50 Setting up network listener on 172.16.10.100:1984 2007-10-01 17:18:50 Setting up local listener 2007-10-01 17:18:50 Setting up signal handlers 2007-10-01 17:18:50 Setting up hobbitd channels 2007-10-01 17:18:50 Setting up logfiles If you get this far, then Hobbit is basically running, and any problems should be reported in the logfiles. You can try running the command "bb 127.0.0.1 hobbitdboard" to check that the Hobbit daemon is responding to queries, it should dump the full state of everything monitored. Regards, Henrik This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.
list Henrik Størner
▸
On Mon, Oct 01, 2007 at 12:32:23PM -0400, David Gilmore wrote:
After reading Henrik's response again I realized I attempted hobbitlaunch without the specified commands so of course it failed. When I did enter it in exactly it still failed, so I changed to: ./hobbitlaunch --no-daemon --config=/home/hobbit/server/etc/hobbitlaunch.cfg It start to launch, and then I get 'Aborted (core dumped)' 'Could not connect to bbd at 10.254.254.12:1984 - Connection refused' 'Whoops ! bb failed to send message - Connection failed'
Try deleting the "*.chk" files in the Hobbit "tmp" directory. If these have been trashed, it can cause that. Regards, henrik
list David Gilmore
Henrik, Thank you for the last piece of advice. My attempts to fix it were not so successful. I basically trashed the hobbit install, started clean, and then brought in the restored info piece by piece. It is up and running now and I am praying that I never see two failed drives on a RAID 5 array ever again. Thank you,
▸
David
-----Original Message-----
From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: Monday, October 01, 2007 4:06 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Hobbit setup after Catastrophic failure
On Mon, Oct 01, 2007 at 12:32:23PM -0400, David Gilmore wrote:After reading Henrik's response again I realized I attempted hobbitlaunch without the specified commands so of course it failed. When I did enter it in exactly it still failed, so I changed to: ./hobbitlaunch --no-daemon --config=/home/hobbit/server/etc/hobbitlaunch.cfg It start to launch, and then I get 'Aborted (core dumped)' 'Could not connect to bbd at 10.254.254.12:1984 - Connection refused' 'Whoops ! bb failed to send message - Connection failed'
Try deleting the "*.chk" files in the Hobbit "tmp" directory. If these have been trashed, it can cause that. Regards, henrik This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.