Problems starting hobbit
list Shaun Kasperowicz
Well after battling numerous other problems, I finally have what (I think) is a working version of the Hobbit server. Although I think I have everything configured properly, I'm unable to start it. I get the following in hobbitlaunch.log: 2006-06-15 16:05:11 hobbitlaunch starting 2006-06-15 16:05:11 Loading tasklist configuration from /usr/local/hobbit/server/etc/hobbitlaunch.cfg 2006-06-15 16:05:11 Loading hostnames 2006-06-15 16:05:11 Loading saved state 2006-06-15 16:05:11 Setting up network listener on 0.0.0.0:1984 2006-06-15 16:05:11 Setting up signal handlers 2006-06-15 16:05:11 Setting up hobbitd channels 2006-06-15 16:05:11 Could not attach shm Permission denied 2006-06-15 16:05:11 Cannot setup status channel 2006-06-15 16:05:11 Task hobbitd terminated, status 1 I've tweaked the shmseg setting in /etc/system (7 didn't work, I ended up at 10), but from reading the list archives it seems that I may need to tune shmmni and shmmax as well. Any truth to this, or am I barking up the wrong tree? Thanks, -Shaun
list Henrik Størner
▸
On Thu, Jun 15, 2006 at 04:48:37PM -0700, Kasperowicz, Shaun wrote:
Well after battling numerous other problems, I finally have what (I think) is a working version of the Hobbit server. Although I think I have everything configured properly, I'm unable to start it. I get the following in hobbitlaunch.log: 2006-06-15 16:05:11 hobbitlaunch starting 2006-06-15 16:05:11 Loading tasklist configuration from /usr/local/hobbit/server/etc/hobbitlaunch.cfg 2006-06-15 16:05:11 Loading hostnames 2006-06-15 16:05:11 Loading saved state 2006-06-15 16:05:11 Setting up network listener on 0.0.0.0:1984 2006-06-15 16:05:11 Setting up signal handlers 2006-06-15 16:05:11 Setting up hobbitd channels 2006-06-15 16:05:11 Could not attach shm Permission denied 2006-06-15 16:05:11 Cannot setup status channel
The "Permission denied" error is odd, it indicates that this shared memory segment already exists, but was created by another user. If you login as root and run "ipcs -m" you'll get a list of the existing shared memory segments. With hobbit running, it typically looks like this: ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x0102b575 2031619 hobbit 600 262144 2 0x0202b575 2064388 hobbit 600 262144 2 0x0302b575 2097157 hobbit 600 262144 2 0x0402b575 2129926 hobbit 600 262144 2 0x0502b575 2162695 hobbit 600 262144 1 0x0602b575 2195464 hobbit 600 32768 1 0x0702b575 2228233 hobbit 600 524288 2 0x0802b575 2261002 hobbit 600 524288 2 The "bytes" and "nattch" are typical for a default Hobbit setup. If a Hobbit run has crashed, it might leave these behind. In that case, use "ipcrm" to delete them. (Hobbit will do that automatically when restarting, but if they were created by another user then it cannot). Regards, Henrik
list Shaun Kasperowicz
Thanks Henrik, I was able to delete the stale shared memory segments and start hobbit. Still getting some errors in hobbitlaunch.log though: 2006-06-21 10:10:21 hobbitlaunch starting 2006-06-21 10:10:21 Loading tasklist configuration from /usr/local/hobbit/server/etc/hobbitlaunch.cfg 2006-06-21 10:10:21 Loading hostnames 2006-06-21 10:10:21 Loading saved state 2006-06-21 10:10:21 Setting up network listener on 0.0.0.0:1984 2006-06-21 10:10:21 Setting up signal handlers 2006-06-21 10:10:21 Setting up hobbitd channels 2006-06-21 10:10:21 Setting up logfiles 2006-06-21 10:10:26 Task bbhistory terminated, status 1 2006-06-21 10:10:26 Task bbpage terminated, status 1 2006-06-21 10:10:26 Task rrdstatus terminated, status 1 2006-06-21 10:10:26 Task rrddata terminated, status 1 2006-06-21 10:10:27 Task bbdisplay terminated by signal 9 2006-06-21 10:10:27 Task clientdata terminated, status 1 2006-06-21 10:10:27 Task bbcombotest terminated by signal 9 2006-06-21 10:10:32 Task clientdata terminated, status 1 2006-06-21 10:10:32 Task rrddata terminated, status 1 2006-06-21 10:10:32 Task rrdstatus terminated, status 1 2006-06-21 10:10:32 Task bbhistory terminated, status 1 I assume these are not normal? Hobbitd is running, but no web pages are being generated. Thanks, -Shaun
▸
-----Original Message-----
From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: Tuesday, June 20, 2006 11:08 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Problems starting hobbit
On Thu, Jun 15, 2006 at 04:48:37PM -0700, Kasperowicz, Shaun wrote:Well after battling numerous other problems, I finally have what (I think) is a working version of the Hobbit server. Although I think I have everything configured properly, I'm unable to start it. I get the following in hobbitlaunch.log: 2006-06-15 16:05:11 hobbitlaunch starting 2006-06-15 16:05:11 Loading tasklist configuration from /usr/local/hobbit/server/etc/hobbitlaunch.cfg 2006-06-15 16:05:11 Loading hostnames 2006-06-15 16:05:11 Loading saved state 2006-06-15 16:05:11 Setting up network listener on 0.0.0.0:1984 2006-06-15 16:05:11 Setting up signal handlers 2006-06-15 16:05:11 Setting up hobbitd channels 2006-06-15 16:05:11 Could not attach shm Permission denied 2006-06-15 16:05:11 Cannot setup status channel
The "Permission denied" error is odd, it indicates that this shared memory segment already exists, but was created by another user. If you login as root and run "ipcs -m" you'll get a list of the existing shared memory segments. With hobbit running, it typically looks like this: ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x0102b575 2031619 hobbit 600 262144 2 0x0202b575 2064388 hobbit 600 262144 2 0x0302b575 2097157 hobbit 600 262144 2 0x0402b575 2129926 hobbit 600 262144 2 0x0502b575 2162695 hobbit 600 262144 1 0x0602b575 2195464 hobbit 600 32768 1 0x0702b575 2228233 hobbit 600 524288 2 0x0802b575 2261002 hobbit 600 524288 2 The "bytes" and "nattch" are typical for a default Hobbit setup. If a Hobbit run has crashed, it might leave these behind. In that case, use "ipcrm" to delete them. (Hobbit will do that automatically when restarting, but if they were created by another user then it cannot). Regards, Henrik
list Henrik Størner
▸
On Wed, Jun 21, 2006 at 11:17:12AM -0700, Kasperowicz, Shaun wrote:
Thanks Henrik, I was able to delete the stale shared memory segments and start hobbit. Still getting some errors in hobbitlaunch.log though: 2006-06-21 10:10:21 Setting up hobbitd channels 2006-06-21 10:10:21 Setting up logfiles 2006-06-21 10:10:26 Task bbhistory terminated, status 1 2006-06-21 10:10:26 Task bbpage terminated, status 1 2006-06-21 10:10:26 Task rrdstatus terminated, status 1 2006-06-21 10:10:26 Task rrddata terminated, status 1 2006-06-21 10:10:27 Task bbdisplay terminated by signal 9 2006-06-21 10:10:27 Task clientdata terminated, status 1 2006-06-21 10:10:27 Task bbcombotest terminated by signal 9 2006-06-21 10:10:32 Task clientdata terminated, status 1 2006-06-21 10:10:32 Task rrddata terminated, status 1 2006-06-21 10:10:32 Task rrdstatus terminated, status 1 2006-06-21 10:10:32 Task bbhistory terminated, status 1 I assume these are not normal? Hobbitd is running, but no web pages are being generated.
It's definitely not normal. What's in the other logfiles besides hobbitlaunch.log ? Does you "hobbit" user have write access to the logfile directory and the ~hobbit/server/www/ directory ? What happens if you login as the hobbit user, then run ./server/bin/bbcmd hobbitd_channel --channel=stachg hobbitd_history Regards, Henrik
list Shaun Kasperowicz
Henrik, In the other logfiles I have nothing. Hobbitlaunch.log is the only one getting any data. Hobbit user is owner of both /var/log/hobbit and everything under /usr/local/hobbit/. And when I run the bbcmd command you suggested, I get this: bash-2.03$ /usr/local/hobbit/server/bin/bbcmd hobbitd_channel --channel=stachg hobbitd_history 2006-06-28 15:15:16 Using default environment file /usr/local/hobbit/server/etc/hobbitserver.cfg 2006-06-28 15:15:16 Could not get shm of size 262144: No such file or directory 2006-06-28 15:15:16 Channel not available Looks like still more shared memory problems? Regards,
▸
-Shaun
-----Original Message-----
From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: Tuesday, June 27, 2006 5:38 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Problems starting hobbit
On Wed, Jun 21, 2006 at 11:17:12AM -0700, Kasperowicz, Shaun wrote:Thanks Henrik, I was able to delete the stale shared memory segments and start hobbit. Still getting some errors in hobbitlaunch.log though: 2006-06-21 10:10:21 Setting up hobbitd channels 2006-06-21 10:10:21 Setting up logfiles 2006-06-21 10:10:26 Task bbhistory terminated, status 1 2006-06-21 10:10:26 Task bbpage terminated, status 1 2006-06-21 10:10:26 Task rrdstatus terminated, status 1 2006-06-21 10:10:26 Task rrddata terminated, status 1 2006-06-21 10:10:27 Task bbdisplay terminated by signal 9 2006-06-21 10:10:27 Task clientdata terminated, status 1 2006-06-21 10:10:27 Task bbcombotest terminated by signal 9 2006-06-21 10:10:32 Task clientdata terminated, status 1 2006-06-21 10:10:32 Task rrddata terminated, status 1 2006-06-21 10:10:32 Task rrdstatus terminated, status 1 2006-06-21 10:10:32 Task bbhistory terminated, status 1 I assume these are not normal? Hobbitd is running, but no web pages are being generated.
It's definitely not normal. What's in the other logfiles besides hobbitlaunch.log ? Does you "hobbit" user have write access to the logfile directory and the ~hobbit/server/www/ directory ? What happens if you login as the hobbit user, then run ./server/bin/bbcmd hobbitd_channel --channel=stachg hobbitd_history Regards, Henrik
list Charles Jones
Don't know if it will help, but the last time I had this problem, the fix was completely stop hobbit, make sure all of the processes are gone, and run /usr/bin/ipcs. I had a bunch of shared memory segments that were hung, that I was able to remove using /usr/bin/ipcrm. Note the user in the ipcs output - do not remove segments belong to other users or you may break some other application you have running (I did that too) :-) If this does not help, you may need to edit your /etc/sysctl.conf and increase your shared memory. -Charles
▸
Kasperowicz, Shaun wrote:Henrik, In the other logfiles I have nothing. Hobbitlaunch.log is the only one getting any data. Hobbit user is owner of both /var/log/hobbit and everything under /usr/local/hobbit/. And when I run the bbcmd command you suggested, I get this: bash-2.03$ /usr/local/hobbit/server/bin/bbcmd hobbitd_channel --channel=stachg hobbitd_history 2006-06-28 15:15:16 Using default environment file /usr/local/hobbit/server/etc/hobbitserver.cfg 2006-06-28 15:15:16 Could not get shm of size 262144: No such file or directory 2006-06-28 15:15:16 Channel not available Looks like still more shared memory problems? Regards, -Shaun -----Original Message----- From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid] Sent: Tuesday, June 27, 2006 5:38 AM To: user-ae9b8668bcde@xymon.invalid Subject: Re: [hobbit] Problems starting hobbit On Wed, Jun 21, 2006 at 11:17:12AM -0700, Kasperowicz, Shaun wrote:Thanks Henrik, I was able to delete the stale shared memory segments and start hobbit. Still getting some errors in hobbitlaunch.logthough:2006-06-21 10:10:21 Setting up hobbitd channels 2006-06-21 10:10:21 Setting up logfiles 2006-06-21 10:10:26 Task bbhistory terminated, status 1 2006-06-21 10:10:26 Task bbpage terminated, status 1 2006-06-21 10:10:26 Task rrdstatus terminated, status 1 2006-06-21 10:10:26 Task rrddata terminated, status 1 2006-06-21 10:10:27 Task bbdisplay terminated by signal 9 2006-06-21 10:10:27 Task clientdata terminated, status 1 2006-06-21 10:10:27 Task bbcombotest terminated by signal 9 2006-06-21 10:10:32 Task clientdata terminated, status 1 2006-06-21 10:10:32 Task rrddata terminated, status 1 2006-06-21 10:10:32 Task rrdstatus terminated, status 1 2006-06-21 10:10:32 Task bbhistory terminated, status 1 I assume these are not normal? Hobbitd is running, but no web pagesarebeing generated.It's definitely not normal. What's in the other logfiles besides hobbitlaunch.log ? Does you "hobbit" user have write access to the logfile directory and the ~hobbit/server/www/ directory ? What happens if you login as the hobbit user, then run ./server/bin/bbcmd hobbitd_channel --channel=stachg hobbitd_history
list Henrik Størner
▸
On Wed, Jun 28, 2006 at 03:29:37PM -0700, Kasperowicz, Shaun wrote:
2006-06-28 15:15:16 Could not get shm of size 262144: No such file or directory 2006-06-28 15:15:16 Channel not available Looks like still more shared memory problems?
Perhaps. It doesn't sound right, since this would show up in the
hobbitd startup also, and your last log-snippet didn't show that.
Could you try editing your server/etc/hobbitlaunch.cfg file: The first
entry with the "CMD hobbitd ...." line, add "--debug" at the end of that
(very long) line so it becomes something like
[hobbitd]
ENVFILE /usr/lib/hobbit/server/etc/hobbitserver.cfg
CMD hobbitd --pidfile=$BBSERVERLOGS/hobbitd.pid --restart=$BBTMP/hobbitd.chk --checkpoint-file=$BBTMP/hobbitd.chk --checkpoint-interval=600 --log=$BBSERVERLOGS/hobbitd.log --admin-senders=127.0.0.1,$BBSERVERIP --maint-senders=127.0.0.1 --debug
That should dump a bit more detail about what is happening into
hobbitlaunch.log.
Is there a "hobbitd.log" file in the same directory as hobbitlaunch.log?
Anything in that ?
Henrik
list T.J. Yang
Looks like hobbit has better html/javascript integration than bb. I say we can magnify a portion of trending graph from clicking web page. My current bb server administration require using many bb commands to cleanup a retired host. I hope hobbit server can enable same kind of removal from web gui. T.J. Yang
list Shaun Kasperowicz
I added --debug to hobbitlaunch.cfg, but I don't seem to be getting any more info in hobbitlaunch.log. I started hobbit and am seeing the same stuff: 2006-06-30 16:50:57 Task rrddata terminated, status 1 2006-06-30 16:50:57 Task rrdstatus terminated, status 1 2006-06-30 16:50:57 Task bbpage terminated, status 1 2006-06-30 16:50:57 Task clientdata terminated, status 1 2006-06-30 16:51:03 Task rrddata terminated, status 1 2006-06-30 16:51:03 Task rrdstatus terminated, status 1 2006-06-30 16:51:03 Task bbpage terminated, status 1 2006-06-30 16:51:03 Task clientdata terminated, status 1 2006-06-30 16:51:38 Task bbdisplay terminated by signal 9 2006-06-30 16:52:38 Task bbdisplay terminated by signal 9 The hobbitd.log just had a couple lines from a long time ago: 2006-06-15 11:31:57 Setup complete 2006-06-15 14:23:50 Setup complete
▸
Thanks,
-Shaun
-----Original Message-----
From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: Wednesday, June 28, 2006 10:31 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Problems starting hobbit
On Wed, Jun 28, 2006 at 03:29:37PM -0700, Kasperowicz, Shaun wrote:2006-06-28 15:15:16 Could not get shm of size 262144: No such file or directory 2006-06-28 15:15:16 Channel not available Looks like still more shared memory problems?
Perhaps. It doesn't sound right, since this would show up in the
hobbitd startup also, and your last log-snippet didn't show that.
Could you try editing your server/etc/hobbitlaunch.cfg file: The first
entry with the "CMD hobbitd ...." line, add "--debug" at the end of that
(very long) line so it becomes something like
[hobbitd]
ENVFILE /usr/lib/hobbit/server/etc/hobbitserver.cfg
CMD hobbitd --pidfile=$BBSERVERLOGS/hobbitd.pid
--restart=$BBTMP/hobbitd.chk --checkpoint-file=$BBTMP/hobbitd.chk
--checkpoint-interval=600 --log=$BBSERVERLOGS/hobbitd.log
--admin-senders=127.0.0.1,$BBSERVERIP --maint-senders=127.0.0.1 --debug
That should dump a bit more detail about what is happening into
hobbitlaunch.log.
Is there a "hobbitd.log" file in the same directory as hobbitlaunch.log?
Anything in that ?
Henrik
list Shaun Kasperowicz
Thanks Charles, I actually did run ipcs and ipcrm, which got me to my current problem ;). I have also edited /etc/system (Solaris machine) to add more shared memory segments, but I think this is another problem now. Maybe I should just recompile on a different machine and see if it works. Thanks, -Shaun
▸
From: Charles Jones [mailto:user-e86b4aeade4e@xymon.invalid]
Sent: Wednesday, June 28, 2006 4:11 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Problems starting hobbit
Don't know if it will help, but the last time I had this problem, the
fix was completely stop hobbit, make sure all of the processes are gone,
and run /usr/bin/ipcs. I had a bunch of shared memory segments that were
hung, that I was able to remove using /usr/bin/ipcrm. Note the user in
the ipcs output - do not remove segments belong to other users or you
may break some other application you have running (I did that too) :-)
If this does not help, you may need to edit your /etc/sysctl.conf and
increase your shared memory.
-Charles
Kasperowicz, Shaun wrote:
Henrik,
In the other logfiles I have nothing. Hobbitlaunch.log is the only one
getting any data.
Hobbit user is owner of both /var/log/hobbit and everything under
/usr/local/hobbit/.
And when I run the bbcmd command you suggested, I get this:
bash-2.03$ /usr/local/hobbit/server/bin/bbcmd hobbitd_channel
--channel=stachg hobbitd_history
2006-06-28 15:15:16 Using default environment file
/usr/local/hobbit/server/etc/hobbitserver.cfg
2006-06-28 15:15:16 Could not get shm of size 262144: No such file or
directory
2006-06-28 15:15:16 Channel not available
Looks like still more shared memory problems?
Regards,
-Shaun
-----Original Message-----
From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: Tuesday, June 27, 2006 5:38 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Problems starting hobbit
On Wed, Jun 21, 2006 at 11:17:12AM -0700, Kasperowicz, Shaun wrote:
Thanks Henrik, I was able to delete the stale shared memory segments
and start hobbit. Still getting some errors in hobbitlaunch.log
though:
2006-06-21 10:10:21 Setting up hobbitd channels
2006-06-21 10:10:21 Setting up logfiles
2006-06-21 10:10:26 Task bbhistory terminated, status 1
2006-06-21 10:10:26 Task bbpage terminated, status 1
2006-06-21 10:10:26 Task rrdstatus terminated, status 1
2006-06-21 10:10:26 Task rrddata terminated, status 1
2006-06-21 10:10:27 Task bbdisplay terminated by signal 9
2006-06-21 10:10:27 Task clientdata terminated, status 1
2006-06-21 10:10:27 Task bbcombotest terminated by signal 9
2006-06-21 10:10:32 Task clientdata terminated, status 1
2006-06-21 10:10:32 Task rrddata terminated, status 1
2006-06-21 10:10:32 Task rrdstatus terminated, status 1
2006-06-21 10:10:32 Task bbhistory terminated, status 1
I assume these are not normal? Hobbitd is running, but no web pages
are
being generated.
It's definitely not normal.
What's in the other logfiles besides hobbitlaunch.log ?
Does you "hobbit" user have write access to the logfile directory
and the ~hobbit/server/www/ directory ?
What happens if you login as the hobbit user, then run
./server/bin/bbcmd hobbitd_channel --channel=stachg hobbitd_history
list Henrik Størner
▸
On Thu, Jun 29, 2006 at 08:25:10AM -0500, T.J. Yang wrote:
My current bb server administration require using many bb commands to cleanup a retired host. I hope hobbit server can enable same kind of removal from web gui.
Shouldn't be difficult, but right now you need to do two things:
1) Remove the host from the bb-hosts file.
2) Run the command
bb 127.0.0.1 "drop HOSTNAME"
If you only want to remove a test that is purple because you've stopped
testing it, then
bb 127.0.0.1 "drop HOSTNAME TESTNAME"
will do it.
Regards,
Henrik
list Allan Spencer
▸
Henrik Stoerner wrote:
On Thu, Jun 29, 2006 at 08:25:10AM -0500, T.J. Yang wrote:My current bb server administration require using many bb commands to cleanup a retired host. I hope hobbit server can enable same kind of removal from web gui.Shouldn't be difficult, but right now you need to do two things: 1) Remove the host from the bb-hosts file. 2) Run the command bb 127.0.0.1 "drop HOSTNAME" If you only want to remove a test that is purple because you've stopped testing it, then bb 127.0.0.1 "drop HOSTNAME TESTNAME" will do it. Regards, Henrik
This would be a really good additional to the WEB UI to try and get as much as possible away from accessing the server via ssh or similar. There is already an Edit Hosts component that I know of but is there an edit alerts one as well ? If not perhaps this can be another addition for editing things through the UI. Also another thing someone suggested to me yesterday with regards to the 'critical systems' piece thats now in there, it would be good if possible to have a button on each page for each service that says 'add this test to critical systems' which is nice and lazy but hey just passing on some info that someone said to me :) Cheers Allan
list T.J. Yang
----- Original Message ----- From: "Henrik Stoerner" <user-ce4a2c883f75@xymon.invalid> To: <user-ae9b8668bcde@xymon.invalid> Sent: Monday, July 17, 2006 4:44 PM Subject: Re: [hobbit] Feature request: Remove a purple host hobbit server webpage
▸
On Thu, Jun 29, 2006 at 08:25:10AM -0500, T.J. Yang wrote:My current bb server administration require using many bb commands to cleanup a retired host. I hope hobbit server can enable same kind of removal from web gui.Shouldn't be difficult, but right now you need to do two things: 1) Remove the host from the bb-hosts file. 2) Run the command bb 127.0.0.1 "drop HOSTNAME" If you only want to remove a test that is purple because you've stopped testing it, then bb 127.0.0.1 "drop HOSTNAME TESTNAME" will do it.
Henrik, thanks for the repply First applogize for not fufilling my promise yet(autoconfigsicate hobbit). It is easier say then done(lack of time,skill and will). Secondly, for web GUI enhancement of hobbit, can you try YUI (see R1) ? Tigra menu is ok but YUI(version 0.11.0) looks very promissing and powerful. Lots of features in Tigra Gold/Pro are avaiable in YUI also. R1. http://developer.yahoo.com/yui/ tj yang
Regards, Henrik