And can you stop the hobbit server with hobbit.sh or is one process still running after that?
<br><br><br>>From: "Vernon Everett" <user-99fc6b22a3a3@xymon.invalid><br>>Reply-To: user-ae9b8668bcde@xymon.invalid<br>>To: <user-ae9b8668bcde@xymon.invalid><br>>Subject: RE: [hobbit] Status Unavailable<br>>Date: Mon, 4 Jul 2005 14:23:56 +0800<br>><br>>Yes.<br>>Quite often.<br>>---snip---<br>>2005-07-04 14:09:17 Whoops ! bb failed to send message - timeout<br>>2005-07-04 14:09:17 Could not get the Hobbit statuslog-list<br>>2005-07-04 14:09:50 Whoops ! bb failed to send message - timeout<br>>2005-07-04 14:09:50 hobbitd status-board not available<br>>2005-07-04 14:10:49 Whoops ! bb failed to send message - timeout<br>>2005-07-04 14:10:49 hobbitd status-board not available<br>>2005-07-04 14:11:49 Whoops ! bb failed to send message - timeout<br>>2005-07-04 14:11:49 hobbitd status-board not available<br>>2005-07-04 14:12:52 Whoops ! bb failed to send message - timeout<br>>2005-07-04 14:12:52 hobbitd status-board not available<br>>2005-07-04 14:13:50 Whoops ! bb failed to send message - timeout<br>>2005-07-04 14:13:50 hobbitd status-board not available<br>>2005-07-04 14:14:50 Whoops ! bb failed to send message - timeout<br>>2005-07-04 14:14:50 hobbitd status-board not available<br>>2005-07-04 14:16:22 Whoops ! bb failed to send message - timeout<br>>2005-07-04 14:16:22 hobbitd status-board not available<br>>2005-07-04 14:16:22 WARNING: Runtime 61 longer than BBSLEEP (60)<br>>2005-07-04 14:16:52 Whoops ! bb failed to send message - timeout<br>>2005-07-04 14:16:52 hobbitd status-board not available<br>>2005-07-04 14:17:52 Whoops ! bb failed to send message - timeout<br>>2005-07-04 14:17:52 hobbitd status-board not available<br>>2005-07-04 14:18:52 Whoops ! bb failed to send message - timeout<br>>2005-07-04 14:18:52 hobbitd status-board not available<br>>2005-07-04 14:19:52 Whoops ! bb failed to send message - timeout<br>>2005-07-04 14:19:52 hobbitd status-board not available<br>>2005-07-04 14:21:26 Whoops ! bb failed to send message - timeout<br>>2005-07-04 14:21:26 hobbitd status-board not available<br>>2005-07-04 14:21:26 WARNING: Runtime 61 longer than BBSLEEP (60)<br>>2005-07-04 14:21:59 Whoops ! bb failed to send message - timeout<br>>2005-07-04 14:21:59 hobbitd status-board not available<br>>---snip---<br>><br>><br>>-----Original Message-----<br>>From: Stefan Loos [mailto:user-dea24d965402@xymon.invalid]<br>>Sent: Monday, 4 July 2005 2:16 PM<br>>To: user-ae9b8668bcde@xymon.invalid<br>>Subject: RE: [hobbit] Status Unavailable<br>><br>>Hello Vernon,<br>><br>>can you tell me, if there is anything like "hobbitd status board not<br>>available" in the bb-display.log?<br>><br>>Regards,<br>><br>>Stefan<br>><br>><br><br><br>&gt;From: &quot;Vernon Everett&quot;<br>>&lt;user-99fc6b22a3a3@xymon.invalid&gt;<br>&gt;Reply-To:<br>>user-ae9b8668bcde@xymon.invalid<br>&gt;To: &lt;user-ae9b8668bcde@xymon.invalid&gt;<br>&gt;Subject: RE:<br>>[hobbit] Status Unavailable<br>&gt;Date: Fri, 1 Jul 2005 16:56:38<br>>+0800<br>&gt;<br>&gt;Hi Henrik<br>&gt;<br>&gt;It should be idle. All the<br>>system does is run hobbit. :-)<br>&gt;<br>&gt;Hobbitd is currently dead<br>>in<br>>the water.<br>&gt; [root at pengo log]# strace -p 3025<br>&gt;<br>>Process 3025<br>>attached - interrupt to quit<br>&gt; futex(0x40141b20, FUTEX_WAIT, 2,<br>><br>>NULL<br>&gt;<br>&gt;And it's been like this a while.<br>&gt;When I did<br>>the<br>>kill -6 I got this.<br>&gt; [root at pengo log]# strace -p 3025<br>&gt;<br>>Process<br>>3025 attached - interrupt to quit<br>&gt; futex(0x40141b20,<br>>FUTEX_WAIT, 2,<br>>NULL) = -1 EINTR (Interrupted<br>&gt;system call)<br>&gt; ---<br>>SIGABRT<br>>(Aborted) @ 0 (0) ---<br>&gt; Process 3025 detached<br>&gt;Which I<br>>suppose<br>>was expected :-)<br>&gt;<br>&gt;I restarted it, and got<br>>this.<br>&gt; [root at pengo etc]# strace -p 9223<br>&gt; Process<br>>9223 attached<br>>- interrupt to quit<br>&gt; semop(32769, 0xbfffe3a0, 1<br>&gt;Nope,<br>>there is<br>>nothing I forgot to cut and paste.<br>&gt;That really was<br>>it.<br>&gt;<br>&gt;And this shit just gets stranger and<br>>stranger.<br>&gt;It isn't dumping core.<br>&gt;I hit it with a kill -6<br>>and nothing happens.<br>&gt;I then thought maybe we were both mistaken,<br>>and had the command wrong or<br>&gt;my linux was defaulted to not core,<br>>so I started vi in a session and did<br>&gt;a kill -6 on that. That<br>>dumped?!<br>&gt;Hobbit isn't dumping.<br>&gt;<br>&gt;I rebooted and<br>>tried again.<br>&gt;I managed to get a nice strace output - see attached<br>>- but still no damn<br>&gt;core.<br>&gt;<br>&gt;OK, I added debug, and<br>>restarted.<br>&gt;When I went to check the logs, I found this in<br>>hobbitlaunch.log.<br>&gt;---snip---<br>&gt;2005-07-01 16:37:21 Loading<br>>tasklist configuration<br>>from<br>&gt;/usr/lib/hobbit/server/etc/hobbitlaunch.cfg<br>&gt;2005-07-0<br>>1<br>>16:37:21 Loading hostnames<br>&gt;2005-07-01 16:37:21 Loading saved<br>>state<br>&gt;2005-07-01 16:37:21 Setting up network listener on<br>>0.0.0.0:1984<br>&gt;2005-07-01 16:37:21 Cannot bind to listen socket<br>>(Address already in<br>&gt;use)<br>&gt;2005-07-01 16:37:21 Task hobbitd<br>>started with PID 4761<br>&gt;2005-07-01 16:37:26 Task hobbitd<br>>terminated, status 1<br>&gt;2005-07-01 16:37:26 Loading<br>>hostnames<br>&gt;2005-07-01<br>>16:37:26 Loading saved state<br>&gt;2005-07-01 16:37:26 Task hobbitd<br>>started with PID 4765<br>&gt;2005-07-01 16:37:26 Setting up network<br>>listener on<br>>0.0.0.0:1984<br>&gt;2005-07-01 16:37:26 Cannot bind to listen socket<br>>(Address already in<br>&gt;use)<br>&gt;2005-07-01 16:37:26 Task hobbitd<br>>terminated, status 1<br>&gt;2005-07-01 16:37:31 Loading<br>>hostnames<br>&gt;2005-07-01 16:37:31 Loading saved<br>>state<br>&gt;2005-07-01<br>>16:37:31 Task hobbitd started with PID 4770<br>&gt;2005-07-01 16:37:31<br>>Setting up network listener on 0.0.0.0:1984<br>&gt;2005-07-01 16:37:31<br>>Cannot bind to listen socket (Address already<br>>in<br>&gt;use)<br>&gt;2005-07-01 16:37:31 Task hobbitd terminated,<br>>status<br>>1<br>&gt;2005-07-01 16:37:36 Task hobbitd started with PID<br>>4774<br>&gt;2005-07-01 16:37:36 Loading hostnames<br>&gt;2005-07-01<br>>16:37:36 Loading saved state<br>&gt;2005-07-01 16:37:36 Setting up<br>>network listener on 0.0.0.0:1984<br>&gt;2005-07-01 16:37:36 Cannot bind<br>>to listen socket (Address already in<br>&gt;use)<br>&gt;2005-07-01<br>>16:37:36 Task hobbitd terminated, status 1<br>&gt;2005-07-01 16:37:41<br>>Task hobbitd started with PID 4778<br>&gt;2005-07-01 16:37:41 Loading<br>>hostnames<br>&gt;2005-07-01<br>>16:37:41 Loading saved state<br>&gt;2005-07-01 16:37:41 Setting up<br>>network listener on 0.0.0.0:1984<br>&gt;2005-07-01 16:37:41 Cannot bind<br>>to listen socket (Address already in<br>&gt;use)<br>&gt;2005-07-01<br>>16:37:41 Task hobbitd terminated, status 1<br>&gt;2005-07-01 16:37:46<br>>Task hobbitd started with PID 4783<br>&gt;2005-07-01 16:37:46 Loading<br>>hostnames<br>&gt;2005-07-01<br>>16:37:46 Loading saved state<br>&gt;2005-07-01 16:37:46 Setting up<br>>network listener on 0.0.0.0:1984<br>&gt;2005-07-01 16:37:46 Cannot bind<br>>to listen socket (Address already in<br>&gt;use)<br>&gt;2005-07-01<br>>16:37:46 Task hobbitd terminated, status<br>>1<br>&gt;---snip---<br>&gt;<br>&gt;Looks like a clue.<br>&gt;I will add<br>>the output of netstat -a<br>&gt;<br>&gt;Got the hobbitd.log file for you<br>>too.<br>&gt;<br>&gt;Let me know if there is<br>>anything else I can get you.<br>&gt;<br>&gt;Regards<br>&gt;<br>>Vernon<br>&gt;<br>&gt;P.S. Your cold one is quickly becoming many cold<br>>ones if you ever get<br>>to<br>&gt;Perth<br>&gt;<br>&gt;<br>&gt;<br>&gt;<br>&gt;<br>&gt;-----Orig<br>>inal<br>>Message-----<br>&gt;From: Henrik Stoerner<br>>[mailto:user-ce4a2c883f75@xymon.invalid]<br>&gt;Sent: Friday, 1 July 2005 3:38<br>>PM<br>&gt;To:<br>>user-ae9b8668bcde@xymon.invalid<br>&gt;Subject: Re: [hobbit] Status<br>>Unavailable<br>&gt;<br>&gt;On Fri, Jul 01, 2005 at 03:25:30PM +0800,<br>>Vernon Everett wrote:<br>&gt; &gt; Thanks for helping on this.<br>&gt;<br>>&gt; I rebooted this morning. Could the memory leak still effect me in<br>>that<br>&gt;<br>&gt; &gt; short time?<br>&gt;<br>&gt;Probably not. Just<br>>wanted to rule out this possibility.<br>&gt;<br>&gt; &gt; No<br>>&quot;failed allocation&quot; in dmesg output.<br>&gt; &gt; Do you want<br>>the full output?<br>&gt;<br>&gt;No, I dont think that is<br>>necessary.<br>&gt;<br>&gt; &gt; [root at pengo log]# vmstat 4<br>>20<br>&gt;<br>&gt;And your system is mostly idle with no swap or disk<br>>activity.<br>&gt;<br>&gt; &gt; [hobbit at pengo hobbit]$ server/bin/bb<br>>127.0.0.1 &quot;hobbitdboard&quot;<br>&gt; &gt;<br>>2005-07-01 15:21:45 Whoops ! bb failed to send message -<br>>timeout<br>&gt;<br>&gt;Could you try running &quot;strace -p<br>>&lt;process-ID of the hobbitd process&gt;&quot;<br>&gt;for a minute or<br>>two and send me the output, then do a &quot;kill<br>>-6<br>&gt;&lt;process-id&gt;&quot; and mail me the core-file from<br>>~hobbit/server/tmp/<br>&gt;together with the ~hobbit/server/bin/hobbitd<br>>file ?<br>&gt;<br>&gt;Also, after this try adding a &quot;--debug&quot;<br>>to the hobbitd commandline in<br>&gt;hobbitlaunch.cfg.<br>>Let it run for a while and then mail me the<br>&gt;hobbitd.log<br>>file.<br>&gt;<br>&gt;This bug sounds a bit nasty, I think<br>>....<br>&gt;<br>&gt;<br>&gt;Regards,<br>&gt;Henrik<br>&gt;<br>&gt;<br>&g<br>>t;To<br>>unsubscribe from the hobbit list, send an e-mail<br>>to<br>&gt;user-095ef1c764a2@xymon.invalid<br>&gt;<br>&gt;<br>&gt;_ _ _ _ _ _<br>>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _<br>>_<br>&gt;<br>&gt;NOTICE: This message and any attachments are<br>>confidential and may contain copyright material<br>&gt;of Australian<br>>Finance Group Limited or a third party. It is intended solely for the<br>>purpose of the<br>&gt;addressee and any other named recipient. If you<br>>are not the intended recipient, any use,<br>&gt;distribution, disclosure<br>>or copying of this message is strictly prohibited. The confidentiality<br>>attached<br>&gt;to this message is not waived or lost by reason of the<br>>mistaken transmission or delivery to any<br>&gt;unintended party. If you<br>>have received this message in error, please notify the author<br>>immediately or<br>&gt;contact Australian Finance Group on +61 8 9420<br>>7888.<br>&gt;<br>&gt;<br>&gt;To unsubscribe from the hobbit list, send<br>>an e-mail to<br>&gt;user-095ef1c764a2@xymon.invalid<br>&gt;<br>&gt;<br><br>><br>><br>><br>>To unsubscribe from the hobbit list, send an e-mail to<br>>user-095ef1c764a2@xymon.invalid<br>><br>><br>>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _<br>><br>>NOTICE: This message and any attachments are confidential and may contain copyright material<br>>of Australian Finance Group Limited or a third party. It is intended solely for the purpose of the<br>>addressee and any other named recipient. If you are not the intended recipient, any use,<br>>distribution, disclosure or copying of this message is strictly prohibited. The confidentiality attached<br>>to this message is not waived or lost by reason of the mistaken transmission or delivery to any<br>>unintended party. If you have received this message in error, please notify the author immediately or<br>>contact Australian Finance Group on +61 8 9420 7888.<br>><br>><br>>To unsubscribe from the hobbit list, send an e-mail to<br>>user-095ef1c764a2@xymon.invalid<br>><br>><br>