moving hobbit from Ubuntu to Debian
This probably isn't going to be much help with this problem, but Henrik might find the feed back of some interest. I recently rebooted a Sun server. Later the hobbit client status for the server went purple and I noticed I was getting these errors in the hobbitclient.log on that server: 2006-05-25 06:58:38 Whoops ! bb failed to send message - timeout 2006-05-25 07:03:43 Whoops ! bb failed to send message - timeout 2006-05-25 07:08:47 Whoops ! bb failed to send message - timeout However I was getting one status from the client. It was from an add-on application called SE Tool Kit. It displayed the following status: green Disks 2%busy No activity green Nets No activity green NFS No client NFS/RPC activity green Swap There is a lot of unused swap space green RAM 86%free RAM available green Kmem No worries, mate green CPU CPU idling green Mutex No worries, mate green DNLC No worries, mate green Inode No activity yellow TCP High retransmissions, check TCP patch level From this we were able to determine there was a network problem. Our LAN group got involved and resolved the retransmission errors from their side. Getting back to the original Whoops error. In our case it seems the hobbit client was not getting thru because of the high retransmissions. As for why the addon messages got thru and the rest of the hobbit client messages didn't I will leave that up to Henrik for his thoughts on it. For those who may be interested in setting up SE Tool Kit with the hobbit client for Sun. You can get it from here http://www.sunfreeware.com/setoolkit.html. There is an example file included in the package called bigbrother.se. I renamed it to hobbit.se, updated the path for calling bin/bb and changed the BBHOST variable to BBDISP. Add an entry to clientlaunch.cfg: [se] ENVFILE $HOBBITCLIENTHOME/etc/hobbitclient.cfg CMD /opt/RICHPse/bin/se /opt/hobbit/client/hobbit.se LOGFILE $HOBBITCLIENTHOME/logs/hobbitclient.log We also run Orca(http://www.orcaware.com/orca/). It's a performance and trend analysis package. It also uses the SE Tool kit and creates detailed graphs. Here is an what it looks like. http://www.orcaware.com/orca/docs/orcallator.html#TCP_retransmission_duplicate_received_percentage With how fast Hobbit is growing we may someday be able to drop the Orca package. John Dan Simoes wrote:
I appreciate the help. Running the debug you sent, it seems that the box cannot connect to itself on 1984 using its public IP. I will see if I can remove the firewall for now (it's not iptables, it's something the provider gives you). Thanks again. On 5/31/06, Henrik Stoerner <user-ce4a2c883f75@xymon.invalid> wrote:One more thing before I go to sleep :-) Check your hobbitserver.cfg - if you have BBDISP set the same as BBSERVERIP. That *is* the default, but you might have changed it. It should be BBDISP="$BBSERVERIP" in hobbitserver.cfg. Regards, Henrik On Thu, Jun 01, 2006 at 12:04:55AM +0200, Henrik Stoerner wrote:Login as the hobbit user, then run ./server/bin/bbcmd bbgen --debug | less A bit into that log is where bbgen tries to fetch the current status from the Hobbit daemon. It looks like this: 2006-06-01 00:02:24 load_state() 2006-06-01 00:02:24 Transport setup is: 2006-06-01 00:02:24 bbdportnumber = 1984 2006-06-01 00:02:24 bbdispproxyhost = NONE 2006-06-01 00:02:24 bbdispproxyport = 0 2006-06-01 00:02:24 Recipient listed as '127.0.0.1' 2006-06-01 00:02:24 Standard BB protocol on port 1984 2006-06-01 00:02:24 Will connect to address 127.0.0.1 port 1984 2006-06-01 00:02:24 Connect status is 0 2006-06-01 00:02:24 Sent 118 bytes 2006-06-01 00:02:24 Read 3364 bytes Does it report the right IP-address as the 'Recipient' ? Henrik On Wed, May 31, 2006 at 03:00:19PM -0700, Dan Simoes wrote:Telnet to localhost as well as 127.0.0.1 work, yes. The hobbitdboard returns a lot of data. I've literally been looking at this for hours, I don't know what else is wrong. I even did a completely fresh install, dropped in my bb-hosts, same results. I know bb-hosts is OK since it's working elsewhere (I did change the bbd line of course). On 5/31/06, Henrik Stoerner <user-ce4a2c883f75@xymon.invalid> wrote:Can you telnet to 127.0.0.1 port 1984 ? If that times out, then you have a seriously heavy firewall setup. Something like iptables -I INPUT 1 -i lo -j ACCEPT iptables -I OUTPUT 1 -i lo -j ACCEPT should cancel out any paranoid firewall setups locally on the box, for the loopback-interface. Can you do a "~hobbit/server/bin/bb 127.0.0.1 hobbitdboard" and get any data back ? If you're running all of the network tests locally, I think you can get away with setting BBSERVERIP="127.0.0.1" in your hobbitserver.cfg, and let all of the communications happen on the loopback-interface. Of course, for clients to work you'll need to get port 1984 opened up for access from the clients. But loopback should be ok for just getting the Hobbit server up and running. Henrik On Wed, May 31, 2006 at 02:48:33PM -0700, Dan Simoes wrote:I thought of that, eventually, and corrected the error. Same problem. localhost is responding on 1984, but the site's firewall is blocking1984.I don't have any clients connecting at the moment, so I don't imagine this is a problem, but thought I would mention it. On 5/31/06, Henrik Stoerner <user-ce4a2c883f75@xymon.invalid> wrote:Check the IP-adresses for the Hobbit server in the hobbitserver.cfg file. From your logs, it seems that the query to fetch the current total status from the Hobbit daemon goes to the wrong server (the oldIP,probably), so there is no response and therefore the webpages dont get any hosts listed. Henrik On Wed, May 31, 2006 at 02:41:25PM -0700, Dan Simoes wrote:I got an account on a new virtual host (debian 3.1 on virtuozzo), and dropped in a tar of my hobbit dir. I figured I could just edit bb-hosts and be up and running in minutes. Hours later, I'm still not up. OK, so I was missing fping, and rrdtool. Hobbit did not like the older rrdtool 1.X under debian stable, so I switch to testing, upgraded, and recompiled and reinstalled hobbit. It's still not working, even with a simplified bb-hosts. hobbit runs, but the webpage output is devoid of any host data. I can't figure out what else could be wrong, this should have been a very simple port. Logs show: bb--display: 2006-05-31 17:30:22 Whoops ! bb failed to send message - timeout 2006-05-31 17:30:22 hobbitd status-board not available 2006-05-31 17:30:22 WARNING: Runtime 92 longer than BBSLEEP(60)2006-05-31 17:31:09 Whoops ! bb failed to send message - timeout bb-network: 2006-05-31 17:30:27 Whoops ! bb failed to send message - timeout 2006-05-31 17:30:27 WARNING: Runtime 140 longer than time limit(120)2006-05-31 17:31:14 Whoops ! bb failed to send message - timeout hobbitclient.log: 2006-05-31 17:32:47 Whoops ! bb failed to send message - timeout error parsing /proc/net/snmp: Success 2006-05-31 17:34:47 Whoops ! bb failed to send message - timeout error parsing /proc/net/snmp: Success-- Henrik Storner-- Henrik Storner-- Henrik Storner-- Henrik Storner