funny errors in the logs and weird random http monitoring failures 🔗 link

Hello all, 
 At risk of covering an already well worn topic (I did check the archives first, I swear) I'm getting two strange behaviors that may be related.

I've inherited a Hobbit server that was working fine until the unit was reset (rudely), and since then it's been working but has been a little funky.

In my bb-display.log and bb-network I see:
2005-10-11 16:10:39 Could not connect to bbd at 127.0.0.1:1984 - Connection refused
2005-10-11 16:10:39 Whoops ! bb failed to send message - Connection failed
2005-10-11 16:10:39 hobbitd status-board not available
2005-10-11 16:10:39 Could not connect to bbd at 127.0.0.1:1984 - Connection refused
2005-10-11 16:10:39 Whoops ! bb failed to send message - Connection failed
2005-10-11 16:10:43 Could not connect to bbd at 127.0.0.1:1984 - Connection refused
2005-10-11 16:10:43 Whoops ! bb failed to send message - Connection failed
2005-10-11 16:10:43 hobbitd status-board not available
2005-10-11 16:10:43 Could not connect to bbd at 127.0.0.1:1984 - Connection refused
2005-10-11 16:10:43 Whoops ! bb failed to send message - Connection failed


When I do a "netstat -anp" I do see
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
tcp        0         0 	0.0.0.0:1984            0.0.0.0:*               LISTEN     	6237/hobbitd 

So I'm not sure what gives. It also doesn't seem to be a consistent condition.

Also, perhaps related, I have a few servers in my bb-hosts file that are configured to check http which go into alarm for periods for no apparent reason.
I'm obviously pretty new to Hobbit, so any pointers would be GREATLY appreciated.

Thanks!
-Paul

On Wed, Oct 12, 2005 at 04:56:56PM -0400, Paul Bliss wrote:

What version of Hobbit is this ? If you don't known, running
"~hobbit/server/bin/bbgen --version" should tell you.

Could you check the "hobbitlaunch.log" file in the same directory
if there's any mention of the "hobbitd" task terminating with
a signal or a non-zero exit code ?

Also, perhaps related, I have a few servers in my bb-hosts file that are configured to check http which go into alarm for periods for no apparent reason.

Hmm - I don't see any relation in this to the other problem you have,
but let's handle one problem at a time :-)


Regards,
Henrik

Hello,

I've only just started migrating from the BB client to the hobbit client,
starting with my hobbit server.

Everything looks fine, except the netstat and vmstat columns are now
purple.  The odd thing is that the RRD files are still being updated!

Under BB I was running the LARRD external scripts netstat-bf.sh and
vmstat-bf.sh to generate the pages.  But I see there are sections for
netstat and vmstat in the ~/client/bin/hobbitclient-sunos.sh script so I'm
not keen to add these back in to the clientlaunch.cfg file.

Also, although my vmstat.rrd file is being generated, the graph is
broken.  Oddly enough, the graph on the netstat page is generating fine
with the current data even though the page status is purple.

If anyone can shed some light on what I may have missed I'd appreciate it!

Thanks and regards,
Andy.

#####################################################################################

This email is intended for the person to whom it is addressed
only. If you are not the intended recipient, do not read, copy
or use the contents in any way. The opinions expressed may not
necessarily reflect those of ZESPRI Group of Companies ('ZESPRI').

While every effort has been made to verify the information
contained herein, ZESPRI does not make any representations 
as to the accuracy of the information or to the performance
of any data, information or the products mentioned herein.
ZESPRI will not accept liability for any losses, damage or
consequence, however, resulting directly or indirectly from
the use of this e-mail/attachments.
#####################################################################################

On Thu, Oct 13, 2005 at 12:30:13PM +1300, Andy France wrote:

Hobbit doesn't generate a specific column for the netstat and vmstat
data - similar to how LARRD worked if you had LARDCOMM=DATA configured
in your bf-larrd.sh script. You probably had LARRDCOMM=STATUS ...

It shouldn't break just because of switching from the BB client to the
Hobbit client. It WILL break, if you migrate your server from BB
to Hobbit - the BB vmstat data was a terrible mess with different data
layouts for each operating system, so the format of the vmstat RRD
files just isn't compatible between BB and Hobbit.

Any errors in your rrd-data.log (or larrd-data.log) file ?


Regards,
Henrik

user-ce4a2c883f75@xymon.invalid wrote on 13/10/2005 06:39:05 p.m.:

Oh!  That's probably it :-)  I've dropped the columns and will rely on
trends to view the graphs from now on.

Only the odd "illegal attempt to update using time" message for various
hosts, nothing specific to errors in vmstat.

This server has only ever run hobbit and not BB.  I have tried removing the
vmstat.rrd file and letting it get re-created but this still results in a
broken graph.

I guess I need to brush up on the rrdtool syntax to check if the data file
looks as it should?

Regards,
Henrik

Thanks,

Andy.

#####################################################################################

This email is intended for the person to whom it is addressed
only. If you are not the intended recipient, do not read, copy
or use the contents in any way. The opinions expressed may not
necessarily reflect those of ZESPRI Group of Companies ('ZESPRI').

While every effort has been made to verify the information
contained herein, ZESPRI does not make any representations 
as to the accuracy of the information or to the performance
of any data, information or the products mentioned herein.
ZESPRI will not accept liability for any losses, damage or
consequence, however, resulting directly or indirectly from
the use of this e-mail/attachments.
#####################################################################################

On Fri, Oct 14, 2005 at 09:47:38AM +1300, Andy France wrote:

"Broken" - do you mean there is no graph at all, or it just looks 
wrong ?

Could you send me the vmstat.rrd file that gets created, and a copy of
the client data reported by the Hobbit client (you can pick it up from
the "Client data" link on the cpu-, disk- or memory-status displays
for that host).


Regards,
Henrik

Hi,

Andy France schrieb:

That's exactly what I get in my larrd-status.log. It's filling up
rapidly but the memory graphs are ok.

2005-10-14 08:26:58 RRD error updating 
/data1/hobbit/data/rrd/xnode6/memory.actual.rrd from 1x.1x.1x.3x: 
illegal attempt to update using time 1129271218 when last update time is 
1129271218 (minimum one second step)

Regards,
Dirk