bbgen frequent yellow alerts - hobbitd problem? 🔗 link

On Mon, Nov 06, 2006 at 07:35:27AM -0800, Mr-Pope wrote:

We are running a new installation of Hobbit 4.2 on Solaris 10 running
in a non-global zone.  Server is a v240 but I don't think that matters
here.

The problem here is that our bbgen status turns yellow with fairly
high frequency, sometimes multiple times an hour, at (what seem like)
random intervals.  In the yellow alert bbgen reports:
"hobbitd status-board not available"

Below are the output from some commands/logs.  These logs don't really
seem to help, so let me know if there is anything else that I can send
along to debug this issue.

$BB --debug $BBDISP "hobbitdboard"
(with no --debug on a 'failure' I get no output.  I'm assuming this is
the same cause of the bbgen yellow alert)

bbgen --debug --report (this one turned bbgen yellow/unavailable.
Note the quick disconnect.)
2006-11-03 09:51:03 load_state()
2006-11-03 09:51:03 Transport setup is:
2006-11-03 09:51:03 bbdportnumber = 1984
2006-11-03 09:51:03 bbdispproxyhost = NONE
2006-11-03 09:51:03 bbdispproxyport = 0
2006-11-03 09:51:03 Recipient listed as '10.xxx.xxx.xxx'
2006-11-03 09:51:03 Standard BB protocol on port 1984
2006-11-03 09:51:03 Will connect to address 10.xxx.xxx.xxx port 1984
2006-11-03 09:51:03 Connect status is 0
2006-11-03 09:51:03 Sent 126 bytes
2006-11-03 09:51:03 Closing connection

On Mon, Nov 06, 2006 at 05:29:59PM +0100, Henrik Stoerner wrote:

I'd like you to add 8 lines between these two:

	n = recv(sockfd, recvbuf, sizeof(recvbuf)-1, 0);
	if (n < 0) {
		dbgprintf("recv() returned error: %s\n", strerror(errno));
		if (errno == EAGAIN) continue; 
	}
	if (n == 0) {
		dbgprintf("recv() gave us 0 bytes\n");
		continue;

Oops - one line too many among those extra ones I sent you:

On Mon, Nov 06, 2006 at 05:29:59PM +0100, Henrik Stoerner wrote:

I'd like you to add 8 lines between these two:

               n = recv(sockfd, recvbuf, sizeof(recvbuf)-1, 0);
               if (n < 0) {
                               dbgprintf("recv() returned error: %s\n", 
strerror(errno));
                               if (errno == EAGAIN) continue;                }
               if (n == 0) {
                               dbgprintf("recv() gave us 0 bytes\n");
                               continue;

                ^^^^^^^^^ dont put this "continue" line in there.


Regards,
Henrik

	From: Matthew G Armstrong [mailto:user-0523254e8394@xymon.invalid] 
	Sent: Monday, November 06, 2006 10:39 AM
	To: user-ae9b8668bcde@xymon.invalid
	Subject: Re: [hobbit] bbgen frequent yellow alerts - hobbitd
problem?
	
	
	are there many problems encountered when you using Solaris with
Hobbit?  I was going to base my design on Solaris but I am now wondering
if I should steer more towards an AIX solution.
	
	
	This is a PRIVATE message. If you are not the intended
recipient, please delete without copying and kindly advise us by e-mail
of the mistake in delivery. NOTE: Regardless of content, this e-mail
shall not operate to bind CSC to any order or other contract unless
pursuant to explicit written agreement or government initiative
expressly permitting the use of e-mail for such purpose.
	
	
Henrik Stoerner <user-ce4a2c883f75@xymon.invalid> 

06/11/2006 16:33 
Please respond to
user-ae9b8668bcde@xymon.invalid


To
user-ae9b8668bcde@xymon.invalid 
cc
Subject
Re: [hobbit] bbgen frequent yellow alerts - hobbitd problem?

	
	Oops - one line too many among those extra ones I sent you:
	
	On Mon, Nov 06, 2006 at 05:29:59PM +0100, Henrik Stoerner wrote:
	

I'd like you to add 8 lines between these two:

                 n = recv(sockfd, recvbuf, sizeof(recvbuf)-1,

0);

                 if (n < 0) {
                                  dbgprintf("recv() returned
error: %s\n", strerror(errno));
                                  if (errno == EAGAIN)
continue; 
                 }
                 if (n == 0) {
                                  dbgprintf("recv() gave us 0
bytes\n");
                                  continue;

	               ^^^^^^^^^ dont put this "continue" line in there.
	
	
	Regards,
	Henrik

On Mon, Nov 06, 2006 at 04:39:22PM +0000, Matthew G Armstrong wrote:

are there many problems encountered when you using Solaris with Hobbit?  I 
was going to base my design on Solaris but I am now wondering if I should 
steer more towards an AIX solution.

On 11/6/06, Henrik Stoerner <user-ce4a2c883f75@xymon.invalid> wrote:

On Mon, Nov 06, 2006 at 07:35:27AM -0800, Mr-Pope wrote:

We are running a new installation of Hobbit 4.2 on Solaris 10 running
in a non-global zone.  Server is a v240 but I don't think that matters
here.

The problem here is that our bbgen status turns yellow with fairly
high frequency, sometimes multiple times an hour, at (what seem like)
random intervals.  In the yellow alert bbgen reports:
"hobbitd status-board not available"

The reports I've had of this only have one thing in common: They all
happen on Solaris 10. So I'm beginning to suspect that maybe Solaris
doesn't work quite the way other systems do.

Or perhaps there is a bug, and something special in Solaris triggers it.

Below are the output from some commands/logs.  These logs don't really
seem to help, so let me know if there is anything else that I can send
along to debug this issue.

$BB --debug $BBDISP "hobbitdboard"
(with no --debug on a 'failure' I get no output.  I'm assuming this is
the same cause of the bbgen yellow alert)

Yes.

bbgen --debug --report (this one turned bbgen yellow/unavailable.
Note the quick disconnect.)
2006-11-03 09:51:03 load_state()
2006-11-03 09:51:03 Transport setup is:
2006-11-03 09:51:03 bbdportnumber = 1984
2006-11-03 09:51:03 bbdispproxyhost = NONE
2006-11-03 09:51:03 bbdispproxyport = 0
2006-11-03 09:51:03 Recipient listed as '10.xxx.xxx.xxx'
2006-11-03 09:51:03 Standard BB protocol on port 1984
2006-11-03 09:51:03 Will connect to address 10.xxx.xxx.xxx port 1984
2006-11-03 09:51:03 Connect status is 0
2006-11-03 09:51:03 Sent 126 bytes
2006-11-03 09:51:03 Closing connection

Interesting.

Since it seems that this bites you more than most others, I'd like you
to do a couple of things for me to figure out what is going on. I need
you to add a couple of debugging lines to Hobbit.

First, in the bbdisplay/loaddata.c file, around line 436 you'll find the
code that prints out the "hobbitd status board not available" message.
It looks like this:
        errprintf("hobbitd status-board not available\n");
I want you to change that to
        errprintf("hobbitd status-board not available, code %d\n", hobbitdresult);


Next, in the lib/sendmsg.c file around line 340 is where the code is
that receives data from Hobbit. You'll find these lines:

        n = recv(sockfd, recvbuf, sizeof(recvbuf)-1, 0);
        if (n > 0) {

I'd like you to add 8 lines between these two:

        n = recv(sockfd, recvbuf, sizeof(recvbuf)-1, 0);
        if (n < 0) {
                dbgprintf("recv() returned error: %s\n", strerror(errno));
                if (errno == EAGAIN) continue;
        }
        if (n == 0) {
                dbgprintf("recv() gave us 0 bytes\n");
                continue;
        }
        if (n > 0) {

(it isn't the prettiest of programming, but it does the job for now).


After making these two changes, run "make clean; make" and copy the
bbdisplay/bbgen binary into your ~hobbit/server/bin/ directory. Let
Hobbit run as normal (with --debug on the bbgen command) and when it
fails I am very interested to see what's in the logfile.


Regards,
Henrik

-----Original Message-----
From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: 06 November 2006 16:30
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] bbgen frequent yellow alerts - hobbitd problem?

On Mon, Nov 06, 2006 at 07:35:27AM -0800, Mr-Pope wrote:

We are running a new installation of Hobbit 4.2 on Solaris 10 running
in a non-global zone.  Server is a v240 but I don't think that matters
here.

The problem here is that our bbgen status turns yellow with fairly
high frequency, sometimes multiple times an hour, at (what seem like)
random intervals.  In the yellow alert bbgen reports:
"hobbitd status-board not available"

The reports I've had of this only have one thing in common: They all
happen on Solaris 10. So I'm beginning to suspect that maybe Solaris
doesn't work quite the way other systems do.

Or perhaps there is a bug, and something special in Solaris triggers it.

Below are the output from some commands/logs.  These logs don't really
seem to help, so let me know if there is anything else that I can send
along to debug this issue.

$BB --debug $BBDISP "hobbitdboard"
(with no --debug on a 'failure' I get no output.  I'm assuming this is
the same cause of the bbgen yellow alert)

Yes.

bbgen --debug --report (this one turned bbgen yellow/unavailable.
Note the quick disconnect.)
2006-11-03 09:51:03 load_state()
2006-11-03 09:51:03 Transport setup is:
2006-11-03 09:51:03 bbdportnumber = 1984
2006-11-03 09:51:03 bbdispproxyhost = NONE
2006-11-03 09:51:03 bbdispproxyport = 0
2006-11-03 09:51:03 Recipient listed as '10.xxx.xxx.xxx'
2006-11-03 09:51:03 Standard BB protocol on port 1984
2006-11-03 09:51:03 Will connect to address 10.xxx.xxx.xxx port 1984
2006-11-03 09:51:03 Connect status is 0
2006-11-03 09:51:03 Sent 126 bytes
2006-11-03 09:51:03 Closing connection

Interesting.

Since it seems that this bites you more than most others, I'd like you
to do a couple of things for me to figure out what is going on. I need
you to add a couple of debugging lines to Hobbit.

First, in the bbdisplay/loaddata.c file, around line 436 you'll find the
code that prints out the "hobbitd status board not available" message.
It looks like this:
        errprintf("hobbitd status-board not available\n");
I want you to change that to
        errprintf("hobbitd status-board not available, code %d\n",
hobbitdresult);


Next, in the lib/sendmsg.c file around line 340 is where the code is
that receives data from Hobbit. You'll find these lines:

n = recv(sockfd, recvbuf, sizeof(recvbuf)-1, 0);
if (n > 0) {

I'd like you to add 8 lines between these two:

n = recv(sockfd, recvbuf, sizeof(recvbuf)-1, 0);
if (n < 0) {
dbgprintf("recv() returned error: %s\n",
strerror(errno));
if (errno == EAGAIN) continue;
}
if (n == 0) {
dbgprintf("recv() gave us 0 bytes\n");
continue;
}
if (n > 0) {

(it isn't the prettiest of programming, but it does the job for now).


After making these two changes, run "make clean; make" and copy the
bbdisplay/bbgen binary into your ~hobbit/server/bin/ directory. Let
Hobbit run as normal (with --debug on the bbgen command) and when it
fails I am very interested to see what's in the logfile.


Regards,
Henrik

On 11/6/06, Mike Rowell <user-63f3e97eb1de@xymon.invalid> wrote:

Henrik,

It might be worth checking to make sure these problems are only on
Solaris 10 x86 as that is the only architecture I've seen this problem
on, sparc seems fine so might help you in narrowing down the problem.

Regards,

Mike Rowell
-----Original Message-----
From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: 06 November 2006 16:30
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] bbgen frequent yellow alerts - hobbitd problem?

On Mon, Nov 06, 2006 at 07:35:27AM -0800, Mr-Pope wrote:

We are running a new installation of Hobbit 4.2 on Solaris 10 running
in a non-global zone.  Server is a v240 but I don't think that matters
here.

The problem here is that our bbgen status turns yellow with fairly
high frequency, sometimes multiple times an hour, at (what seem like)
random intervals.  In the yellow alert bbgen reports:
"hobbitd status-board not available"

The reports I've had of this only have one thing in common: They all
happen on Solaris 10. So I'm beginning to suspect that maybe Solaris
doesn't work quite the way other systems do.

Or perhaps there is a bug, and something special in Solaris triggers it.

Below are the output from some commands/logs.  These logs don't really
seem to help, so let me know if there is anything else that I can send
along to debug this issue.

$BB --debug $BBDISP "hobbitdboard"
(with no --debug on a 'failure' I get no output.  I'm assuming this is
the same cause of the bbgen yellow alert)

Yes.

bbgen --debug --report (this one turned bbgen yellow/unavailable.
Note the quick disconnect.)
2006-11-03 09:51:03 load_state()
2006-11-03 09:51:03 Transport setup is:
2006-11-03 09:51:03 bbdportnumber = 1984
2006-11-03 09:51:03 bbdispproxyhost = NONE
2006-11-03 09:51:03 bbdispproxyport = 0
2006-11-03 09:51:03 Recipient listed as '10.xxx.xxx.xxx'
2006-11-03 09:51:03 Standard BB protocol on port 1984
2006-11-03 09:51:03 Will connect to address 10.xxx.xxx.xxx port 1984
2006-11-03 09:51:03 Connect status is 0
2006-11-03 09:51:03 Sent 126 bytes
2006-11-03 09:51:03 Closing connection

Interesting.

Since it seems that this bites you more than most others, I'd like you
to do a couple of things for me to figure out what is going on. I need
you to add a couple of debugging lines to Hobbit.

First, in the bbdisplay/loaddata.c file, around line 436 you'll find the
code that prints out the "hobbitd status board not available" message.
It looks like this:
        errprintf("hobbitd status-board not available\n");
I want you to change that to
        errprintf("hobbitd status-board not available, code %d\n",
hobbitdresult);


Next, in the lib/sendmsg.c file around line 340 is where the code is
that receives data from Hobbit. You'll find these lines:

        n = recv(sockfd, recvbuf, sizeof(recvbuf)-1, 0);
        if (n > 0) {

I'd like you to add 8 lines between these two:

        n = recv(sockfd, recvbuf, sizeof(recvbuf)-1, 0);
        if (n < 0) {
                dbgprintf("recv() returned error: %s\n",
strerror(errno));
                if (errno == EAGAIN) continue;
        }
        if (n == 0) {
                dbgprintf("recv() gave us 0 bytes\n");
                continue;
        }
        if (n > 0) {

(it isn't the prettiest of programming, but it does the job for now).


After making these two changes, run "make clean; make" and copy the
bbdisplay/bbgen binary into your ~hobbit/server/bin/ directory. Let
Hobbit run as normal (with --debug on the bbgen command) and when it
fails I am very interested to see what's in the logfile.


Regards,
Henrik


This email has been scanned for all viruses by the MessageLabs service.

This email has been scanned for all viruses by the MessageLabs service.

--