Connection problems with bbproxy 🔗 link

On Mon, Aug 08, 2005 at 06:38:12PM -0600, Kasperowicz, Shaun (Contractor) wrote:

So far I have been able to make the external clients send to the bbproxy
server, and the bbproxy server can make contact through the firewall to
the BBDISPLAY server on tcp1984.  However, the BBDISPLAY server is not
doing anything with the data.  Both the bbproxy and the external client
are in bb-hosts, and I can see tcp connections being made on both ends
via snoop.

And this is what I see when I tail the bbproxy logs:

2005-08-08 16:48:12 10.88.19.38 : status btndev8.disk
2005-08-08 16:48:14 10.88.19.38 : status btndev8.top
2005-08-08 16:48:38 Server not responding, message lost

-----Original Message-----
From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid] 
Sent: Monday, August 08, 2005 10:32 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Connection problems with bbproxy

On Mon, Aug 08, 2005 at 06:38:12PM -0600, Kasperowicz, Shaun
(Contractor) wrote:

So far I have been able to make the external clients send to the 
bbproxy server, and the bbproxy server can make contact through the 
firewall to the BBDISPLAY server on tcp1984.  However, the BBDISPLAY 
server is not doing anything with the data.  Both the bbproxy and the 
external client are in bb-hosts, and I can see tcp connections being 
made on both ends via snoop.

And this is what I see when I tail the bbproxy logs:

2005-08-08 16:48:12 10.88.19.38 : status btndev8.disk
2005-08-08 16:48:14 10.88.19.38 : status btndev8.top
2005-08-08 16:48:38 Server not responding, message lost

BB uses the file $BBHOME/etc/security to list the IP-adresses that are
allowed to connect to the bbd daemon. My first guess would be that the
IP-address that bbd sees for the incoming connections from the proxy is
not listed in this file, which causes the bbd daemon to drop the
connection immediately.

I believe you need to restart BB after changing this file.


Regards,
Henrik

-----Original Message-----
From: Kasperowicz, Shaun (Contractor) [mailto:user-b86a185566a6@xymon.invalid] 
Sent: Monday, August 08, 2005 5:38 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: [hobbit] Connection problems with bbproxy

Hello,

I have a BB server (and clients) running 1.8b.  Most of our systems are inside the firewall and are monitored easily.  For the few external servers we used bbfetch with good results for several years.  Recently we added a large amount of servers outside the private network, which caused some bottlenecking and performance issues with bbfetch, so I decided to try bbproxy.

So far I have been able to make the external clients send to the bbproxy server, and the bbproxy server can make contact through the firewall to the BBDISPLAY server on tcp1984.  However, the BBDISPLAY server is not doing anything with the data.  Both the bbproxy and the external client are in bb-hosts, and I can see tcp connections being made on both ends via snoop.  I also don't see any errors in BBOUT, even when I run bbd in debug mode.  I've eliminated the firewall as well, and still have the same results.

This is how I am starting bbproxy:

/opt/bb/bin/bbproxy \
--bbdisplay=10.88.19.48 \
--pidfile=/var/run/bbproxy.pid \
--logfile=/opt/bb/tmp/bbproxy.log \
--log-details \
--debug

And this is what I see when I tail the bbproxy logs:

2005-08-08 16:48:12 10.88.19.38 : status btndev8.disk
2005-08-08 16:48:14 10.88.19.38 : status btndev8.top
2005-08-08 16:48:38 Server not responding, message lost
2005-08-08 16:48:38 Server not responding, message lost
2005-08-08 16:48:38 Server not responding, message lost
2005-08-08 16:48:38 Server not responding, message lost
2005-08-08 16:48:40 Server not responding, message lost

Has anyone run into this before?  Do I need to be running BB/Hobbit on the proxy server too?  Any insight is appreciated.

Thanks,

-Shaun Kasperowicz

On Tue, Aug 09, 2005 at 01:03:21PM -0600, Kasperowicz, Shaun (Contractor) wrote:

Well after some more snooping, we find that bbproxy never makes a complete tcp 
connection with the BB server.  You can see several connection attempts, but they 
are never completed, and the data is never sent from bbproxy to the BB server.  

The bbproxy never gets that far...as if it's doing something during the 
negotiation that our old BB server doesn't like, so it drops the connections.
I'm beginning to think that our BB1.8b clients and server are too old for us 
to use bbproxy with them.  Is this possible?  I'm not sure how far back Hobbit 
is compatible with BB.  

Also, is there a way to make logging more verbose with bbproxy?  That might 
help the troubleshooting.

-Shaun


-----Original Message-----
From: Henrik Stoerner [mailto:user-ce4a2c883f75@xymon.invalid] 
Sent: Tuesday, August 09, 2005 2:20 PM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] Connection problems with bbproxy

On Tue, Aug 09, 2005 at 01:03:21PM -0600, Kasperowicz, Shaun
(Contractor) wrote:

Well after some more snooping, we find that bbproxy never makes a 
complete tcp connection with the BB server.  You can see several 
connection attempts, but they are never completed, and the data is
never sent from bbproxy to the BB server.

I assume what you see in the snoop are the outgoing SYN packets from the
proxy, but not SYN-ACK packets coming back from the BB server ?

Could you run snoop on both servers - the one running bbproxy and the BB
server at the same time ? I'd like to have a look at that traffic.

The bbproxy never gets that far...as if it's doing something during 
the negotiation that our old BB server doesn't like, so it drops the
connections.
I'm beginning to think that our BB1.8b clients and server are too old 
for us to use bbproxy with them.  Is this possible?  I'm not sure how 
far back Hobbit is compatible with BB.

It *should* work - I dont think there's been any change in the BB daemon
for a very long time. I haven't tested bbproxy with a 1.8b server, but I
don't know of anything that would make it fail like this.

There's really nothing special in the way bbproxy sets up the connection
to the BB server - it just invokes the standard "connect()" routine.

Also, is there a way to make logging more verbose with bbproxy?  That 
might help the troubleshooting.

Start bbproxy as a foreground process with debugging enabled. Run it
with all of the usual options, plus "--debug --no-daemon".


One thing I come to think of ... what's the MAXLINE setting on your BB
server (in the $BBHOME/src/bb.h file) ? bbproxy will combine status-
messages into combo-messages - if it ends up building larger messages
than your BB server can accept, they could end up being discarded. But
then you should see the message going over the wire ...


Henrik