purple clients problem 🔗 link

Longina

--
Longina Przybyszewska, system programmer
Dept. of Math. & Comp. Sci. - IMADA
University of Southern Denmark, Odense
Campusvej 55,DK-5230 Odense M, Denmark

tel: +XX XXXX XXXX - http://www.imada.sdu.dk         email: user-db0a93c0e294@xymon.invalid
--

On Wed, 28 Dec 2005, Rich Smrcina wrote:

Check your MRTG configuration.  It can not write to a file due to a 
permission problem.  I think I have mine writing to a directory that the 
hobbit user owns (eg: /home/hobbit/...).

Longina Przybyszewska wrote:

I have a problem with clients turning purple after reboot.
Client processes run, tmp/msg.txt is updated on client.
Display shows GREEN on conn|info|trends, and PURPLE on the rest  - 
cpu|disk|memory|procs.

I set debug option on hobbit client in hobbitlaunch.cfg -
but I don't quite understand output in clientdata.log:

CMD hobbitd_channel --channel=client --log=$BBSERVERLOGS/clientdata.log 
--debug hobbitd_client

from hobbitlaunch.log  on the server
2005-12-28 14:10:31 Task mrtg terminated, status 13
2005-12-28 14:14:21 hobbitlaunch starting
2005-12-28 14:14:21 Loading tasklist configuration from 
/opt/hobbit/server/etc/hobbitlaunch.cfg
2005-12-28 14:14:21 Loading hostnames
2005-12-28 14:14:21 Loading saved state
2005-12-28 14:14:21 Setting up network listener on 0.0.0.0:1984
2005-12-28 14:14:21 Setting up signal handlers
2005-12-28 14:14:21 Setting up hobbitd channels
2005-12-28 14:14:21 Setting up logfiles
ERROR: writing /etc/mrtg.ok.ok: Permission denied at 
/usr/local/bin/../lib/mrtg2/MRTG_lib.pm line 1477.
2005-12-28 14:14:27 Task mrtg terminated, status 13
ERROR: writing /etc/mrtg.ok.ok: Permission denied at 
/usr/local/bin/../lib/mrtg2/MRTG_lib.pm line 1477.
2005-12-28 14:19:28 Task mrtg terminated, status 13

from clientdata.log
2005-12-28 14:14:11 Semaphore wait aborted: Interrupted system call
2005-12-28 14:14:26 Setting up client channel (id=7)
2005-12-28 14:14:26 calling ftok('/opt/hobbit/server',7)
2005-12-28 14:14:26 ftok() returns: 0x7004344
2005-12-28 14:14:26 shmget() returns: 0x1712

What test should I make to find out what is wrong?

Thanks in advance for any ideas!

regards
Longina
-- 
Longina Przybyszewska, system programmer
Dept. of Math. & Comp. Sci. - IMADA
University of Southern Denmark, Odense
Campusvej 55,DK-5230 Odense M, Denmark

tel: +XX XXXX XXXX - http://www.imada.sdu.dk         email: 
user-db0a93c0e294@xymon.invalid
-- 

-- 
Rich Smrcina
VM Assist, Inc.
Main: (262)392-2026
Cell: (XXX)XXX-XXXX
Ans Service:  (360)715-2467
user-61add9955ef9@xymon.invalid

Catch the WAVV!  http://www.wavv.org
WAVV 2006 - Chattanooga, TN - April 7-11, 2006

On Thu, Dec 29, 2005 at 03:01:28PM +0100, Longina Przybyszewska wrote:

I started bb on the PURPLE client in debugging mode.
It seems that on the client is everything OK:

fra hobbitclient.log
----
2005-12-29 14:55:23 Transport setup is:
2005-12-29 14:55:23 bbdportnumber = 1984
2005-12-29 14:55:23 bbdispproxyhost = NONE
2005-12-29 14:55:23 bbdispproxyport = 0
2005-12-29 14:55:23 Recipient listed as '130.225.128.14'
2005-12-29 14:55:23 Standard BB protocol on port 1984
2005-12-29 14:55:23 Will connect to address 130.225.128.14 port 1984
2005-12-29 14:55:23 Connect status is 0
2005-12-29 14:55:23 Sent 12003 bytes
2005-12-29 14:55:23 Closing connection
----

How kan I debug server side to see where is the black whole for all the 
client information?

regards
Longina

--
Longina Przybyszewska, system programmer
Dept. of Math. & Comp. Sci. - IMADA
University of Southern Denmark, Odense
Campusvej 55,DK-5230 Odense M, Denmark

tel: +XX XXXX XXXX - http://www.imada.sdu.dk         email: user-db0a93c0e294@xymon.invalid
--

On Thu, 29 Dec 2005, Henrik Stoerner wrote:

On Thu, Dec 29, 2005 at 03:01:28PM +0100, Longina Przybyszewska wrote:

I started bb on the PURPLE client in debugging mode.
It seems that on the client is everything OK:

fra hobbitclient.log
----
2005-12-29 14:55:23 Transport setup is:
2005-12-29 14:55:23 bbdportnumber = 1984
2005-12-29 14:55:23 bbdispproxyhost = NONE
2005-12-29 14:55:23 bbdispproxyport = 0
2005-12-29 14:55:23 Recipient listed as '130.225.128.14'
2005-12-29 14:55:23 Standard BB protocol on port 1984
2005-12-29 14:55:23 Will connect to address 130.225.128.14 port 1984
2005-12-29 14:55:23 Connect status is 0
2005-12-29 14:55:23 Sent 12003 bytes
2005-12-29 14:55:23 Closing connection
----

How kan I debug server side to see where is the black whole for all the
client information?

On the server, login as the hobbit user and run
   bbcmd hobbitd_channel --channel=client cat

When the client sends a message, you should see it dumped to the screen.
If it does, then at least the client message arrives OK on the Hobbit
server, and the server recognizes the client hostname OK.

If you don't see any such message, then the hostname that the client
reports is not listed in bb-hosts - you can see the "ghost" hostname
in the status for the hobbit servers' "hobbitd" column.

The first line you see when dumping the client message should look like
this:
  @@client#24|1135865239.108541|127.0.0.1|localhost|linux
where "localhost" is the hostname, and "linux" is the operating system
that the client reports. The OS must be one of the recognized OS'es:
solaris, sunos, linux, aix, hp-ux, freebsd etc.

If you get that far and still haven't found the problem, add the "--debug"
option to the hobbitd_client command in hobbitlaunch.cfg, and check the
clientdata.log logfile for any errors.


Regards,
Henrik

On Mon, Jan 09, 2006 at 02:24:38PM +0100, Longina Przybyszewska wrote:

On the server, login as the hobbit user and run
  bbcmd hobbitd_channel --channel=client cat

When the client sends a message, you should see it dumped to the screen.
If it does, then at least the client message arrives OK on the Hobbit
server, and the server recognizes the client hostname OK.

If you don't see any such message, then the hostname that the client
reports is not listed in bb-hosts - you can see the "ghost" hostname
in the status for the hobbit servers' "hobbitd" column.

I did the test, and I could see that the purple machines were not polled 
by the server!

So, I made another test, where in  bb-hosts file were _only _ servers 
and purple worksations . All  purple workstations turned green now.
The one failed to show (cpu|disk|memory|procs) -  and only showed green on 
network tests (conn|info|trends).

hobbitd_client runs with --debug option, and the clientdata.log says:
----
2006-01-09 11:35:17 Semaphore wait aborted: Interrupted system call
2006-01-09 11:35:49 Setting up client channel (id=7)

If you don't see any such message, then the hostname that the client
reports is not listed in bb-hosts - you can see the "ghost" hostname
in the status for the hobbit servers' "hobbitd" column.

I did the test, and I could see that the purple machines were not polled
by the server!

Please try setting BBGHOSTS to "2" in hobbitserver.cfg. Restart Hobbit,
and after 10 minutes check the "hobbitd" column on the Hobbit server.
Are there any hosts listed there ?

hobbitd_client runs with --debug option, and the clientdata.log says:
----
2006-01-09 11:35:17 Semaphore wait aborted: Interrupted system call
2006-01-09 11:35:49 Setting up client channel (id=7)

I think you put the "--debug" on the wrong part of the command in
hobbitlaunch.cfg. It must be at the end, like this:

CMD hobbitd_channel --channel=client --log=$BBSERVERLOGS/clientdata.log hobbitd_client --debug

Regards
Longina

--
Longina Przybyszewska, system programmer
Dept. of Math. & Comp. Sci. - IMADA
University of Southern Denmark, Odense
Campusvej 55,DK-5230 Odense M, Denmark

tel: +XX XXXX XXXX - http://www.imada.sdu.dk         email: 
user-db0a93c0e294@xymon.invalid
--

On Thu, Jan 12, 2006 at 02:08:32PM +0100, Longina Przybyszewska wrote:

Please try setting BBGHOSTS to "2" in hobbitserver.cfg. Restart Hobbit,
and after 10 minutes check the "hobbitd" column on the Hobbit server.
Are there any hosts listed there ?

Yes, all my missig clients are listed as ghost + 2 clients which are at both places (as ghosts and showing green dimonds)+ 1 client that just turned to purple.

in ghosts file IP numbers are translated to nonqualified names - if compared with bb-hosts:

in bb-hosts:
----
130.225.128.126   sieglinde.imada.sdu.dk
130.225.128.167   zaira.imada.sdu.dk

----
in ghosts:
130.225.128.126 reported host sieglinde
130.225.128.167 reported host zaira

...
The hostname you see in the "ghosts" list is the hostname the the
clients use when they report data. This *must* match the one found
in the bb-hosts file, or Hobbit will simply discard the client message
because it doesn't know which host it belongs to.

So you can either change your clients to use the "--hostname" option (e.g. use "--hostname=sieglinde.imada.sdu.dk" when calling
runclient.sh), or add "CLIENT:sieglinde" to the entries in the bb-hosts file.
 

On Thu, 12 Jan 2006, Henrik Stoerner wrote:

in bb-hosts:
----
130.225.128.126   sieglinde.imada.sdu.dk
130.225.128.167   zaira.imada.sdu.dk

----
in ghosts:
130.225.128.126 reported host sieglinde
130.225.128.167 reported host zaira

The hostname you see in the "ghosts" list is the hostname the the
clients use when they report data. This *must* match the one found
in the bb-hosts file, or Hobbit will simply discard the client message
because it doesn't know which host it belongs to.

So you can either change your clients to use the "--hostname" option
(e.g. use "--hostname=sieglinde.imada.sdu.dk" when calling
runclient.sh), or add "CLIENT:sieglinde" to the entries in the
bb-hosts file.

regards
Longina

--
Longina Przybyszewska, system programmer
Dept. of Math. & Comp. Sci. - IMADA
University of Southern Denmark, Odense
Campusvej 55,DK-5230 Odense M, Denmark

tel: +XX XXXX XXXX - http://www.imada.sdu.dk         email: user-db0a93c0e294@xymon.invalid
--

regards
Longina


--
Longina Przybyszewska, system programmer
Dept. of Math. & Comp. Sci. - IMADA
University of Southern Denmark, Odense
Campusvej 55,DK-5230 Odense M, Denmark

tel: +XX XXXX XXXX - http://www.imada.sdu.dk         email: user-db0a93c0e294@xymon.invalid
--

On Fri, 13 Jan 2006, Longina Przybyszewska wrote:

On Thu, 12 Jan 2006, Henrik Stoerner wrote:

in bb-hosts:
----
130.225.128.126   sieglinde.imada.sdu.dk
130.225.128.167   zaira.imada.sdu.dk

----
in ghosts:
130.225.128.126 reported host sieglinde
130.225.128.167 reported host zaira

The hostname you see in the "ghosts" list is the hostname the the
clients use when they report data. This *must* match the one found
in the bb-hosts file, or Hobbit will simply discard the client message
because it doesn't know which host it belongs to.

So you can either change your clients to use the "--hostname" option
(e.g. use "--hostname=sieglinde.imada.sdu.dk" when calling
runclient.sh), or add "CLIENT:sieglinde" to the entries in the
bb-hosts file.

It seems to partly(!) explain my puzzle:)
The way client presents to the server obviously was changed in the last upgrade (the day since my problems started).
The workstations which were down during my hobbit uprade could after restart continue with old hobbit_client until they become først PURPLE then '-' and were not upgraded.

Still I don't understand why my 2 servers (sparc/solaris9) stay GREEN on display even if there is an obvious name mismatch - AND there is no shade for incoming data from both servers in clientdata.log! (I am running hobbitd_client with --debug option)

in bb-hosts:
----
130.225.128.9      bach.imada.sdu.dk            # ssh dns rpc=mount,nfs
130.225.128.10     mozart.imada.sdu.dk          # http://www.imada.sdu.dk

in ghosts:
----
130.225.128.9   reported host bach
130.225.128.10  reported host mozart

in ...client/tmp/msg.txt:
----bach----
client bach.sunos
[date]
Fri Jan 13 13:37:05 MET 2006
....

----mozart----
client mozart.sunos
[date]
Fri Jan 13 12:43:58 CET 2006
....

on DISPLAY:
----
fredag 13 januar 2006, 13:39:17 CET - Disk partitions on mozart.imada.sdu.dk OK
fredag 13 januar 2006, 13:39:17 CET up: 164 day(s), 4 users, 77 procs, load=11

regards
Longina

--
Longina Przybyszewska, system programmer
Dept. of Math. & Comp. Sci. - IMADA
University of Southern Denmark, Odense
Campusvej 55,DK-5230 Odense M, Denmark

tel: +XX XXXX XXXX - http://www.imada.sdu.dk         email: user-db0a93c0e294@xymon.invalid
--

Regards
Longina

--
Longina Przybyszewska, system programmer
Dept. of Math. & Comp. Sci. - IMADA
University of Southern Denmark, Odense
Campusvej 55,DK-5230 Odense M, Denmark

tel: +XX XXXX XXXX - http://www.imada.sdu.dk         email: user-db0a93c0e294@xymon.invalid
--

On Fri, 13 Jan 2006, Longina Przybyszewska wrote:

UPS. I have just noticed that there are bbrun processes running
on those 2 servers from old BB installation.
I think  it explains all my the troubles -... make upgrades carefully...

By the way , is there a special bb_channel for BBclient-Hobbitserver
communication?

regards
Longina


--
Longina Przybyszewska, system programmer
Dept. of Math. & Comp. Sci. - IMADA
University of Southern Denmark, Odense
Campusvej 55,DK-5230 Odense M, Denmark

tel: +XX XXXX XXXX - http://www.imada.sdu.dk         email: user-db0a93c0e294@xymon.invalid
--

On Fri, 13 Jan 2006, Longina Przybyszewska wrote:

On Thu, 12 Jan 2006, Henrik Stoerner wrote:

in bb-hosts:
----
130.225.128.126   sieglinde.imada.sdu.dk
130.225.128.167   zaira.imada.sdu.dk

----
in ghosts:
130.225.128.126 reported host sieglinde
130.225.128.167 reported host zaira

The hostname you see in the "ghosts" list is the hostname the the
clients use when they report data. This *must* match the one found
in the bb-hosts file, or Hobbit will simply discard the client message
because it doesn't know which host it belongs to.

So you can either change your clients to use the "--hostname" option
(e.g. use "--hostname=sieglinde.imada.sdu.dk" when calling
runclient.sh), or add "CLIENT:sieglinde" to the entries in the
bb-hosts file.

It seems to partly(!) explain my puzzle:)
The way client presents to the server obviously was changed in the last upgrade (the day since my problems started).
The workstations which were down during my hobbit uprade could after restart continue with old hobbit_client until they become først PURPLE then '-' and were not upgraded.

Still I don't understand why my 2 servers (sparc/solaris9) stay GREEN on display even if there is an obvious name mismatch - AND there is no shade for incoming data from both servers in clientdata.log! (I am running hobbitd_client with --debug option)

in bb-hosts:
----
130.225.128.9      bach.imada.sdu.dk            # ssh dns rpc=mount,nfs
130.225.128.10     mozart.imada.sdu.dk          # http://www.imada.sdu.dk

in ghosts:
----
130.225.128.9   reported host bach
130.225.128.10  reported host mozart

in ...client/tmp/msg.txt:
----bach----
client bach.sunos
[date]
Fri Jan 13 13:37:05 MET 2006
....

----mozart----
client mozart.sunos
[date]
Fri Jan 13 12:43:58 CET 2006
....

on DISPLAY:
----
fredag 13 januar 2006, 13:39:17 CET - Disk partitions on mozart.imada.sdu.dk OK
fredag 13 januar 2006, 13:39:17 CET up: 164 day(s), 4 users, 77 procs, load=11

regards
Longina

--
Longina Przybyszewska, system programmer
Dept. of Math. & Comp. Sci. - IMADA
University of Southern Denmark, Odense
Campusvej 55,DK-5230 Odense M, Denmark

tel: +XX XXXX XXXX - http://www.imada.sdu.dk         email: user-db0a93c0e294@xymon.invalid
--

-----Original Message-----
From: Longina Przybyszewska [mailto:user-db0a93c0e294@xymon.invalid] Sent: 13 January 2006 14:11
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] purple clients problem

still I have some linux clients that use single name instead of FQname -
they run the same software now ;)
I wouldn't like to mess the bb-hosts file since all our workstations suppose to be identical!

So, how is made the first line in client's msg.txt:

client aida.linux
  or
client ariadne,imada,sdu,dk.linux

Regards
Longina

--
Longina Przybyszewska, system programmer
Dept. of Math. & Comp. Sci. - IMADA
University of Southern Denmark, Odense
Campusvej 55,DK-5230 Odense M, Denmark

tel: +XX XXXX XXXX - http://www.imada.sdu.dk         email: user-db0a93c0e294@xymon.invalid
--

On Fri, 13 Jan 2006, Longina Przybyszewska wrote:

UPS. I have just noticed that there are bbrun processes running
on those 2 servers from old BB installation.
I think  it explains all my the troubles -... make upgrades carefully...

By the way , is there a special bb_channel for BBclient-Hobbitserver
communication?

regards
Longina


--
Longina Przybyszewska, system programmer
Dept. of Math. & Comp. Sci. - IMADA
University of Southern Denmark, Odense
Campusvej 55,DK-5230 Odense M, Denmark

tel: +XX XXXX XXXX - http://www.imada.sdu.dk         email: user-db0a93c0e294@xymon.invalid
--

On Fri, 13 Jan 2006, Longina Przybyszewska wrote:

On Thu, 12 Jan 2006, Henrik Stoerner wrote:

in bb-hosts:
----
130.225.128.126   sieglinde.imada.sdu.dk
130.225.128.167   zaira.imada.sdu.dk

----
in ghosts:
130.225.128.126 reported host sieglinde
130.225.128.167 reported host zaira

The hostname you see in the "ghosts" list is the hostname the the
clients use when they report data. This *must* match the one found
in the bb-hosts file, or Hobbit will simply discard the client message
because it doesn't know which host it belongs to.

So you can either change your clients to use the "--hostname" option
(e.g. use "--hostname=sieglinde.imada.sdu.dk" when calling
runclient.sh), or add "CLIENT:sieglinde" to the entries in the
bb-hosts file.

It seems to partly(!) explain my puzzle:)
The way client presents to the server obviously was changed in the last upgrade (the day since my problems started).
The workstations which were down during my hobbit uprade could after restart continue with old hobbit_client until they become først PURPLE then '-' and were not upgraded.

Still I don't understand why my 2 servers (sparc/solaris9) stay GREEN on display even if there is an obvious name mismatch - AND there is no shade for incoming data from both servers in clientdata.log! (I am running hobbitd_client with --debug option)

in bb-hosts:
----
130.225.128.9      bach.imada.sdu.dk            # ssh dns rpc=mount,nfs
130.225.128.10     mozart.imada.sdu.dk          # http://www.imada.sdu.dk

in ghosts:
----
130.225.128.9   reported host bach
130.225.128.10  reported host mozart

in ...client/tmp/msg.txt:
----bach----
client bach.sunos
[date]
Fri Jan 13 13:37:05 MET 2006
....

----mozart----
client mozart.sunos
[date]
Fri Jan 13 12:43:58 CET 2006
....

on DISPLAY:
----
fredag 13 januar 2006, 13:39:17 CET - Disk partitions on mozart.imada.sdu.dk OK
fredag 13 januar 2006, 13:39:17 CET up: 164 day(s), 4 users, 77 procs, load=11

regards
Longina

--
Longina Przybyszewska, system programmer
Dept. of Math. & Comp. Sci. - IMADA
University of Southern Denmark, Odense
Campusvej 55,DK-5230 Odense M, Denmark

tel: +XX XXXX XXXX - http://www.imada.sdu.dk         email: user-db0a93c0e294@xymon.invalid
--