After Server Restart get an error and all tests are purple
list Stefan Freisler
Hi, After Server Restart i get an error and all tests are purple. (except conn, http) i get the following error: 2007-01-18 08:28:37 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:28:39 Could not connect to user-88eecf256074@xymon.invalid:1984 - Connection refused what can i do? thx Stefan
list Rolf Schrittenlocher
Hi Stefan, please give some more information. Restart of hobbit server or reboot of the machine? Is the client running (it should start automatically while restarting the server but this doesn't work properly)? Are tests from other machines concerned or only local tests on the server? What do the client logs say? greetings Rolf
▸
Hi, After Server Restart i get an error and all tests are purple. (except conn, http) i get the following error: 2007-01-18 08:28:37 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:28:39 Could not connect to user-88eecf256074@xymon.invalid:1984 - Connection refused what can i do? thx Stefan
--
Mit freundlichen Gruessen
Rolf Schrittenlocher
HRZ/BDV, Senckenberganlage 31, 60054 Frankfurt
Tel: (XX) XX - XXX XXXXX Fax: (XX) XX XXX XXXX
LBS: user-1e39a1813094@xymon.invalid
Persoenlich: user-6ea8e907e200@xymon.invalid
list Stefan Freisler
Hi Stefan,
Hi
▸
please give some more information. Restart of hobbit server or reboot of
the machine?
Of the machine. But if i try to restart hobbit server there is no other result.
Is the client running (it should start automatically while restarting the server but this doesn't work properly)?
Yes it runs.
▸
Are tests from other machines concerned or only local tests on the server?
No all other systems can send tests.
What do the client logs say?
=================================== cat clientlaunch.log|tail -n 30 2006-11-22 13:34:21 hobbitlaunch starting 2006-11-22 13:34:21 Loading tasklist configuration from /opt/hobbit/current/client/etc/clientlaunch.cfg 2006-11-22 16:32:47 hobbitlaunch starting 2006-11-22 16:32:47 Loading tasklist configuration from /opt/hobbit/current/client/etc/clientlaunch.cfg 2007-01-17 13:51:12 hobbitlaunch starting 2007-01-17 13:51:12 Loading tasklist configuration from /opt/hobbit/current/client/etc/clientlaunch.cfg 2007-01-17 15:19:16 hobbitlaunch starting 2007-01-17 15:19:16 Loading tasklist configuration from /opt/hobbit/current/client/etc/clientlaunch.cfg 2007-01-17 16:24:23 hobbitlaunch starting 2007-01-17 16:24:23 Loading tasklist configuration from /opt/hobbit/current/client/etc/clientlaunch.cfg 2007-01-17 16:26:37 hobbitlaunch starting 2007-01-17 16:26:37 Loading tasklist configuration from /opt/hobbit/current/client/etc/clientlaunch.cfg 2007-01-17 16:37:11 hobbitlaunch starting 2007-01-17 16:37:11 Loading tasklist configuration from /opt/hobbit/current/client/etc/clientlaunch.cfg 2007-01-18 08:28:36 hobbitlaunch starting 2007-01-18 08:28:36 Loading tasklist configuration from /opt/hobbit/current/client/etc/clientlaunch.cfg clientlaunch.log =================================== cat hobbitclient.log|tail -n 30 2007-01-18 08:53:57 Unknown token 'localhost:' ignored at line 351 2007-01-18 08:54:00 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 08:54:00 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:54:03 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 08:54:03 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:54:06 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 08:54:06 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:54:09 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 08:54:09 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:54:09 Failed to get a message, terminating 2007-01-18 08:59:01 Unknown token 'localhost:' ignored at line 351 2007-01-18 08:59:04 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 08:59:04 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:59:07 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 08:59:07 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:59:10 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 08:59:10 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:59:13 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 08:59:13 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:59:13 Failed to get a message, terminating 2007-01-18 09:04:04 Unknown token 'localhost:' ignored at line 351 2007-01-18 09:04:07 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 09:04:07 Whoops ! bb failed to send message - Connection failed 2007-01-18 09:04:10 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 09:04:10 Whoops ! bb failed to send message - Connection failed 2007-01-18 09:04:13 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 09:04:13 Whoops ! bb failed to send message - Connection failed 2007-01-18 09:04:16 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 09:04:16 Whoops ! bb failed to send message - Connection failed 2007-01-18 09:04:16 Failed to get a message, terminating cu Stefan
list Rolf Schrittenlocher
Hi Stefan, so I suppose your client is gathering data correctly (in the client/tmp directory) but cannot send it. What is striking me is the line: Unknown token 'localhost:' ignored at line 351 As all other clients can send to your server it seems that only the server's client is missing something. But unfortunately I have no idea what it could be. Maybe you should grep for localhost in the client-directories. regards Rolf
▸
Hi Stefan,Hiplease give some more information. Restart of hobbit server or reboot of the machine?Of the machine. But if i try to restart hobbit server there is no other result.Is the client running (it should start automatically while restarting the server but this doesn't work properly)?Yes it runs.Are tests from other machines concerned or only local tests on the server?No all other systems can send tests.What do the client logs say?=================================== cat clientlaunch.log|tail -n 30 2006-11-22 13:34:21 hobbitlaunch starting 2006-11-22 13:34:21 Loading tasklist configuration from /opt/hobbit/current/client/etc/clientlaunch.cfg 2006-11-22 16:32:47 hobbitlaunch starting 2006-11-22 16:32:47 Loading tasklist configuration from /opt/hobbit/current/client/etc/clientlaunch.cfg 2007-01-17 13:51:12 hobbitlaunch starting 2007-01-17 13:51:12 Loading tasklist configuration from /opt/hobbit/current/client/etc/clientlaunch.cfg 2007-01-17 15:19:16 hobbitlaunch starting 2007-01-17 15:19:16 Loading tasklist configuration from /opt/hobbit/current/client/etc/clientlaunch.cfg 2007-01-17 16:24:23 hobbitlaunch starting 2007-01-17 16:24:23 Loading tasklist configuration from /opt/hobbit/current/client/etc/clientlaunch.cfg 2007-01-17 16:26:37 hobbitlaunch starting 2007-01-17 16:26:37 Loading tasklist configuration from /opt/hobbit/current/client/etc/clientlaunch.cfg 2007-01-17 16:37:11 hobbitlaunch starting 2007-01-17 16:37:11 Loading tasklist configuration from /opt/hobbit/current/client/etc/clientlaunch.cfg 2007-01-18 08:28:36 hobbitlaunch starting 2007-01-18 08:28:36 Loading tasklist configuration from /opt/hobbit/current/client/etc/clientlaunch.cfg clientlaunch.log =================================== cat hobbitclient.log|tail -n 30 2007-01-18 08:53:57 Unknown token 'localhost:' ignored at line 351 2007-01-18 08:54:00 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 08:54:00 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:54:03 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 08:54:03 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:54:06 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 08:54:06 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:54:09 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 08:54:09 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:54:09 Failed to get a message, terminating 2007-01-18 08:59:01 Unknown token 'localhost:' ignored at line 351 2007-01-18 08:59:04 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 08:59:04 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:59:07 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 08:59:07 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:59:10 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 08:59:10 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:59:13 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 08:59:13 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:59:13 Failed to get a message, terminating 2007-01-18 09:04:04 Unknown token 'localhost:' ignored at line 351 2007-01-18 09:04:07 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 09:04:07 Whoops ! bb failed to send message - Connection failed 2007-01-18 09:04:10 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 09:04:10 Whoops ! bb failed to send message - Connection failed 2007-01-18 09:04:13 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 09:04:13 Whoops ! bb failed to send message - Connection failed 2007-01-18 09:04:16 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 09:04:16 Whoops ! bb failed to send message - Connection failed 2007-01-18 09:04:16 Failed to get a message, terminating cu Stefan
-- Mit freundlichen Gruessen Rolf Schrittenlocher HRZ/BDV, Senckenberganlage 31, 60054 Frankfurt Tel: (XX) XX - XXX XXXXX Fax: (XX) XX XXX XXXX LBS: user-1e39a1813094@xymon.invalid Persoenlich: user-6ea8e907e200@xymon.invalid
list Stefan Freisler
Hi, has noone an Idea? Here two more logs: hobbitlaunch.log: 2007-01-18 13:46:37 hobbitlaunch starting 2007-01-18 13:46:37 Loading tasklist configuration from /opt/hobbit/current/server/etc/hobbitlaunch.cfg 2007-01-18 13:46:38 Loading hostnames 2007-01-18 13:46:38 Loading saved state 2007-01-18 13:46:38 Setting up network listener on 0.0.0.0:1984 2007-01-18 13:46:38 Setting up signal handlers 2007-01-18 13:46:38 Setting up hobbitd channels 2007-01-18 13:46:38 Setting up logfiles bb-display.log 2006-11-15 16:13:20 Cannot create file Test///Test.html.tmp (in /var/www/html/hobbit): Too many open files in system 2006-11-15 16:13:20 Cannot create file bb2.html.tmp: Too many open files in system 2006-11-15 16:13:20 Cannot create file bbnk.html.tmp: Too many open files in system 2006-11-15 16:13:20 Whoops ! bb failed to send message - Cannot get a socket mfg Stefan user-75055581652a@xymon.invalid schrieb am 18.01.2007 08:35:55:
▸
Hi, After Server Restart i get an error and all tests are purple. ( except conn, http) i get the following error: 2007-01-18 08:28:37 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:28:39 Could not connect to user-88eecf256074@xymon.invalid:1984 - Connection refused what can i do? thx Stefan
list Gary Baluha
I think the bb-display.log file has your answer; the system can't open a socket connection to the BB (hobbit) server. Is some process running away with using up server resources? In our environment under Linux, we've had issues with auditd filling up the process table; maybe the same thing is happening on your machine? For us, this happens when /var fills up about 80 or 90% (I forget the exact number).
▸
Hi, has noone an Idea? Here two more logs: hobbitlaunch.log: 2007-01-18 13:46:37 hobbitlaunch starting 2007-01-18 13:46:37 Loading tasklist configuration from /opt/hobbit/current/server/etc/hobbitlaunch.cfg 2007-01-18 13:46:38 Loading hostnames 2007-01-18 13:46:38 Loading saved state 2007-01-18 13:46:38 Setting up network listener on 0.0.0.0:1984 2007-01-18 13:46:38 Setting up signal handlers 2007-01-18 13:46:38 Setting up hobbitd channels 2007-01-18 13:46:38 Setting up logfiles bb-display.log 2006-11-15 16:13:20 Cannot create file Test///Test.html.tmp (in /var/www/html/hobbit): Too many open files in system 2006-11-15 16:13:20 Cannot create file bb2.html.tmp: Too many open files in system 2006-11-15 16:13:20 Cannot create file bbnk.html.tmp: Too many open files in system 2006-11-15 16:13:20 Whoops ! bb failed to send message - Cannot get a socket mfg Stefan user-75055581652a@xymon.invalid schrieb am 18.01.2007 08:35:55:Hi, After Server Restart i get an error and all tests are purple. ( except conn, http) i get the following error: 2007-01-18 08:28:37 Whoops ! bb failed to send message - Connection failed 2007-01-18 08:28:39 Could not connect to user-88eecf256074@xymon.invalid:1984 - Connection refused what can i do? thx Stefan
list Greg L Hubbard
looks like a "ulimit" problem -- you need to permit more concurrent open filess. This could be a symptom of another problem -- like files not getting closed when they should. GLH From: user-75055581652a@xymon.invalid [mailto:user-75055581652a@xymon.invalid] Sent: Thursday, January 18, 2007 8:15 AM To: user-ae9b8668bcde@xymon.invalid Subject: [hobbit] Antwort: [hobbit] After Server Restart get an
▸
error and all tests are purple
Hi, has noone an Idea? Here two more logs:
hobbitlaunch.log: 2007-01-18 13:46:37 hobbitlaunch starting
2007-01-18 13:46:37 Loading tasklist configuration from
/opt/hobbit/current/server/etc/hobbitlaunch.cfg
2007-01-18 13:46:38 Loading hostnames
2007-01-18 13:46:38 Loading saved state
2007-01-18 13:46:38 Setting up network listener on 0.0.0.0:1984
2007-01-18 13:46:38 Setting up signal handlers
2007-01-18 13:46:38 Setting up hobbitd channels
2007-01-18 13:46:38 Setting up logfiles
bb-display.log 2006-11-15 16:13:20 Cannot create file Test///Test.html.tmp (in
/var/www/html/hobbit): Too many open files in system
2006-11-15 16:13:20 Cannot create file bb2.html.tmp: Too many
open files in system
2006-11-15 16:13:20 Cannot create file bbnk.html.tmp: Too many
open files in system
2006-11-15 16:13:20 Whoops ! bb failed to send message - Cannot
get a socket
mfg Stefan
user-75055581652a@xymon.invalid schrieb am 18.01.2007 08:35:55:
Hi, > After Server Restart i get an error and all tests are purple.
(
except conn, http) > > i get the following error: > 2007-01-18 08:28:37 Whoops ! bb failed to send message -
Connection failed > 2007-01-18 08:28:39 Could not connect to user-88eecf256074@xymon.invalid:1984 - Connection refused
what can i do? > > thx > Stefan
list Stefan Freisler
Hi, oh sorry! I see this log file was from November sorry my fault. But i have an other thing: /opt/hobbit/current/client/bin/bb --debug 10.28.1.10 "status bla,blab,bla,de.msgs gree Only a test" 2007-01-18 16:32:00 Transport setup is: 2007-01-18 16:32:00 bbdportnumber = 1984 2007-01-18 16:32:00 bbdispproxyhost = NONE 2007-01-18 16:32:00 bbdispproxyport = 0 2007-01-18 16:32:00 Recipient listed as '10.28.1.10' 2007-01-18 16:32:00 Standard BB protocol on port 1984 2007-01-18 16:32:00 Will connect to address 10.28.1.10 port 1984 2007-01-18 16:32:03 Connect status is 111 2007-01-18 16:32:03 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 16:32:03 Whoops ! bb failed to send message - Connection failed Can i somewhere read what status 111 means? thx a lot mfg Stefan
list Thomas Pedersen
Have you checked that the deamon is listening on port 1984 on the server 10.28.1.10 ? next thing could be to telnet from the client to the server on the port to check if any firewall/routing issues are not in the way.
▸
user-75055581652a@xymon.invalid wrote:Hi, oh sorry! I see this log file was from November sorry my fault. But i have an other thing: /opt/hobbit/current/client/bin/bb --debug 10.28.1.10 "status bla,blab,bla,de.msgs gree Only a test" 2007-01-18 16:32:00 Transport setup is: 2007-01-18 16:32:00 bbdportnumber = 1984 2007-01-18 16:32:00 bbdispproxyhost = NONE 2007-01-18 16:32:00 bbdispproxyport = 0 2007-01-18 16:32:00 Recipient listed as '10.28.1.10' 2007-01-18 16:32:00 Standard BB protocol on port 1984 2007-01-18 16:32:00 Will connect to address 10.28.1.10 port 1984 2007-01-18 16:32:03 Connect status is 111 2007-01-18 16:32:03 Could not connect to bbd at 10.28.1.10:1984 - Connection refused 2007-01-18 16:32:03 Whoops ! bb failed to send message - Connection failed Can i somewhere read what status 111 means? thx a lot mfg Stefan