Xymon Mailing List Archive search

conn test fails / comes back

12 messages in this thread

list Stephane Caminade · Fri, 21 Jul 2006 11:41:55 +0200 · 
Hi,

I just upgraded to RC, and noticed a very strange thing.
When I use fping OR hobbitping, the connection test goes red, then 
green, and cycles like that for all hosts.

I made sure I had no phantom process on the machine by killing 
everything owned by the hobbit user, then starting the server from 
scratch, and all tests are fine, except for the connection test.

Any idea where that might come from ?

Thanks,
Stephane

-- 
Stephane Caminade
Administrateur Systèmes et Réseaux
                                   \  <user-b664a384004a@xymon.invalid>
Institut d'Astrophysique Spatiale  /  tel : (XX) (X) XX XX XX XX
Batiment 121, Universite Paris XI  \  fax : (XX) (X) XX XX XX XX
F-91405 ORSAY Cedex                /  www : http://www.ias.u-psud.fr/
list Stephane Caminade · Tue, 25 Jul 2006 11:42:48 +0200 · 
Bump.

Just ran
~/BETA/server/bin$ ./bbtest-net  --ping --checkresponse --debug

All the servers have green status, however often i run this command.

I m at a loss as to the nature of the problem.
Stephane
▸ quoted from Stephane Caminade

Stephane Caminade wrote:
Hi,

I just upgraded to RC, and noticed a very strange thing.
When I use fping OR hobbitping, the connection test goes red, then 
green, and cycles like that for all hosts.

I made sure I had no phantom process on the machine by killing 
everything owned by the hobbit user, then starting the server from 
scratch, and all tests are fine, except for the connection test.

Any idea where that might come from ?

Thanks,
Stephane

-- 
Stephane Caminade
Administrateur Systèmes et Réseaux
                                   \  <user-b664a384004a@xymon.invalid>
Institut d'Astrophysique Spatiale  /  tel : (XX) (X) XX XX XX XX
Batiment 121, Universite Paris XI  \  fax : (XX) (X) XX XX XX XX
F-91405 ORSAY Cedex                /  www : http://www.ias.u-psud.fr/
list Nicolas Lienard · Tue, 25 Jul 2006 12:05:06 +0200 (CEST) · 
Hi,

Os: FreeBSD 5.1
Server Hobbit: 4.2-RC-20060712
Client Hobbit: 4.2-RC-20060712

I m trying to configure trigger on msgs check but don't manage :(

here the configuration on server side, maybe there is a mistake :

Goal: report yellow/red alerts on some keyword in /var/log/messages

1. hobbit-clients.cfg on server side

HOST=baldo.tou.nu
       [...]
       LOG %/var/log/messages kernel COLOR=yellow
       LOG %/var/log/messages %(I/O|read).error COLOR=red
       LOG %/var/log/messages diff-bkp-mx-dom COLOR=red

2. client-local.cfg on server side

[freebsd]
log:/var/log/maillog:10240
file:/usr/local/bin/bash:md5
log:/var/log/messages:10240

[baldo.tou.nu]
file:/usr/local/bin/bash:md5
log:/var/log/messages:10240
trigger diff-bkp-mx-dom
trigger kernel
log:/var/log/maillog:10240
ignore spamd
trigger deferred


3. The [green]  msgs column of the host reports this:

No entries in /var/log/messages

No entries in /var/log/maillog

Full log /var/log/messages
Jul 25 11:19:36 baldo sudo:   hobbit : TTY=unknown ;
PWD=/usr/home/hobbit/client ; USER=root ; COMMAND=/usr/bin/netstat -i -b
-n
Jul 25 11:19:36 baldo sudo:   hobbit : TTY=unknown ;
PWD=/usr/home/hobbit/client ; USER=root ; COMMAND=/usr/bin/netstat -s
Jul 25 11:19:36 baldo sudo:   hobbit : TTY=unknown ;
PWD=/usr/home/hobbit/client ; USER=root ; COMMAND=/usr/bin/netstat -na -f
inet
[...]
Jul 25 11:49:43 baldo sudo:   hobbit : TTY=unknown ;
PWD=/usr/home/hobbit/client ; USER=root ; COMMAND=/bin/ps -ax -ww -o
pid,ppid,user,start,state,pri,pcpu,cputime,pmem,rss,vsz,args
Jul 25 11:49:43 baldo sudo:   hobbit : TTY=unknown ;
PWD=/usr/home/hobbit/client ; USER=root ; COMMAND=/usr/bin/top -n 20

Full log /var/log/maillog
Cannot open logfile /var/log/maillog : Permission denied


I know there is a permission problem for maillog but i don 't care at this
time, i d like to monitore /var/log/messages at first ;-)

Thanks in advance.

Regards,
Nicolas
list David Gore · Tue, 25 Jul 2006 11:25:31 +0000 ·
▸ quoted from Nicolas Lienard

Nicolas wrote:
Hi,

Os: FreeBSD 5.1
Server Hobbit: 4.2-RC-20060712
Client Hobbit: 4.2-RC-20060712

I m trying to configure trigger on msgs check but don't manage :(

here the configuration on server side, maybe there is a mistake :

Goal: report yellow/red alerts on some keyword in /var/log/messages

1. hobbit-clients.cfg on server side

HOST=baldo.tou.nu
       [...]
       LOG %/var/log/messages kernel COLOR=yellow
       LOG %/var/log/messages %(I/O|read).error COLOR=red
       LOG %/var/log/messages diff-bkp-mx-dom COLOR=red

2. client-local.cfg on server side

[freebsd]
log:/var/log/maillog:10240
file:/usr/local/bin/bash:md5
log:/var/log/messages:10240

[baldo.tou.nu]
file:/usr/local/bin/bash:md5
log:/var/log/messages:10240
trigger diff-bkp-mx-dom
trigger kernel
log:/var/log/maillog:10240
ignore spamd
trigger deferred
  
I am not sure you can have multiple triggers?  How about something like this

trigger diff-bkp-mx-dom|kernel
▸ quoted from Nicolas Lienard
3. The [green]  msgs column of the host reports this:

No entries in /var/log/messages

No entries in /var/log/maillog

Full log /var/log/messages
Jul 25 11:19:36 baldo sudo:   hobbit : TTY=unknown ;
PWD=/usr/home/hobbit/client ; USER=root ; COMMAND=/usr/bin/netstat -i -b
-n
Jul 25 11:19:36 baldo sudo:   hobbit : TTY=unknown ;
PWD=/usr/home/hobbit/client ; USER=root ; COMMAND=/usr/bin/netstat -s
Jul 25 11:19:36 baldo sudo:   hobbit : TTY=unknown ;
PWD=/usr/home/hobbit/client ; USER=root ; COMMAND=/usr/bin/netstat -na -f
inet
[...]
Jul 25 11:49:43 baldo sudo:   hobbit : TTY=unknown ;
PWD=/usr/home/hobbit/client ; USER=root ; COMMAND=/bin/ps -ax -ww -o
pid,ppid,user,start,state,pri,pcpu,cputime,pmem,rss,vsz,args
Jul 25 11:49:43 baldo sudo:   hobbit : TTY=unknown ;
PWD=/usr/home/hobbit/client ; USER=root ; COMMAND=/usr/bin/top -n 20

Full log /var/log/maillog
Cannot open logfile /var/log/maillog : Permission denied


I know there is a permission problem for maillog but i don 't care at this
time, i d like to monitore /var/log/messages at first ;-)

Thanks in advance.

Regards,
Nicolas
list Nicolas Lienard · Tue, 25 Jul 2006 20:27:48 +0200 (CEST) ·
▸ quoted from David Gore
I am not sure you can have multiple triggers?  How about something like
this

trigger diff-bkp-mx-dom|kernel

i tried it:

[freebsd]
log:/var/log/maillog:10240
file:/usr/local/bin/bash:md5
log:/var/log/messages:10240

[baldo.tou.nu]
file:/usr/local/bin/bash:md5
log:/var/log/messages:10240
trigger kernel|diff-bkp-mx-dom
log:/var/log/maillog:10240
ignore spamd
trigger deferred

and

HOST=baldo.tou.nu
     LOG %/var/log/messages kernel COLOR=yellow
     LOG %/var/log/messages %(I/O|read).error COLOR=red


Unfortunately, there is the same result :

 System logs at Tue Jul 25 20:18:36 CEST 2006

No entries in /var/log/messages

No entries in /var/log/maillog


Full log /var/log/messages
Jul 25 19:48:32 baldo sudo:   hobbit : TTY=unknown ;
PWD=/usr/home/hobbit/client ; USER=root ; COMMAND=/usr/bin/netstat -i -b
-n
Jul 25 19:48:32 baldo sudo:   hobbit : TTY=unknown ;
▸ quoted from David Gore
PWD=/usr/home/hobbit/client ; USER=root ; COMMAND=/usr/bin/netstat -s
[...]

Full log /var/log/maillog
Cannot open logfile /var/log/maillog : Permission denied



Regards,
nicolas
list Stephane Caminade · Wed, 26 Jul 2006 10:22:45 +0200 · 
Just noticed on the bbtest page this little message :

Error output:
Could not run the command 'fping -Ae' (exec failed)
▸ quoted from Stephane Caminade


Stephane Caminade wrote:
Bump.

Just ran
~/BETA/server/bin$ ./bbtest-net  --ping --checkresponse --debug

All the servers have green status, however often i run this command.

I m at a loss as to the nature of the problem.
Stephane

Stephane Caminade wrote:
Hi,

I just upgraded to RC, and noticed a very strange thing.
When I use fping OR hobbitping, the connection test goes red, then 
green, and cycles like that for all hosts.

I made sure I had no phantom process on the machine by killing 
everything owned by the hobbit user, then starting the server from 
scratch, and all tests are fine, except for the connection test.

Any idea where that might come from ?

Thanks,
Stephane

-- 
Stephane Caminade
Administrateur Systèmes et Réseaux
                                   \  <user-b664a384004a@xymon.invalid>
Institut d'Astrophysique Spatiale  /  tel : (XX) (X) XX XX XX XX
Batiment 121, Universite Paris XI  \  fax : (XX) (X) XX XX XX XX
F-91405 ORSAY Cedex                /  www : http://www.ias.u-psud.fr/
list Kent Brodie · Wed, 26 Jul 2006 11:25:42 -0500 · 
Does your hobbit user account have privileges to run fping?     You'll need to set fping to suid to make that work..   

Kent C. Brodie - user-da7f7d5174c0@xymon.invalid
Department of Physiology
Medical College of Wisconsin
(XXX) XXX-XXXX
▸ quoted from Stephane Caminade
-----Original Message-----
From: Stephane Caminade [mailto:user-b664a384004a@xymon.invalid] 
Sent: Wednesday, July 26, 2006 3:23 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] conn test fails / comes back

Just noticed on the bbtest page this little message :

Error output:
Could not run the command 'fping -Ae' (exec failed)


Stephane Caminade wrote:
Bump.

Just ran
~/BETA/server/bin$ ./bbtest-net  --ping --checkresponse --debug

All the servers have green status, however often i run this command.

I m at a loss as to the nature of the problem.
Stephane

Stephane Caminade wrote:
Hi,

I just upgraded to RC, and noticed a very strange thing.
When I use fping OR hobbitping, the connection test goes red, then 
green, and cycles like that for all hosts.

I made sure I had no phantom process on the machine by killing 
everything owned by the hobbit user, then starting the server from 
scratch, and all tests are fine, except for the connection test.

Any idea where that might come from ?

Thanks,
Stephane

-- 
Stephane Caminade
Administrateur Systèmes et Réseaux
                                   \  <user-b664a384004a@xymon.invalid>
Institut d'Astrophysique Spatiale  /  tel : (XX) (X) XX XX XX XX
Batiment 121, Universite Paris XI  \  fax : (XX) (X) XX XX XX XX
F-91405 ORSAY Cedex                /  www : http://www.ias.u-psud.fr/
list Stephane Caminade · Thu, 27 Jul 2006 08:27:02 +0200 · 
I can run without any problem the bbtest-net command, under the hobbit account, and get all the fping results.
The fping file is suid as well.
And since once in a while it works and manages to collect results (from hobbit), I m not sure that is the problem.
▸ quoted from Kent Brodie

Brodie, Kent wrote:
Does your hobbit user account have privileges to run fping?     You'll need to set fping to suid to make that work..   
Kent C. Brodie - user-da7f7d5174c0@xymon.invalid
Department of Physiology
Medical College of Wisconsin
(XXX) XXX-XXXX
-----Original Message-----
From: Stephane Caminade [mailto:user-b664a384004a@xymon.invalid] Sent: Wednesday, July 26, 2006 3:23 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] conn test fails / comes back

Just noticed on the bbtest page this little message :

Error output:
Could not run the command 'fping -Ae' (exec failed)


Stephane Caminade wrote:
  
Bump.

Just ran
~/BETA/server/bin$ ./bbtest-net  --ping --checkresponse --debug

All the servers have green status, however often i run this command.

I m at a loss as to the nature of the problem.
Stephane

Stephane Caminade wrote:
    
Hi,

I just upgraded to RC, and noticed a very strange thing.
When I use fping OR hobbitping, the connection test goes red, then green, and cycles like that for all hosts.

I made sure I had no phantom process on the machine by killing everything owned by the hobbit user, then starting the server from scratch, and all tests are fine, except for the connection test.

Any idea where that might come from ?

Thanks,
Stephane
list Jerry Yu · Fri, 28 Jul 2006 10:58:03 -0400 · 
I have a different & similar problem. I had a cont= test for http.  it was a
bad link so the test never ever succeded.  It stayed as 'red' as expected.

When I change bb-alerts to email 'RECOVERED' as well, this test starts to
turn red->clear (with alert email saying the  same thing) at each check
cycle for a test which always fails.

More annoyinyly, this stayed true even if I removed the 'RECOVERED' from
hobbit-alerts.cfg, after day.
▸ quoted from Stephane Caminade

On 7/27/06, Stephane Caminade <user-b664a384004a@xymon.invalid> wrote:
I can run without any problem the bbtest-net command, under the hobbit
account, and get all the fping results.
The fping file is suid as well.
And since once in a while it works and manages to collect results (from
hobbit), I m not sure that is the problem.

Brodie, Kent wrote:
Does your hobbit user account have privileges to run fping?     You'll
need to set fping to suid to make that work..

Kent C. Brodie - user-da7f7d5174c0@xymon.invalid
Department of Physiology
Medical College of Wisconsin
(XXX) XXX-XXXX
-----Original Message-----
From: Stephane Caminade [mailto:user-b664a384004a@xymon.invalid]
Sent: Wednesday, July 26, 2006 3:23 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] conn test fails / comes back

Just noticed on the bbtest page this little message :

Error output:
Could not run the command 'fping -Ae' (exec failed)


Stephane Caminade wrote:

Bump.

Just ran
~/BETA/server/bin$ ./bbtest-net  --ping --checkresponse --debug

All the servers have green status, however often i run this command.

I m at a loss as to the nature of the problem.
Stephane

Stephane Caminade wrote:

Hi,

I just upgraded to RC, and noticed a very strange thing.
When I use fping OR hobbitping, the connection test goes red, then
green, and cycles like that for all hosts.

I made sure I had no phantom process on the machine by killing
everything owned by the hobbit user, then starting the server from
scratch, and all tests are fine, except for the connection test.

Any idea where that might come from ?

Thanks,
Stephane
list Richard Leyton · Fri, 28 Jul 2006 16:02:53 +0100 · 
Just a thought - You don't, perhaps, have a second hobbit server  that's perhaps not up to date in the same way? You can hit just this  sort of thing a few times where tests flip flop between green and  red, because the other server is sending a fail for some reason, but  the server you're checking is working just fine.

r.

--
Richard Leyton - user-787ca786c598@xymon.invalid
http://www.leyton.org
▸ quoted from Jerry Yu


On 28 Jul 2006, at 15:58, Jerry Yu wrote:

I have a different & similar problem. I had a cont= test for http.   it was a bad link so the test never ever succeded.  It stayed as  'red' as expected.

When I change bb-alerts to email 'RECOVERED' as well, this test  starts to turn red->clear (with alert email saying the  same thing)  at each check cycle for a test which always fails.

More annoyinyly, this stayed true even if I removed the 'RECOVERED'  from hobbit-alerts.cfg, after day.


On 7/27/06, Stephane Caminade < user-b664a384004a@xymon.invalid>  wrote:I can run without any problem the bbtest-net command, under  the hobbit
▸ quoted from Jerry Yu
account, and get all the fping results.
The fping file is suid as well.
And since once in a while it works and manages to collect results  (from
hobbit), I m not sure that is the problem.

Brodie, Kent wrote:
Does your hobbit user account have privileges to run fping?      You'll need to set fping to suid to make that work..

Kent C. Brodie - user-da7f7d5174c0@xymon.invalid
Department of Physiology
Medical College of Wisconsin
(XXX) XXX-XXXX
-----Original Message-----
From: Stephane Caminade [mailto:user-b664a384004a@xymon.invalid]
Sent: Wednesday, July 26, 2006 3:23 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] conn test fails / comes back

Just noticed on the bbtest page this little message :

Error output:
Could not run the command 'fping -Ae' (exec failed)


Stephane Caminade wrote:

Bump.

Just ran
~/BETA/server/bin$ ./bbtest-net  --ping --checkresponse --debug

All the servers have green status, however often i run this  command.

I m at a loss as to the nature of the problem.
Stephane

Stephane Caminade wrote:

Hi,

I just upgraded to RC, and noticed a very strange thing.
When I use fping OR hobbitping, the connection test goes red, then
green, and cycles like that for all hosts.

I made sure I had no phantom process on the machine by killing
everything owned by the hobbit user, then starting the server from
scratch, and all tests are fine, except for the connection test.

Any idea where that might come from ?

Thanks,
Stephane
list Jerry Yu · Fri, 28 Jul 2006 11:15:02 -0400 · 
interesting. Let me think. emm, no, I have only one hobbit server. There's
another bigsister server nearby. however, friewall rule disallows hobbit
server to receive inbound connection from the bigsister. moreover, the
client has only hobbit-client installed. the http test
(cont=http;http://blah.host/) is only configured for this host in
hobbit-server's bb-hosts.
▸ quoted from Richard Leyton

On 7/28/06, Richard Leyton <user-787ca786c598@xymon.invalid> wrote:
Just a thought - You don't, perhaps, have a second hobbit server
that's perhaps not up to date in the same way? You can hit just this
sort of thing a few times where tests flip flop between green and
red, because the other server is sending a fail for some reason, but
the server you're checking is working just fine.

r.

--
Richard Leyton - user-787ca786c598@xymon.invalid
http://www.leyton.org


On 28 Jul 2006, at 15:58, Jerry Yu wrote:

I have a different & similar problem. I had a cont= test for http.
it was a bad link so the test never ever succeded.  It stayed as
'red' as expected.

When I change bb-alerts to email 'RECOVERED' as well, this test
starts to turn red->clear (with alert email saying the  same thing)
at each check cycle for a test which always fails.

More annoyinyly, this stayed true even if I removed the 'RECOVERED'
from hobbit-alerts.cfg, after day.

On 7/27/06, Stephane Caminade < user-b664a384004a@xymon.invalid>
wrote:I can run without any problem the bbtest-net command, under
the hobbit
account, and get all the fping results.
The fping file is suid as well.
And since once in a while it works and manages to collect results
(from
hobbit), I m not sure that is the problem.

Brodie, Kent wrote:
Does your hobbit user account have privileges to run fping?
You'll need to set fping to suid to make that work..
Kent C. Brodie - user-da7f7d5174c0@xymon.invalid
Department of Physiology
Medical College of Wisconsin
(XXX) XXX-XXXX
-----Original Message-----
From: Stephane Caminade [mailto:user-b664a384004a@xymon.invalid]
Sent: Wednesday, July 26, 2006 3:23 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] conn test fails / comes back

Just noticed on the bbtest page this little message :

Error output:
Could not run the command 'fping -Ae' (exec failed)


Stephane Caminade wrote:

Bump.

Just ran
~/BETA/server/bin$ ./bbtest-net  --ping --checkresponse --debug

All the servers have green status, however often i run this
command.

I m at a loss as to the nature of the problem.
Stephane

Stephane Caminade wrote:

Hi,

I just upgraded to RC, and noticed a very strange thing.
When I use fping OR hobbitping, the connection test goes red, then
green, and cycles like that for all hosts.

I made sure I had no phantom process on the machine by killing
everything owned by the hobbit user, then starting the server from
scratch, and all tests are fine, except for the connection test.

Any idea where that might come from ?

Thanks,
Stephane
list Stephane Caminade · Fri, 28 Jul 2006 19:39:46 +0200 · 
Ok, that was the source of my problem.
I had tried to set up unsuccessfuly a secondary hobbit server, and had given it the same configuration file as my original one.
But, and there I hadn t been careful, I hadn t given the secondary server the same network rights as the original one, and therefore it reported that all hosts were down.

Thanks for the "thought" !

Stephane
▸ quoted from Richard Leyton


Richard Leyton wrote:
Just a thought - You don't, perhaps, have a second hobbit server that's perhaps not up to date in the same way? You can hit just this sort of thing a few times where tests flip flop between green and red, because the other server is sending a fail for some reason, but the server you're checking is working just fine.

r.

-- 
Richard Leyton - user-787ca786c598@xymon.invalid
http://www.leyton.org


On 28 Jul 2006, at 15:58, Jerry Yu wrote:

I have a different & similar problem. I had a cont= test for http.  it was a bad link so the test never ever succeded.  It stayed as 'red' as expected.

When I change bb-alerts to email 'RECOVERED' as well, this test starts to turn red->clear (with alert email saying the  same thing) at each check cycle for a test which always fails.

More annoyinyly, this stayed true even if I removed the 'RECOVERED' from hobbit-alerts.cfg, after day.

On 7/27/06, Stephane Caminade < user-b664a384004a@xymon.invalid> wrote:I can run without any problem the bbtest-net command, under the hobbit
account, and get all the fping results.
The fping file is suid as well.
And since once in a while it works and manages to collect results (from
hobbit), I m not sure that is the problem.

Brodie, Kent wrote:
Does your hobbit user account have privileges to run fping?     You'll need to set fping to suid to make that work..

Kent C. Brodie - user-da7f7d5174c0@xymon.invalid
Department of Physiology
Medical College of Wisconsin
(XXX) XXX-XXXX
-----Original Message-----
From: Stephane Caminade [mailto:user-b664a384004a@xymon.invalid]
Sent: Wednesday, July 26, 2006 3:23 AM
To: user-ae9b8668bcde@xymon.invalid
Subject: Re: [hobbit] conn test fails / comes back

Just noticed on the bbtest page this little message :

Error output:
Could not run the command 'fping -Ae' (exec failed)


Stephane Caminade wrote:

Bump.

Just ran
~/BETA/server/bin$ ./bbtest-net  --ping --checkresponse --debug

All the servers have green status, however often i run this command.

I m at a loss as to the nature of the problem.
Stephane

Stephane Caminade wrote:

Hi,

I just upgraded to RC, and noticed a very strange thing.
When I use fping OR hobbitping, the connection test goes red, then
green, and cycles like that for all hosts.

I made sure I had no phantom process on the machine by killing
everything owned by the hobbit user, then starting the server from
scratch, and all tests are fine, except for the connection test.

Any idea where that might come from ?

Thanks,
Stephane