xymonnet crashing after upgrade to 4.3.15

10 messages in this thread

list Johan Sjöberg · Tue, 4 Feb 2014 21:17:41 +0000 ·

Hi.

Updated our server from 4.3.13 to 4.3.15 today, and after that xymonnet has crashed a few times. Is it possible to enable some debug log or something to find out what happens?

The error in the xymonnet test is:

"red - Program crashed

Fatal signal caught!"

/Johan

list Henrik Størner · Tue, 04 Feb 2014 22:22:59 +0100 ·

▸ quoted from Johan Sjöberg

Den 04-02-2014 22:17, Johan Sjöberg skrev:

Hi.

Updated our server from 4.3.13 to 4.3.15 today, and after that xymonnet
has crashed a few times. Is it possible to enable some debug log or
something to find out what happens?

The error in the xymonnet test is:

“red - Program crashed

I have a couple of fixes, but none that really relate to xymonnet. You 
should get a 'core' file in ~xymon/server/tmp/ - if you do, could you 
try getting a stack trace using this procedure: 
http://www.xymon.com/xymon/help/known-issues.html#bugreport


Regards,
Henrik

list Johan Sjöberg · Wed, 5 Feb 2014 08:07:00 +0000 ·

-----Original Message-----
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Henrik
Størner
Sent: den 4 februari 2014 22:23
To: xymon at xymon.com
Subject: Re: [Xymon] xymonnet crashing after upgrade to 4.3.15

▸ quoted from Johan Sjöberg


Den 04-02-2014 22:17, Johan Sjöberg skrev:

Hi.

Updated our server from 4.3.13 to 4.3.15 today, and after that
xymonnet has crashed a few times. Is it possible to enable some debug
log or something to find out what happens?

The error in the xymonnet test is:

"red - Program crashed

I have a couple of fixes, but none that really relate to xymonnet. You should
get a 'core' file in ~xymon/server/tmp/ - if you do, could you try getting a
stack trace using this procedure:
http://www.xymon.com/xymon/help/known-issues.html#bugreport


Regards,
Henrik

Hi.

I ran gdb on the core file and got the output below.
We also see the same crashes on our staging xymon server after the upgrade.

mon01:/usr/local/xymon/server/tmp# gdb /usr/local/xymon/server/bin/xymonnet core
GNU gdb (GDB) 7.0.1-debian
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>;...
Reading symbols from /usr/local/xymon/server/bin/xymonnet...done.

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /usr/lib/libcares.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libcares.so.2
Reading symbols from /usr/lib/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libldap_r-2.4.so.2
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /usr/lib/libssl.so.0.9.8...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libssl.so.0.9.8
Reading symbols from /usr/lib/libcrypto.so.0.9.8...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libcrypto.so.0.9.8
Reading symbols from /lib/libpcre.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib/libpcre.so.3
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /usr/lib/liblber-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/liblber-2.4.so.2
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /usr/lib/libgnutls.so.26...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgnutls.so.26
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /usr/lib/libtasn1.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libtasn1.so.3
Reading symbols from /usr/lib/libgcrypt.so.11...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgcrypt.so.11
Reading symbols from /usr/lib/libgpg-error.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgpg-error.so.0
Core was generated by `/usr/local/xymon/server/bin/xymonnet --ping --checkresponse asx-b5.xxxx.d'.
Program terminated with signal 6, Aborted.
#0  0x00007f66ca0171b5 in raise () from /lib/libc.so.6
(gdb) bt
#0  0x00007f66ca0171b5 in raise () from /lib/libc.so.6
#1  0x00007f66ca019fc0 in abort () from /lib/libc.so.6
#2  0x000000000041ed43 in sigsegv_handler (signum=<value optimized out>) at sig.c:57
#3  <signal handler called>
#4  0x00000000004239dc in xmh_find_item (host=0xbbad20, item=<value optimized out>) at loadhosts.c:226
#5  0x000000000040fefe in send_content_results (httptest=<value optimized out>, host=<value optimized out>, nonetpage=<value optimized out>,
    contenttestname=<value optimized out>, failgoesclear=<value optimized out>) at httpresult.c:456
#6  0x000000000040a8ed in main (argc=6, argv=<value optimized out>) at xymonnet.c:2484

/Johan

list Henrik Størner · Wed, 05 Feb 2014 09:13:39 +0100 ·

Hi Johan,

▸ quoted from Johan Sjöberg


Den 2014-02-05 9:07, Johan Sjöberg skrev:

#2  0x000000000041ed43 in sigsegv_handler (signum=<value optimized
out>) at sig.c:57
#3  <signal handler called>
#4  0x00000000004239dc in xmh_find_item (host=0xbbad20, item=<value
optimized out>) at loadhosts.c:226
#5  0x000000000040fefe in send_content_results (httptest=<value
optimized out>, host=<value optimized out>, nonetpage=<value 
optimized
out>,
    contenttestname=<value optimized out>, failgoesclear=<value
optimized out>) at httpresult.c:456

could you try this patch, please?


Regards,
Henrik

Attachments (1)

attachment.ksh application/octet-stream · 1.6 KB

list Johan Sjöberg · Wed, 5 Feb 2014 08:24:47 +0000 ·

▸ quoted from Henrik Størner

-----Original Message-----
From: user-ce4a2c883f75@xymon.invalid [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: den 5 februari 2014 09:14
To: Johan Sjöberg
Cc: xymon at xymon.com
Subject: RE: [Xymon] xymonnet crashing after upgrade to 4.3.15

Hi Johan,

Den 2014-02-05 9:07, Johan Sjöberg skrev:

#2  0x000000000041ed43 in sigsegv_handler (signum=<value optimized
out>) at sig.c:57
#3  <signal handler called>
#4  0x00000000004239dc in xmh_find_item (host=0xbbad20, item=<value
optimized out>) at loadhosts.c:226
#5  0x000000000040fefe in send_content_results (httptest=<value
optimized out>, host=<value optimized out>, nonetpage=<value optimized
out>,
    contenttestname=<value optimized out>, failgoesclear=<value
optimized out>) at httpresult.c:456

could you try this patch, please?


Regards,
Henrik

Thanks. Can I just copy the new xymonnet binary after compiling, and skip the make install?

/Johan

list Henrik Størner · Wed, 05 Feb 2014 09:50:20 +0100 ·

Den 2014-02-05 9:24, Johan Sjöberg skrev:

-----Original Message-----
From: user-ce4a2c883f75@xymon.invalid [mailto:user-ce4a2c883f75@xymon.invalid]

▸ quoted from Johan Sjöberg

could you try this patch, please?

Thanks. Can I just copy the new xymonnet binary after compiling, and
skip the make install?

Yes, that will work fine.


Regards,
Henrik

list Johan Sjöberg · Wed, 5 Feb 2014 10:00:11 +0000 ·

▸ quoted from Henrik Størner

-----Original Message-----
From: user-ce4a2c883f75@xymon.invalid [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: den 5 februari 2014 09:50
To: Johan Sjöberg
Cc: xymon at xymon.com
Subject: RE: [Xymon] xymonnet crashing after upgrade to 4.3.15

Den 2014-02-05 9:24, Johan Sjöberg skrev:

-----Original Message-----
From: user-ce4a2c883f75@xymon.invalid [mailto:user-ce4a2c883f75@xymon.invalid] could you try this
patch, please?

Thanks. Can I just copy the new xymonnet binary after compiling, and
skip the make install?

Yes, that will work fine.


Regards,
Henrik

Ok, I have applied it on one of our machines. I'll wait and see if it makes any difference. The crashes are not too frequent, it has happened 3-4 times on either server after the upgrade.

/Johan

list Gavin Stone-Tolcher · Thu, 6 Feb 2014 04:23:17 +0000 ·

Just reporting that I too have seen 1 x core dump from xymonnet on our test xymon setup since upgrading to 4.3.15 on Monday. Relevant gdb output follows:

[xymon at aztec server]$ gdb bin/xymonnet tmp/core.30031
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
....
....
Core was generated by `xymonnet --report --ping --ping-tasks=5 --test-untagged --concurrency=64 --cont'.
Program terminated with signal 6, Aborted.
....
 (gdb) bt
#0  0x0000003cfea32925 in raise () from /lib64/libc.so.6
#1  0x0000003cfea34105 in abort () from /lib64/libc.so.6
#2  0x0000000000426433 in sigsegv_handler (signum=<value optimized out>) at sig.c:57
#3  <signal handler called>
#4  0x000000000042b07c in xmh_find_item (host=<value optimized out>, item=XMH_FLAG_HTTP_HEADER_MATCH) at loadhosts.c:226
#5  0x000000000040f84e in send_content_results (httptest=<value optimized out>, host=<value optimized out>, nonetpage=<value optimized out>, contenttestname=0x967030 "cont", 
    failgoesclear=1) at httpresult.c:456
#6  0x000000000040a26d in main (argc=14, argv=0x7fffaf67cce8) at xymonnet.c:2484

I have also applied the supplied patch and will report any issues.

Cheers, 
Gavin Stone-Tolcher, IT Support Officer, Network Operations and Incident Response
Information Technology Services
The University of Queensland
Level 4, Prentice Building, St Lucia 4072
T: +61 7 334 66645, M: +61 401 140 838
E: user-6e44775a32e5@xymon.invalid W: www.its.uq.edu.au

ITS: Service. Team. Accountability. Results.

IMPORTANT: This email and any attachments are intended solely for the addressee(s), contain copyright material and are confidential. We do not waive any legal privilege or rights in respect of copyright or confidentiality. Except as intended addressees are otherwise permitted, you do not have permission to use, disclose, reproduce or communicate any part of this email or its attachments. Statements, opinions and information not related to the official business of The University of Queensland are neither given nor endorsed by us. By using this email (including accessing any attachments or links) you agree we are not liable for any loss or damage of any kind arising in connection with any electronic defect, virus or other malicious code we did not intentionally include.

Please consider the environment before printing this email.

CRICOS Code 00025B

▸ quoted from Johan Sjöberg


-----Original Message-----
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Johan Sjöberg
Sent: Wednesday, 5 February 2014 8:00 PM
To: user-ce4a2c883f75@xymon.invalid
Cc: xymon at xymon.com
Subject: Re: [Xymon] xymonnet crashing after upgrade to 4.3.15

-----Original Message-----
From: user-ce4a2c883f75@xymon.invalid [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: den 5 februari 2014 09:50
To: Johan Sjöberg
Cc: xymon at xymon.com
Subject: RE: [Xymon] xymonnet crashing after upgrade to 4.3.15

Den 2014-02-05 9:24, Johan Sjöberg skrev:

-----Original Message-----
From: user-ce4a2c883f75@xymon.invalid [mailto:user-ce4a2c883f75@xymon.invalid] could you try this 
patch, please?

Thanks. Can I just copy the new xymonnet binary after compiling, and 
skip the make install?

Yes, that will work fine.


Regards,
Henrik

Ok, I have applied it on one of our machines. I'll wait and see if it makes any difference. The crashes are not too frequent, it has happened 3-4 times on either server after the upgrade.

/Johan

list Johan Sjöberg · Thu, 6 Feb 2014 08:53:30 +0000 ·

Hi.

Just wanted to report that I installed the patch yesterday, and have not seen any crash since then.

/Johan

-----Original Message-----
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Gavin
Stone-Tolcher
Sent: den 6 februari 2014 05:23

▸ quoted from Gavin Stone-Tolcher

To: xymon at xymon.com
Subject: Re: [Xymon] xymonnet crashing after upgrade to 4.3.15

Just reporting that I too have seen 1 x core dump from xymonnet on our test
xymon setup since upgrading to 4.3.15 on Monday. Relevant gdb output
follows:

[xymon at aztec server]$ gdb bin/xymonnet tmp/core.30031 GNU gdb (GDB)
Red Hat Enterprise Linux (7.2-60.el6_4.1) ....
....


Core was generated by `xymonnet --report --ping --ping-tasks=5 --test-
untagged --concurrency=64 --cont'.

▸ quoted from Gavin Stone-Tolcher

Program terminated with signal 6, Aborted.
....
 (gdb) bt
#0  0x0000003cfea32925 in raise () from /lib64/libc.so.6
#1  0x0000003cfea34105 in abort () from /lib64/libc.so.6
#2  0x0000000000426433 in sigsegv_handler (signum=<value optimized out>)
at sig.c:57
#3  <signal handler called>
#4  0x000000000042b07c in xmh_find_item (host=<value optimized out>,
item=XMH_FLAG_HTTP_HEADER_MATCH) at loadhosts.c:226
#5  0x000000000040f84e in send_content_results (httptest=<value optimized
out>, host=<value optimized out>, nonetpage=<value optimized out>,
contenttestname=0x967030 "cont",
    failgoesclear=1) at httpresult.c:456
#6  0x000000000040a26d in main (argc=14, argv=0x7fffaf67cce8) at
xymonnet.c:2484

I have also applied the supplied patch and will report any issues.

Cheers,
Gavin Stone-Tolcher, IT Support Officer, Network Operations and Incident
Response Information Technology Services The University of Queensland
Level 4, Prentice Building, St Lucia 4072
T: +61 7 334 66645, M: +61 401 140 838
E: user-6e44775a32e5@xymon.invalid W: www.its.uq.edu.au

ITS: Service. Team. Accountability. Results.

IMPORTANT: This email and any attachments are intended solely for the
addressee(s), contain copyright material and are confidential. We do not
waive any legal privilege or rights in respect of copyright or confidentiality.
Except as intended addressees are otherwise permitted, you do not have
permission to use, disclose, reproduce or communicate any part of this email
or its attachments. Statements, opinions and information not related to the
official business of The University of Queensland are neither given nor
endorsed by us. By using this email (including accessing any attachments or
links) you agree we are not liable for any loss or damage of any kind arising in
connection with any electronic defect, virus or other malicious code we did
not intentionally include.

Please consider the environment before printing this email.

CRICOS Code 00025B

-----Original Message-----
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Johan
Sjöberg
Sent: Wednesday, 5 February 2014 8:00 PM
To: user-ce4a2c883f75@xymon.invalid
Cc: xymon at xymon.com
Subject: Re: [Xymon] xymonnet crashing after upgrade to 4.3.15

-----Original Message-----
From: user-ce4a2c883f75@xymon.invalid [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: den 5 februari 2014 09:50
To: Johan Sjöberg
Cc: xymon at xymon.com
Subject: RE: [Xymon] xymonnet crashing after upgrade to 4.3.15

Den 2014-02-05 9:24, Johan Sjöberg skrev:

-----Original Message-----
From: user-ce4a2c883f75@xymon.invalid [mailto:user-ce4a2c883f75@xymon.invalid] could you try this
patch, please?

Thanks. Can I just copy the new xymonnet binary after compiling, and
skip the make install?

Yes, that will work fine.


Regards,
Henrik

Ok, I have applied it on one of our machines. I'll wait and see if it makes any
difference. The crashes are not too frequent, it has happened 3-4 times on
either server after the upgrade.

/Johan

list Dominique Frise · Thu, 06 Feb 2014 10:14:56 +0100 ·

We've seen this also.

[bb at prdexp server]$ gdb bin/xymonnet tmp/core.64739
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
...
...
Core was generated by `xymonnet --report --ping --checkresponse'.
Program terminated with signal 6, Aborted.
...
(gdb) bt
#0  0x00000035bdc32925 in raise () from /lib64/libc.so.6
#1  0x00000035bdc34105 in abort () from /lib64/libc.so.6
#2  0x000000000041e5c3 in sigsegv_handler (signum=<value optimized out>)
     at sig.c:57
#3  <signal handler called>
#4  0x000000000042320c in xmh_find_item (host=<value optimized out>,
     item=XMH_FLAG_HTTP_HEADER_MATCH) at loadhosts.c:226
#5  0x000000000040f7de in send_content_results (
     httptest=<value optimized out>, host=<value optimized out>,
     nonetpage=<value optimized out>, contenttestname=0x42abde "content",
     failgoesclear=1) at httpresult.c:456
#6  0x000000000040a1fd in main (argc=4, argv=0x7fff3c97eaf8) at 
xymonnet.c:2484


Dominique

▸ quoted from Gavin Stone-Tolcher



On 02/06/2014 05:23 AM, Gavin Stone-Tolcher wrote:

Just reporting that I too have seen 1 x core dump from xymonnet on our test xymon setup since upgrading to 4.3.15 on Monday. Relevant gdb output follows:

[xymon at aztec server]$ gdb bin/xymonnet tmp/core.30031
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
....
....
Core was generated by `xymonnet --report --ping --ping-tasks=5 --test-untagged --concurrency=64 --cont'.
Program terminated with signal 6, Aborted.
....
  (gdb) bt
#0  0x0000003cfea32925 in raise () from /lib64/libc.so.6
#1  0x0000003cfea34105 in abort () from /lib64/libc.so.6
#2  0x0000000000426433 in sigsegv_handler (signum=<value optimized out>) at sig.c:57
#3  <signal handler called>
#4  0x000000000042b07c in xmh_find_item (host=<value optimized out>, item=XMH_FLAG_HTTP_HEADER_MATCH) at loadhosts.c:226
#5  0x000000000040f84e in send_content_results (httptest=<value optimized out>, host=<value optimized out>, nonetpage=<value optimized out>, contenttestname=0x967030 "cont",
     failgoesclear=1) at httpresult.c:456
#6  0x000000000040a26d in main (argc=14, argv=0x7fffaf67cce8) at xymonnet.c:2484

I have also applied the supplied patch and will report any issues.

Cheers,
Gavin Stone-Tolcher, IT Support Officer, Network Operations and Incident Response
Information Technology Services
The University of Queensland
Level 4, Prentice Building, St Lucia 4072
T: +61 7 334 66645, M: +61 401 140 838
E: user-6e44775a32e5@xymon.invalid W: www.its.uq.edu.au

ITS: Service. Team. Accountability. Results.

IMPORTANT: This email and any attachments are intended solely for the addressee(s), contain copyright material and are confidential. We do not waive any legal privilege or rights in respect of copyright or confidentiality. Except as intended addressees are otherwise permitted, you do not have permission to use, disclose, reproduce or communicate any part of this email or its attachments. Statements, opinions and information not related to the official business of The University of Queensland are neither given nor endorsed by us. By using this email (including accessing any attachments or links) you agree we are not liable for any loss or damage of any kind arising in connection with any electronic defect, virus or other malicious code we did not intentionally include.

Please consider the environment before printing this email.

CRICOS Code 00025B

-----Original Message-----
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Johan Sjöberg
Sent: Wednesday, 5 February 2014 8:00 PM
To: user-ce4a2c883f75@xymon.invalid
Cc: xymon at xymon.com
Subject: Re: [Xymon] xymonnet crashing after upgrade to 4.3.15

-----Original Message-----
From: user-ce4a2c883f75@xymon.invalid [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: den 5 februari 2014 09:50
To: Johan Sjöberg
Cc: xymon at xymon.com
Subject: RE: [Xymon] xymonnet crashing after upgrade to 4.3.15

Den 2014-02-05 9:24, Johan Sjöberg skrev:

-----Original Message-----
From: user-ce4a2c883f75@xymon.invalid [mailto:user-ce4a2c883f75@xymon.invalid] could you try this
patch, please?

Thanks. Can I just copy the new xymonnet binary after compiling, and
skip the make install?

Yes, that will work fine.


Regards,
Henrik

Ok, I have applied it on one of our machines. I'll wait and see if it makes any difference. The crashes are not too frequent, it has happened 3-4 times on either server after the upgrade.

/Johan

xymonnet crashing after upgrade to 4.3.15 🔗 link

xymonnet crashing after upgrade to 4.3.15