Scheduled disable causes crash? 🔗 link

From: Johan Sjöberg [mailto:user-74c177c1220d@xymon.invalid]
Sent: den 5 oktober 2010 11:10
To: xymon at xymon.com
Subject: [xymon] Scheduled disable causes crash?

Hi!

During the last month, we have had some problems with Xymon when using scheduled disabled (added from the web interface).

The first problem we had was on September 17th, when hobbitd crashed while/after running the scheduled disable. We got the following error in hobbit.log
2010-09-17 05:00:00 Fatal error in select: Bad file descriptor
2010-09-17 05:00:00 Setup complete


After that incident, I enabled verbose logging for hobbitd, and turned up the logging for some other process as well. This morning at 06:00 we had a new scheduled disable. This time hobbitd just stopped logging after running the disable. The web interface did not work correctly. When clicking a test, an "Internal server error" message was displayed. Also, the bbgen test went purple (last update at 05:59:26).

The only errors I have been able to find in the logs are in bb-display.log:
2010-10-05 06:00:26 xstrdup: Cannot dup NULL string
2010-10-05 06:01:26 xstrdup: Cannot dup NULL string
2010-10-05 06:02:27 xstrdup: Cannot dup NULL string
2010-10-05 06:12:30 xstrdup: Cannot dup NULL string
2010-10-05 06:13:35 xstrdup: Cannot dup NULL string
2010-10-05 06:14:39 xstrdup: Cannot dup NULL string
2010-10-05 06:24:40 xstrdup: Cannot dup NULL string
2010-10-05 06:25:41 xstrdup: Cannot dup NULL string
2010-10-05 06:26:42 xstrdup: Cannot dup NULL string
2010-10-05 06:36:47 xstrdup: Cannot dup NULL string
2010-10-05 06:37:48 xstrdup: Cannot dup NULL string
2010-10-05 06:39:54 xstrdup: Cannot dup NULL string
2010-10-05 06:40:54 xstrdup: Cannot dup NULL string
2010-10-05 06:41:54 xstrdup: Cannot dup NULL string

Xymon was restarted at 06:43.
Here is the hobbit.log from the time. It was too large to paste in the mail.
http://pastebin.com/JuU0BHje

We are running Xymon 4.2.3 on CentOS 5.

Best regards,
Johan Sjöberg

Reading symbols from /lib/libpcre.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpcre.so.0
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Core was generated by `bbgen --recentgifs --subpagecolumns=2 --report'.
Program terminated with signal 6, Aborted.

/Johan

From: Johan Sjöberg [mailto:user-74c177c1220d@xymon.invalid]
Sent: den 20 oktober 2010 12:32
To: xymon at xymon.com
Subject: [xymon] RE: Scheduled disable causes crash?

Hi.

This happened again yesterday morning. We found that core dumps had been created. Here is what gdb tells us about the core dumps. Does anyone have a clue of what might be causing this problem? The Xymon server is running CentOS 5.5 32-bit .

[root at mon01 acks]# file core.28581
core.28581: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from 'bbgen'

[root at mon01 acks]# gdb /usr/local/xymon/server/bin/bbgen core.28581
Reading symbols from /usr/local/xymon/server/bin/bbgen...done.

warning: .dynamic section for "/lib/libc.so.6" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations
Reading symbols from /lib/libpcre.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpcre.so.0
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Core was generated by `bbgen --recentgifs --subpagecolumns=2 --report'.
Program terminated with signal 6, Aborted.
#0 0x00b2f402 in __kernel_vsyscall ()

/Johan

From: Johan Sjöberg [mailto:user-74c177c1220d@xymon.invalid]
Sent: den 5 oktober 2010 11:10
To: xymon at xymon.com
Subject: [xymon] Scheduled disable causes crash?

Hi!

During the last month, we have had some problems with Xymon when using scheduled disabled (added from the web interface).

The first problem we had was on September 17th, when hobbitd crashed while/after running the scheduled disable. We got the following error in hobbit.log
2010-09-17 05:00:00 Fatal error in select: Bad file descriptor
2010-09-17 05:00:00 Setup complete


After that incident, I enabled verbose logging for hobbitd, and turned up the logging for some other process as well. This morning at 06:00 we had a new scheduled disable. This time hobbitd just stopped logging after running the disable. The web interface did not work correctly. When clicking a test, an "Internal server error" message was displayed. Also, the bbgen test went purple (last update at 05:59:26).

The only errors I have been able to find in the logs are in bb-display.log:
2010-10-05 06:00:26 xstrdup: Cannot dup NULL string
2010-10-05 06:01:26 xstrdup: Cannot dup NULL string
2010-10-05 06:02:27 xstrdup: Cannot dup NULL string
2010-10-05 06:12:30 xstrdup: Cannot dup NULL string
2010-10-05 06:13:35 xstrdup: Cannot dup NULL string
2010-10-05 06:14:39 xstrdup: Cannot dup NULL string
2010-10-05 06:24:40 xstrdup: Cannot dup NULL string
2010-10-05 06:25:41 xstrdup: Cannot dup NULL string
2010-10-05 06:26:42 xstrdup: Cannot dup NULL string
2010-10-05 06:36:47 xstrdup: Cannot dup NULL string
2010-10-05 06:37:48 xstrdup: Cannot dup NULL string
2010-10-05 06:39:54 xstrdup: Cannot dup NULL string
2010-10-05 06:40:54 xstrdup: Cannot dup NULL string
2010-10-05 06:41:54 xstrdup: Cannot dup NULL string

Xymon was restarted at 06:43.
Here is the hobbit.log from the time. It was too large to paste in the mail.
http://pastebin.com/JuU0BHje

We are running Xymon 4.2.3 on CentOS 5.

Best regards,
Johan Sjöberg

2010-09-17 05:00:00 Fatal error in select: Bad file descriptor
2010-09-17 05:00:00 Setup complete

<user-74c177c1220d@xymon.invalid> writes:

During the last month, we have had some problems with Xymon when
using sche=
duled disabled (added from the web interface).

The first problem we had was on September 17th, when hobbitd crashed
while/=
after running the scheduled disable. We got the following error in hobbit.l=
og
2010-09-17 05:00:00 Fatal error in select: Bad file descriptor
2010-09-17 05:00:00 Setup complete

There is a bug lurking in the scheduled-task code, but I haven't been
able to quite nail down where it is. I've seen the same problem that
you have a couple of times, where a scheduled "disable" results in
xymond (hobbitd) crashing immediately afterwards.

One potential bug I did catch is fixed with the following patch:

Index: xymond/xymond.c
==========================================================
=========
--- xymond/xymond.c	(revision 6604)
+++ xymond/xymond.c	(working copy)
@@ -3971,7 +3971,7 @@
 	if (msg->doingwhat == RESPONDING) {
 		shutdown(msg->sock, SHUT_RD);
 	}
-	else {
+	else if (msg->sock >= 0) {
 		shutdown(msg->sock, SHUT_RDWR);
 		close(msg->sock);
 		msg->sock = -1;
@@ -5040,6 +5040,8 @@
 					swalk
= swalk->next;


	memset(&task, 0, sizeof(task));
• task.sock = -1;
• task.doingwhat = NOTALK;

	inet_aton(runtask->sender, (struct in_addr *)
&task.addr.sin_addr.s_addr);

	task.buf = task.bufp = runtask->command;

	task.buflen = strlen(runtask->command); task.bufsz =
task.buflen+1;


So it would be interesting to see if this helps in your setup. This patch
is against the current beta-3 code, but it applies to version 4.2.3 as
well if you run patch and explicitly tell it which file to patch:

   patch hobbit-4.2.3/hobbitd/hobbitd.c < task.patch


I am not sure if this fixes the problem, though. Because if this is
what causes the crash, then it ought to happen before the log message
that the task ran is written. Unless the bug doesn't crash the system
right away, but only triggers some memory corruption that results in
a later crash ...


Regards,
Henrik

/Johan

-----Original Message-----
From: Henrik Størner [mailto:user-ce4a2c883f75@xymon.invalid]
Sent: den 6 december 2010 12:41
To: xymon at xymon.com
Subject: Re: [xymon] Scheduled disable causes crash?

In
<user-06299813fc61@xymon.invalid
ement.se> =?iso-8859-1?Q?Johan_Sj=F6berg?=
<user-74c177c1220d@xymon.invalid> writes:

During the last month, we have had some problems with Xymon when
using sche=
duled disabled (added from the web interface).

The first problem we had was on September 17th, when hobbitd crashed
while/=
after running the scheduled disable. We got the following error in hobbit.l=
og
2010-09-17 05:00:00 Fatal error in select: Bad file descriptor
2010-09-17 05:00:00 Setup complete

There is a bug lurking in the scheduled-task code, but I haven't been
able to quite nail down where it is. I've seen the same problem that
you have a couple of times, where a scheduled "disable" results in
xymond (hobbitd) crashing immediately afterwards.

One potential bug I did catch is fixed with the following patch:

Index: xymond/xymond.c
==========================================================
=========
--- xymond/xymond.c	(revision 6604)
+++ xymond/xymond.c	(working copy)
@@ -3971,7 +3971,7 @@
 	if (msg->doingwhat == RESPONDING) {
 		shutdown(msg->sock, SHUT_RD);
 	}
-	else {
+	else if (msg->sock >= 0) {
 		shutdown(msg->sock, SHUT_RDWR);
 		close(msg->sock);
 		msg->sock = -1;
@@ -5040,6 +5040,8 @@
 					swalk
= swalk->next;


	memset(&task, 0, sizeof(task));
• task.sock = -1;
• task.doingwhat = NOTALK;

	inet_aton(runtask->sender, (struct in_addr *)
&task.addr.sin_addr.s_addr);

	task.buf = task.bufp = runtask->command;

	task.buflen = strlen(runtask->command); task.bufsz =
task.buflen+1;


So it would be interesting to see if this helps in your setup. This patch
is against the current beta-3 code, but it applies to version 4.2.3 as
well if you run patch and explicitly tell it which file to patch:

   patch hobbit-4.2.3/hobbitd/hobbitd.c < task.patch


I am not sure if this fixes the problem, though. Because if this is
what causes the crash, then it ought to happen before the log message
that the task ran is written. Unless the bug doesn't crash the system
right away, but only triggers some memory corruption that results in
a later crash ...


Regards,
Henrik

In <user-1b0573d8080c@xymon.invalid> =?iso-8859-1?Q?Johan_Sj=F6berg?= <user-74c177c1220d@xymon.invalid> writes:

I have not seen this problem since applying the patch. I can't be sure that
it's fixed since it didn't happen every time, but it is looking good.