Too much data for svcstatus.sh?
list Mike Burger
Good morning. Running Xymon 4.3.30, compiled from source tarball, on a RHEL 6.10 server. This morning, I encountered an interesting issue. I have a monitored system that has 197 discrete filesystems/mounts. When I attempt to enter this server's "Trends" section, I receive an internal server error (500). My httpd error log shows this when I attempt to access the trends link for the server in question: [Thu May 21 11:29:41 2020] [error] [client 10.40.24.55] Premature end of script headers: svcstatus.sh, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] *** glibc detected *** /apps/xymon/server/bin/svcstatus.cgi: realloc(): invalid next size: 0x0000000000de7dd0 ***, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x408a4f], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x403453], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x404bb7], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x4022c9], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 00400000-0043c000 r-xp 00000000 fd:08 24906 /apps/xymon/server/bin/svcstatus.cgi, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 0063c000-0063e000 rw-p 0003c000 fd:08 24906 /apps/xymon/server/bin/svcstatus.cgi, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html This does not happen for any other monitored system in my environment. This system is one of the more critical servers in my environment, so I'd like to figure out what's going on and remediate it if possible. -- Mike Burger http://www.bubbanfriends.org "It's always suicide-mission this, save-the-planet that. No one ever just stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1
list Mark O. Stitson
I have seen similar behaviour last week on one of our Solaris servers with
many mounts where trends fails.
Interestingly we run 2 Xymon servers for redundancy and it works flawlessly
on the secondary with exactly the same data.
When I run scvstatus.sh manually with strace a working machine all the RRDs
and then continues, the broken monitored machine throws a segfault:
stat("./zfs.......rrd", {st_mode=S_IFREG|0644, st_size=152528, ...}) = 0
getdents(3, /* 0 entries */, 32768) = 0
close(3) = 0
brk(0x282d000) = 0x282d000
brk(0x2828000) = 0x2828000
brk(0x2849000) = 0x2849000
open("/usr/lib/xymon/server/web/trends_header", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1276, ...}) = 0
read(3, "<!DOCTYPE HTML PUBLIC \"-//W3C//D"..., 1276) = 1276
close(3)
...
Whereas the broken one segfaults:
stat("./zfs.........rrd", {st_mode=S_IFREG|0644, st_size=76592, ...}) = 0
getdents(3, /* 0 entries */, 32768) = 0
close(3) = 0
brk(0x266b000) = 0x266b000
brk(0x2666000) = 0x2666000
mmap(NULL, 159744, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f82e2f29000
brk(0x2661000) = 0x2661000
brk(0x265f000) = 0x265f000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x26631f8} ---
+++ killed by SIGSEGV +++
It is on my list to investigate, but I haven't had time yet.
Mark
▸
-----Original Message----- From: Xymon <xymon-bounces at xymon.com> On Behalf Of Mike Burger Sent: 21 May 2020 16:47 To: xymon at xymon.com Subject: [Xymon] Too much data for svcstatus.sh? Good morning. Running Xymon 4.3.30, compiled from source tarball, on a RHEL 6.10 server. This morning, I encountered an interesting issue. I have a monitored system that has 197 discrete filesystems/mounts. When I attempt to enter this server's "Trends" section, I receive an internal server error (500). My httpd error log shows this when I attempt to access the trends link for the server in question: [Thu May 21 11:29:41 2020] [error] [client 10.40.24.55] Premature end of script headers: svcstatus.sh, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] *** glibc detected *** /apps/xymon/server/bin/svcstatus.cgi: realloc(): invalid next size: 0x0000000000de7dd0 ***, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x408a4f], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x403453], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x404bb7], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x4022c9], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 00400000-0043c000 r-xp 00000000 fd:08 24906 /apps/xymon/server/bin/svcstatus.cgi, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 0063c000-0063e000 rw-p 0003c000 fd:08 24906 /apps/xymon/server/bin/svcstatus.cgi, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html This does not happen for any other monitored system in my environment. This system is one of the more critical servers in my environment, so I'd like to figure out what's going on and remediate it if possible. -- Mike Burger http://www.bubbanfriends.org "It's always suicide-mission this, save-the-planet that. No one ever just stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1
list Mike Burger
▸
On 2020-05-22 07:16, Mark O. Stitson wrote:
I have seen similar behaviour last week on one of our Solaris servers
with
many mounts where trends fails.
Interestingly we run 2 Xymon servers for redundancy and it works
flawlessly
on the secondary with exactly the same data.
When I run scvstatus.sh manually with strace a working machine all the
RRDs
and then continues, the broken monitored machine throws a segfault:
stat("./zfs.......rrd", {st_mode=S_IFREG|0644, st_size=152528, ...}) =
0
getdents(3, /* 0 entries */, 32768) = 0
close(3) = 0
brk(0x282d000) = 0x282d000
brk(0x2828000) = 0x2828000
brk(0x2849000) = 0x2849000
open("/usr/lib/xymon/server/web/trends_header", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1276, ...}) = 0
read(3, "<!DOCTYPE HTML PUBLIC \"-//W3C//D"..., 1276) = 1276
close(3)
...
Whereas the broken one segfaults:
stat("./zfs.........rrd", {st_mode=S_IFREG|0644, st_size=76592, ...}) =
0
getdents(3, /* 0 entries */, 32768) = 0
close(3) = 0
brk(0x266b000) = 0x266b000
brk(0x2666000) = 0x2666000
mmap(NULL, 159744, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) =
0x7f82e2f29000
brk(0x2661000) = 0x2661000
brk(0x265f000) = 0x265f000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x26631f8}
---
+++ killed by SIGSEGV +++
It is on my list to investigate, but I haven't had time yet.
Mark
-----Original Message-----
From: Xymon <xymon-bounces at xymon.com> On Behalf Of Mike Burger
Sent: 21 May 2020 16:47
To: xymon at xymon.com
Subject: [Xymon] Too much data for svcstatus.sh?
Good morning.
Running Xymon 4.3.30, compiled from source tarball, on a RHEL 6.10
server.
This morning, I encountered an interesting issue.
I have a monitored system that has 197 discrete filesystems/mounts.
When I attempt to enter this server's "Trends" section, I receive an
internal server error (500).
My httpd error log shows this when I attempt to access the trends link
for
the server in question:
[Thu May 21 11:29:41 2020] [error] [client 10.40.24.55] Premature end
of
script headers: svcstatus.sh, referer:
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] *** glibc
detected
*** /apps/xymon/server/bin/svcstatus.cgi: realloc(): invalid next size:
0x0000000000de7dd0 ***, referer:
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
/apps/xymon/server/bin/svcstatus.cgi[0x408a4f], referer:
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
/apps/xymon/server/bin/svcstatus.cgi[0x403453], referer:
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
/apps/xymon/server/bin/svcstatus.cgi[0x404bb7], referer:
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
/apps/xymon/server/bin/svcstatus.cgi[0x4022c9], referer:
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
00400000-0043c000 r-xp 00000000 fd:08 24906
/apps/xymon/server/bin/svcstatus.cgi, referer:
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
0063c000-0063e000 rw-p 0003c000 fd:08 24906
/apps/xymon/server/bin/svcstatus.cgi, referer:
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
This does not happen for any other monitored system in my environment.
This system is one of the more critical servers in my environment, so
I'd
like to figure out what's going on and remediate it if possible.Henrik, et al: Any thoughts?
▸
-- Mike Burger http://www.bubbanfriends.org "It's always suicide-mission this, save-the-planet that. No one ever just stops by to say 'hi' anymore." --Colonel Jack O'Neill, SG1
list Marco Avvisano
Hi, i have the same problem on Red Hat Enterprise Linux Server release 6.3, running xymon 4.3.30 compiled from source Best Regards Marco
▸
Il 28/05/2020 16:58, Mike Burger ha scritto:On 2020-05-22 07:16, Mark O. Stitson wrote:I have seen similar behaviour last week on one of our Solaris servers with many mounts where trends fails. Interestingly we run 2 Xymon servers for redundancy and it works flawlessly on the secondary with exactly the same data. When I run scvstatus.sh manually with strace a working machine all the RRDs and then continues, the broken monitored machine throws a segfault: stat("./zfs.......rrd", {st_mode=S_IFREG|0644, st_size=152528, ...}) = 0 getdents(3, /* 0 entries */, 32768)???? = 0 close(3)??????????????????????????????? = 0 brk(0x282d000)????????????????????????? = 0x282d000 brk(0x2828000)????????????????????????? = 0x2828000 brk(0x2849000)????????????????????????? = 0x2849000 open("/usr/lib/xymon/server/web/trends_header", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=1276, ...}) = 0 read(3, "<!DOCTYPE HTML PUBLIC \"-//W3C//D"..., 1276) = 1276 close(3) ... Whereas the broken one segfaults: stat("./zfs.........rrd", {st_mode=S_IFREG|0644, st_size=76592, ...}) = 0 getdents(3, /* 0 entries */, 32768)???? = 0 close(3)??????????????????????????????? = 0 brk(0x266b000)????????????????????????? = 0x266b000 brk(0x2666000)????????????????????????? = 0x2666000 mmap(NULL, 159744, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f82e2f29000 brk(0x2661000)????????????????????????? = 0x2661000 brk(0x265f000)????????????????????????? = 0x265f000 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x26631f8} --- +++ killed by SIGSEGV +++ It is on my list to investigate, but I haven't had time yet. Mark -----Original Message----- From: Xymon <xymon-bounces at xymon.com> On Behalf Of Mike Burger Sent: 21 May 2020 16:47 To: xymon at xymon.com Subject: [Xymon] Too much data for svcstatus.sh? Good morning. Running Xymon 4.3.30, compiled from source tarball, on a RHEL 6.10 server. This morning, I encountered an interesting issue. I have a monitored system that has 197 discrete filesystems/mounts. When I attempt to enter this server's "Trends" section, I receive an internal server error (500). My httpd error log shows this when I attempt to access the trends link for the server in question: [Thu May 21 11:29:41 2020] [error] [client 10.40.24.55] Premature end of script headers: svcstatus.sh, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] *** glibc detected *** /apps/xymon/server/bin/svcstatus.cgi: realloc(): invalid next size: 0x0000000000de7dd0 ***, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x408a4f], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x403453], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x404bb7], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x4022c9], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 00400000-0043c000 r-xp 00000000 fd:08 24906 /apps/xymon/server/bin/svcstatus.cgi, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 0063c000-0063e000 rw-p 0003c000 fd:08 24906 /apps/xymon/server/bin/svcstatus.cgi, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html This does not happen for any other monitored system in my environment. This system is one of the more critical servers in my environment, so I'd like to figure out what's going on and remediate it if possible.Henrik, et al: Any thoughts?
list Henrik Størner
Hi, the best way to debug this is to get a proper stack trace of when the program crashes. For this, you first need to make sure you have a binary of svcstatus.cgi which includes debug information - if you look at the Xymon sources, the build/Makefile.Linux file has a commented-out line setting the CFLAGS variable for debugging: # Compile flags for debugging # CFLAGS = -g -DDEBUG -Wall -D_REENTRANT $(LFSDEF) $(OSDEF) Uncomment the last line, run "make clean; make" and then use the web/svcstatus.cgi binary from that. To run the program in the gdb debugger, I would bypass the web server CGI and run it by hand like this: $ export SCRIPT_NAME="svcstatus.sh" $ export REQUEST_METHOD="GET" $ export QUERY_STRING="HOST=claudio.hswn.dk&SERVICE=disk" $ gdb ./web/svcstatus.cgi (gdb) run The value for the QUERY_STRING should be taken from the URL for your status-page - the part after the '?' sign. The example here is from the xymon.com site, but you really just need to modify the hostname to that of your own. Note that you must run it on the Xymon server, because it reads the file stored there. Regards, Henrik Den 21.05.2020 kl. 17.47 skrev Mike Burger:
▸
Good morning. Running Xymon 4.3.30, compiled from source tarball, on a RHEL 6.10 server. This morning, I encountered an interesting issue. I have a monitored system that has 197 discrete filesystems/mounts. When I attempt to enter this server's "Trends" section, I receive an internal server error (500). My httpd error log shows this when I attempt to access the trends link for the server in question: [Thu May 21 11:29:41 2020] [error] [client 10.40.24.55] Premature end of script headers: svcstatus.sh, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] *** glibc detected *** /apps/xymon/server/bin/svcstatus.cgi: realloc(): invalid next size: 0x0000000000de7dd0 ***, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x408a4f], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x403453], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x404bb7], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] /apps/xymon/server/bin/svcstatus.cgi[0x4022c9], referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 00400000-0043c000 r-xp 00000000 fd:08 24906????????????????????????????? /apps/xymon/server/bin/svcstatus.cgi, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html [Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] 0063c000-0063e000 rw-p 0003c000 fd:08 24906????????????????????????????? /apps/xymon/server/bin/svcstatus.cgi, referer: http://sysmon.fhmc.local/xymon/edmprod/edmprod.html This does not happen for any other monitored system in my environment. This system is one of the more critical servers in my environment, so I'd like to figure out what's going on and remediate it if possible.