I have seen similar behaviour last week on one of our Solaris servers
with
many mounts where trends fails.
Interestingly we run 2 Xymon servers for redundancy and it works
flawlessly
on the secondary with exactly the same data.
When I run scvstatus.sh manually with strace a working machine all the
RRDs
and then continues, the broken monitored machine throws a segfault:
stat("./zfs.......rrd", {st_mode=S_IFREG|0644, st_size=152528, ...}) =
0
getdents(3, /* 0 entries */, 32768) = 0
close(3) = 0
brk(0x282d000) = 0x282d000
brk(0x2828000) = 0x2828000
brk(0x2849000) = 0x2849000
open("/usr/lib/xymon/server/web/trends_header", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1276, ...}) = 0
read(3, "<!DOCTYPE HTML PUBLIC \"-//W3C//D"..., 1276) = 1276
close(3)
...
Whereas the broken one segfaults:
stat("./zfs.........rrd", {st_mode=S_IFREG|0644, st_size=76592, ...}) =
0
getdents(3, /* 0 entries */, 32768) = 0
close(3) = 0
brk(0x266b000) = 0x266b000
brk(0x2666000) = 0x2666000
mmap(NULL, 159744, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) =
0x7f82e2f29000
brk(0x2661000) = 0x2661000
brk(0x265f000) = 0x265f000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x26631f8}
---
+++ killed by SIGSEGV +++
It is on my list to investigate, but I haven't had time yet.
Mark
-----Original Message-----
From: Xymon <xymon-bounces at xymon.com> On Behalf Of Mike Burger
Sent: 21 May 2020 16:47
To: xymon at xymon.com
Subject: [Xymon] Too much data for svcstatus.sh?
Good morning.
Running Xymon 4.3.30, compiled from source tarball, on a RHEL 6.10
server.
This morning, I encountered an interesting issue.
I have a monitored system that has 197 discrete filesystems/mounts.
When I attempt to enter this server's "Trends" section, I receive an
internal server error (500).
My httpd error log shows this when I attempt to access the trends link
for
the server in question:
[Thu May 21 11:29:41 2020] [error] [client 10.40.24.55] Premature end
of
script headers: svcstatus.sh, referer:
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55] *** glibc
detected
*** /apps/xymon/server/bin/svcstatus.cgi: realloc(): invalid next size:
0x0000000000de7dd0 ***, referer:
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
/apps/xymon/server/bin/svcstatus.cgi[0x408a4f], referer:
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
/apps/xymon/server/bin/svcstatus.cgi[0x403453], referer:
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
/apps/xymon/server/bin/svcstatus.cgi[0x404bb7], referer:
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
/apps/xymon/server/bin/svcstatus.cgi[0x4022c9], referer:
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
00400000-0043c000 r-xp 00000000 fd:08 24906
/apps/xymon/server/bin/svcstatus.cgi, referer:
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
[Thu May 21 11:29:59 2020] [error] [client 10.40.24.55]
0063c000-0063e000 rw-p 0003c000 fd:08 24906
/apps/xymon/server/bin/svcstatus.cgi, referer:
http://sysmon.fhmc.local/xymon/edmprod/edmprod.html
This does not happen for any other monitored system in my environment.
This system is one of the more critical servers in my environment, so
I'd
like to figure out what's going on and remediate it if possible.