If it helps, here's the same core, with pstack.
# pstack ./core_smcconsole_xymond_rrd_61_61_1426033626_20899
core './core_smcconsole_xymond_rrd_61_61_1426033626_20899' of 20899:
xymond_rrd --debug --rrddir=/opt/local/xymon/data/rrd --no-cache
feebebc4 _lwp_kill (6, 0, 0, 3a9a0, ffffffff, 6) + 8
fee329b0 abort (0, 1, feebafec, ffb3c, fef35518, 0) + 110
0003a9a0 sigsegv_handler (b, 0, ffbfa160, 0, fe8a2a00, ffbfa160) + 30
feebafec __sighndlr (b, 0, ffbfa160, 3a970, 0, 1) + c
feeaf69c call_user_handler (b, 0, 0, 0, fe8a2a00, ffbfa160) + 3b8
feeaf884 sigacthandler (b, 0, ffbfa160, ffbfb290, 0, fee91924) + 60
--- called from signal handler with signal 11 (SIGSEGV) ---
fee22d10 strlen (514cf, ffbfb3ec, ffbfa9d1, 0, 0, 0) + 50
fee91924 vfprintf (6c3d0, 514c0, ffbfb3e8, 0, a0ba4, 33e1c) + ec
0002fb4c dbgprintf (514c0, 0, 51400, 6c3f0, bf, 2ab388) + a4
00033e1c dump_tcp_services (a0, 1c00, fef37940, 0, 51400, 51400) + 74
000347f0 init_tcp_services (6a400, 51400, 51400, 6c3f0, bf, 23) + 91c
00030188 rrd_setup (93314, ffbfcfc4, 63c00, ffbfcf50, 6a400, 6a400) + 15c
00030518 find_xymon_rrd (93314, 511e8, 54ff8bda, 0, 932f3, 2e) + 4
00016614 main (93314, ffbfcfc4, 63c00, ffbfcf50, 54ff8bda, 93374) + 948
00015a40 _start (0, 0, 0, 0, 0, 0) + 5c
On 11 March 2015 at 08:37, Vernon Everett <user-b3f8dacb72c8@xymon.invalid> wrote:
Hi all
And even with --no-cache, I am still getting these corrupted rrd files.
I tried again with --debug (and --no-cache) and it core dumps.
Here's the backtrace.
# adb /zones/smcconsole/root/opt/local/xymon/server/bin/xymond_rrd
./core_smcconsole_xymond_rrd_61_61_1426033626_20899
core file = ./core_smcconsole_xymond_rrd_61_61_1426033626_20899 -- program
``
/zones/smcconsole/root/opt/local/xymon/server/bin/xymond_rrd'' on platform
SUNW,SPARC-Enterprise-T2000
SIGABRT: Abort
$c
libc.so.1`_lwp_kill+8(6, 0, 0, 3a9a0, ffffffff, 6)
libc.so.1`abort+0x110(0, 1, feebafec, ffb3c, fef35518, 0)
sigsegv_handler+0x30(b, 0, ffbfa160, 0, fe8a2a00, ffbfa160)
libc.so.1`__sighndlr+0xc(b, 0, ffbfa160, 3a970, 0, 1)
libc.so.1`call_user_handler+0x3b8(b, 0, 0, 0, fe8a2a00, ffbfa160)
libc.so.1`sigacthandler+0x60(b, 0, ffbfa160, ffbfb290, 0, fee91924)
libc.so.1`strlen+0x50(514cf, ffbfb3ec, ffbfa9d1, 0, 0, 0)
libc.so.1`vfprintf+0xec(6c3d0, 514c0, ffbfb3e8, 0, a0ba4, 33e1c)
dbgprintf+0xa4(514c0, 0, 51400, 6c3f0, bf, 2ab388)
dump_tcp_services+0x74(a0, 1c00, fef37940, 0, 51400, 51400)
init_tcp_services+0x91c(6a400, 51400, 51400, 6c3f0, bf, 23)
rrd_setup+0x15c(93314, ffbfcfc4, 63c00, ffbfcf50, 6a400, 6a400)
find_xymon_rrd+4(93314, 511e8, 54ff8bda, 0, 932f3, 2e)
main+0x948(93314, ffbfcfc4, 63c00, ffbfcf50, 54ff8bda, 93374)
_start+0x5c(0, 0, 0, 0, 0, 0)
Anything else I can offer that will assist?
Regards
Vernon
On 5 March 2015 at 09:11, J.C. Cleaver <user-87556346d4af@xymon.invalid> wrote:
On Wed, March 4, 2015 2:52 pm, Jeremy Laidman wrote:
On 04/03/2015 6:02 PM, "Vernon Everett" <user-b3f8dacb72c8@xymon.invalid>
wrote:
Looks like we might need to check with JC for more on that GOCLIENT
thing.
I just find it odd that it happened about the same time as the
corruption.
I haven't seen it again today, and haven't seen any other corruption
either.
If there's a correlation it might help us work out where the fault is.
But
it might be only a symptom.
As for the --debug option, it caused xymond_rrd to crash and burn,
dumping cores as we go.
Could be that thensame bug causing the crash during debug is also
causing
the corrupt filename. Have you analyzed the core dumps?
GOCLIENT is indeed the means by which xymond_channel listeners communicate
with xymond for the picking up of messages over SysV IPC. I believe the
messages there are just a side effect of it re-launching the channel
listener pipe to xymond_rrd.
The cache routines in xymond_rrd should be stable at this point. Can you
send a backtrace in from one of the cores? I'm curious where things could
be acting up here.
Regards,
-jc
--
"Accept the challenges so that you can feel the exhilaration of victory"
- General George Patton
--
"Accept the challenges so that you can feel the exhilaration of victory"
- General George Patton