Sudden rash of "logfetch" crashes in libc on RHEL 6 Linux?!? 🔗 link

Starting about 45 minutes ago as I type this, all of a sudden, 5 of
my machines running Xymon 4.3.17 have gotten afflicted with a mysterious
'disease' - "logfetch" is endlessly crashing on all of them.

The core files being created are getting corrupted for some reason,
and they are magically getting deleted as a result, so I can't debug
the problem.

One machine is running 64-bit RHEL 6.5; the rest 64-bit RHEL 6.3.  Client
version is as I mentioned 4.3.17:

xymon-client-4.3.17-1.el6.x86_64

This cycle is getting repeated every 1 minute and 40 seconds - help!

Oct 20 15:42:42 linux1 kernel: logfetch[2334]: segfault at ef ip 000000355a4c835f sp 00007fff0ac56a80 error 6 in libc-2.12.so[355a400000+189000]
Oct 20 15:42:42 linux1 abrt[2335]: //core.2334 fd(-1) is not a regular file with link count 1: Permission denied
Oct 20 15:42:43 linux1 abrt[2335]: Saved core dump of pid 2334 (/usr/bin/logfetch) to /var/spool/abrt/ccpp-2014-10-20-15:42:42-2334 (335872 bytes)
Oct 20 15:42:43 linux1 abrtd: Directory 'ccpp-2014-10-20-15:42:42-2334' creation detected
Oct 20 15:42:43 linux1 abrtd: Package 'xymon-client' isn't signed with proper key
Oct 20 15:42:43 linux1 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2014-10-20-15:42:42-2334' exited with 1
Oct 20 15:42:43 linux1 abrtd: Corrupted or bad directory /var/spool/abrt/ccpp-2014-10-20-15:42:42-2334, deleting

I doubt that "libc" got simultaneously corrupted on all 5 systems so
there must be something wrong in "logfetch".  The segfault ip is always
"000000355a4c835f" on the 4 RHEL 6.3 systems while the stack pointer varies.

	- Greg

P.S. Why am I getting these "isn't signed with proper key" messages as well?

On 21 October 2014 10:01, Greg Earle <user-8f45ae7a27f3@xymon.invalid> wrote:

I wonder if it's the same logfetch bug that's been lingering fat least 3
years:

http://lists.xymon.com/archive/2013-August/038009.html

You might try out the work-around mentioned there to see if it helps.

P.S. Why am I getting these "isn't signed with proper key" messages as well?

Maybe this can help:

http://lists.leap-cf.org/pipermail/leaplist/2012-December/000836.html

Cheers
Jeremy

On Oct 21, 2014, at 13:58 PM +1100, Jeremy Laidman <user-71895fb2e44c@xymon.invalid> wrote:

I wonder if it's the same logfetch bug that's been lingering at least 3
years:

http://lists.xymon.com/archive/2013-August/038009.html

I believe you're probably right, I had found postings from as far back
as December 2011 to this list describing the same sort of crashing.

You might try out the work-around mentioned there to see if it helps.

My work-around was to back-out the two seemingly-harmless "ignore" lines
I had just added to my "client-config.cfg" file!  I would try the memmove()
twiddle (if I can't solve it some other way) but then again I'm using the
Terabithia pre-cooked RPMs, so ...

Ah, interesting, thanks for that.  Not sure I'm too excited about going
'round adding 2 lines to 100+ "/etc/abrt/abrt.conf" files tho  ;)

	- Greg

Yeah, that was a nasty bug :/  4.3.18 is around the corner, so I haven't
backported that to the previous 4.3.17 package, but could.

ignore/trigger should fix the problem for now. Would you be up for testing
the most recent 4.3.18 package in
http://terabithia.org/rpms/xymon/testing/el6/ on a box and see if that
runs fine? (Just to validate there's not something else going on here
causing the crash...)


Regards,

-jc

On Tue, October 21, 2014 4:12 am, Greg Earle wrote: