False positives on RAM and DISK F19
list JT Tullis
I can't seem to find a solution for a problem I'm having: I'm running xymon-client on several Fedora 19 servers. The client version is 4.3.28 release 1.fc19. There isn't an option to upgrade the servers right now from F19. Memory report showing over 100%: server:memory red [758975] red Wed Mar 22 08:56:44 MDT 2017 - Memory CRITICAL Memory Used Total Percentage &red Physical 1236M 123M 1004% &red Actual 9293M 123M 7555% &green Swap 2266M 2867M 79% Disk showing 0 units free even though that isn't the case: server:disk red [208517] red Wed Mar 22 07:42:39 MDT 2017 - Filesystems NOT ok &red 11933080 4623272 73% / (0 units free) has reached the PANIC level (512 units) &red 636 6149980 1% /dev/shm (0 units free) has reached the PANIC level (512 units) &red 1016 6149600 1% /run (0 units free) has reached the PANIC level (512 units) &red 0 6150616 0% /sys/fs/cgroup (0 units free) has reached the PANIC level (512 units) &red 264948 5885668 5% /tmp (0 units free) has reached the PANIC level (512 units) &red 117488 62417 66% /boot (0 units free) has reached the PANIC level (512 units) Filesystem 1024-blocks Used A ail]ble Capacity Mounted on /dev/mapper/fedora_pvapp05-root 17411420 11933080 4623272 73% / tmpfs 6150616 636 6149980 1% /dev/shm tmpfs 6150616 1016 6149600 1% /run tmpfs 6150616 0 6150616 0% /sys/fs/cgroup tmpfs 6150616 264948 5885668 5% /tmp /dev/xvda1 194241 117488 62417 66% /boot Does anyone know if there is a patch for this or has anyone successfully run a newer version of the client on an older OS? Thanks, John
list Bruce Ferrell
▸
On 3/22/17 8:21 AM, JT Tullis wrote:
I can't seem to find a solution for a problem I'm having: I'm running xymon-client on several Fedora 19 servers. The client version is 4.3.28 release 1.fc19. There isn't an option to upgrade the servers right now from F19. Memory report showing over 100%: server:memory red [758975] red Wed Mar 22 08:56:44 MDT 2017 - Memory CRITICAL Memory Used Total Percentage &red Physical 1236M 123M 1004% &red Actual 9293M 123M 7555% &green Swap 2266M 2867M 79% Disk showing 0 units free even though that isn't the case: server:disk red [208517] red Wed Mar 22 07:42:39 MDT 2017 - Filesystems NOT ok &red 11933080 4623272 73% / (0 units free) has reached the PANIC level (512 units) &red 636 6149980 1% /dev/shm (0 units free) has reached the PANIC level (512 units) &red 1016 6149600 1% /run (0 units free) has reached the PANIC level (512 units) &red 0 6150616 0% /sys/fs/cgroup (0 units free) has reached the PANIC level (512 units) &red 264948 5885668 5% /tmp (0 units free) has reached the PANIC level (512 units) &red 117488 62417 66% /boot (0 units free) has reached the PANIC level (512 units) Filesystem 1024-blocks Used A ail]ble Capacity Mounted on /dev/mapper/fedora_pvapp05-root 17411420 11933080 4623272 73% / tmpfs 6150616 636 6149980 1% /dev/shm tmpfs 6150616 1016 6149600 1% /run tmpfs 6150616 0 6150616 0% /sys/fs/cgroup tmpfs 6150616 264948 5885668 5% /tmp /dev/xvda1 194241 117488 62417 66% /boot Does anyone know if there is a patch for this or has anyone successfully run a newer version of the client on an older OS? Thanks, John
John, This *may* not by a xymon issue. I'm seeing similar things with my systems. They are reporting bogus info to my xymon and xymon reports ( and alarms ) on that. what do free and df report at the command line of the effected systems?
list Japheth Cleaver
▸
On 3/22/2017 9:04 AM, Bruce Ferrell wrote:
On 3/22/17 8:21 AM, JT Tullis wrote:I can't seem to find a solution for a problem I'm having: I'm running xymon-client on several Fedora 19 servers. The client version is 4.3.28 release 1.fc19. There isn't an option to upgrade the servers right now from F19. Memory report showing over 100%: server:memory red [758975] red Wed Mar 22 08:56:44 MDT 2017 - Memory CRITICAL Memory Used Total Percentage &red Physical 1236M 123M 1004% &red Actual 9293M 123M 7555% &green Swap 2266M 2867M 79% Disk showing 0 units free even though that isn't the case: server:disk red [208517] red Wed Mar 22 07:42:39 MDT 2017 - Filesystems NOT ok &red 11933080 4623272 73% / (0 units free) has reached the PANIC level (512 units) &red 636 6149980 1% /dev/shm (0 units free) has reached the PANIC level (512 units) &red 1016 6149600 1% /run (0 units free) has reached the PANIC level (512 units) &red 0 6150616 0% /sys/fs/cgroup (0 units free) has reached the PANIC level (512 units) &red 264948 5885668 5% /tmp (0 units free) has reached the PANIC level (512 units) &red 117488 62417 66% /boot (0 units free) has reached the PANIC level (512 units) Filesystem 1024-blocks Used A ail]ble Capacity Mounted on /dev/mapper/fedora_pvapp05-root 17411420 11933080 4623272 73% / tmpfs 6150616 636 6149980 1% /dev/shm tmpfs 6150616 1016 6149600 1% /run tmpfs 6150616 0 6150616 0% /sys/fs/cgroup tmpfs 6150616 264948 5885668 5% /tmp /dev/xvda1 194241 117488 62417 66% /boot Does anyone know if there is a patch for this or has anyone successfully run a newer version of the client on an older OS? Thanks, JohnJohn, This *may* not by a xymon issue. I'm seeing similar things with my systems. They are reporting bogus info to my xymon and xymon reports ( and alarms ) on that. what do free and df report at the command line of the effected systems?
I just span up a F19 VM for validation and things seem to be reporting okay for disk and memory (and other things)... with 4.3.28-1.fc19 The parsing in particular seems normal. Can you provide the [free] and [df] portions of the clientlog for this host? Also, to aid in debugging can you enable --debug (or send -USR2) on your xymond_client process? If there's something awry about these, we may get useful logs there. Also, is there an unusual locale setting on this host? In theory, we should be determining things like disk column headers mostly dynamically, but... Regards, -jc
list John Tullis
I can't seem to find a solution for a problem I'm having: I'm running xymon-client on several Fedora 19 servers. The client version is 4.3.28 release 1.fc19. There isn't an option to upgrade the servers right now from F19. Memory report showing over 100%: server:memory red [758975] red Wed Mar 22 08:56:44 MDT 2017 - Memory CRITICAL Memory Used Total Percentage &red Physical 1236M 123M 1004% &red Actual 9293M 123M 7555% &green Swap 2266M 2867M 79% Disk showing 0 units free even though that isn't the case: server:disk red [208517] red Wed Mar 22 07:42:39 MDT 2017 - Filesystems NOT ok &red 11933080 4623272 73% / (0 units free) has reached the PANIC level (512 units) &red 636 6149980 1% /dev/shm (0 units free) has reached the PANIC level (512 units) &red 1016 6149600 1% /run (0 units free) has reached the PANIC level (512 units) &red 0 6150616 0% /sys/fs/cgroup (0 units free) has reached the PANIC level (512 units) &red 264948 5885668 5% /tmp (0 units free) has reached the PANIC level (512 units) &red 117488 62417 66% /boot (0 units free) has reached the PANIC level (512 units) Filesystem 1024-blocks Used A ail]ble Capacity Mounted on /dev/mapper/fedora_pvapp05-root 17411420 11933080 4623272 73% / tmpfs 6150616 636 6149980 1% /dev/shm tmpfs 6150616 1016 6149600 1% /run tmpfs 6150616 0 6150616 0% /sys/fs/cgroup tmpfs 6150616 264948 5885668 5% /tmp /dev/xvda1 194241 117488 62417 66% /boot Does anyone know if there is a patch for this or has anyone successfully run a newer version of the client on an older OS? Thanks, John
list John Tullis
I realized that I didn’t give all the details. This only happens once or twice a week per server. It will go this way and then recover a few minutes later. Right now, everything is normal (green) but then it will happen again in a couple days. I’m getting the exact behavior on my 15 other Fedora 19 machines. John Tullis
▸
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Bruce Ferrell
Sent: Wednesday, March 22, 2017 10:05 AM
To: xymon at xymon.com
Subject: Re: [Xymon] False positives on RAM and DISK F19
On 3/22/17 8:21 AM, JT Tullis wrote:
I can't seem to find a solution for a problem I'm having:
I'm running xymon-client on several Fedora 19 servers.
The client version is 4.3.28 release 1.fc19.
There isn't an option to upgrade the servers right now from F19.
Memory report showing over 100%:
server:memory red [758975]
red Wed Mar 22 08:56:44 MDT 2017 - Memory CRITICAL
Memory Used Total Percentage
&red Physical 1236M 123M 1004%
&red Actual 9293M 123M 7555%
&green Swap 2266M 2867M 79%
Disk showing 0 units free even though that isn't the case:
server:disk red [208517]
red Wed Mar 22 07:42:39 MDT 2017 - Filesystems NOT ok
&red 11933080 4623272 73% / (0 units free) has reached the PANIC level (512 units)
&red 636 6149980 1% /dev/shm (0 units free) has reached the PANIC level (512 units)
&red 1016 6149600 1% /run (0 units free) has reached the PANIC level (512 units)
&red 0 6150616 0% /sys/fs/cgroup (0 units free) has reached the PANIC level (512 units)
&red 264948 5885668 5% /tmp (0 units free) has reached the PANIC level (512 units)
&red 117488 62417 66% /boot (0 units free) has reached the PANIC level (512 units)
Filesystem 1024-blocks Used A
ail]ble Capacity Mounted on
/dev/mapper/fedora_pvapp05-root 17411420 11933080 4623272 73% /
tmpfs 6150616 636 6149980 1% /dev/shm
tmpfs 6150616 1016 6149600 1% /run
tmpfs 6150616 0 6150616 0% /sys/fs/cgroup
tmpfs 6150616 264948 5885668 5% /tmp
/dev/xvda1 194241 117488 62417 66% /boot
Does anyone know if there is a patch for this or has anyone successfully run a newer version of the client on an older OS?
Thanks, John
John,
This *may* not by a xymon issue.
I'm seeing similar things with my systems. They are reporting bogus info to my xymon and xymon reports ( and alarms ) on that.
what do free and df report at the command line of the effected systems?
list Japheth Cleaver
My suspicion is that there's a corrupted clientlog message coming through, which is throwing the parser off. The snippet you show below has: Filesystem 1024-blocks Used A ail]ble Capacity Mounted on Assuming that's not an HTML/email client munging, then that would probably explain it. I haven't noticed this problem occurring on my local F19 VM however. A few questions: Are there any custom scripts or unusual tweaks in the environment on these boxes? Any processes with unusual names in them (control characters, etc) Do all of these F19 boxes show the same problem at the same time, or is it randomly for each of them. Can you provide a full copy of the clientlog from when it occurs and a normal one for comparison? (Private is fine.) Regards, -jc
▸
On 4/1/2017 10:39 PM, John Tullis wrote:I realized that I didn’t give all the details. This only happens once or twice a week per server. It will go this way and then recover a few minutes later. Right now, everything is normal (green) but then it will happen again in a couple days. I’m getting the exact behavior on my 15 other Fedora 19 machines. John Tullis
*From:*Xymon [mailto:xymon-bounces at xymon.com] *On Behalf Of *Bruce Ferrell
▸
*Sent:* Wednesday, March 22, 2017 10:05 AM
*To:* xymon at xymon.com
*Subject:* Re: [Xymon] False positives on RAM and DISK F19
On 3/22/17 8:21 AM, JT Tullis wrote:
I can't seem to find a solution for a problem I'm having:
I'm running xymon-client on several Fedora 19 servers.
The client version is 4.3.28 release 1.fc19.
There isn't an option to upgrade the servers right now from F19.
Memory report showing over 100%:
server:memory red [758975]
red Wed Mar 22 08:56:44 MDT 2017 - Memory CRITICAL
Memory Used Total Percentage
&red Physical 1236M 123M 1004%
&red Actual 9293M 123M 7555%
&green Swap 2266M 2867M 79%
Disk showing 0 units free even though that isn't the case:
server:disk red [208517]
red Wed Mar 22 07:42:39 MDT 2017 - Filesystems NOT ok
&red 11933080 4623272 73% / (0 units free) has reached the
PANIC level (512 units)
&red 636 6149980 1% /dev/shm (0 units free) has reached
the PANIC level (512 units)
&red 1016 6149600 1% /run (0 units free) has reached the
PANIC level (512 units)
&red 0 6150616 0% /sys/fs/cgroup (0 units free) has
reached the PANIC level (512 units)
&red 264948 5885668 5% /tmp (0 units free) has reached the
PANIC level (512 units)
&red 117488 62417 66% /boot (0 units free) has reached
the PANIC level (512 units)
Filesystem 1024-blocks Used A
ail]ble Capacity Mounted on
/dev/mapper/fedora_pvapp05-root 17411420 11933080 4623272 73% /
tmpfs 6150616 636 6149980 1% /dev/shm
tmpfs 6150616 1016 6149600 1% /run
tmpfs 6150616 0 6150616 0% /sys/fs/cgroup
tmpfs 6150616 264948 5885668 5% /tmp
/dev/xvda1 194241 117488 62417
66% /boot
Does anyone know if there is a patch for this or has anyone
successfully run a newer version of the client on an older OS?
Thanks, John
John,
This *may* not by a xymon issue.
I'm seeing similar things with my systems. They are reporting bogus info to my xymon and xymon reports ( and alarms ) on that.
what do free and df report at the command line of the effected systems?