xymond_channel consuming a lot of CPU?
list David Diep
Hi, I saw a little bit about this back in 8/31/12 (http://lists.xymon.com/archive/2012-August/035381.html), but it was never resolved? Has anyone else seen xymond_channel consuming so much CPU time? I can't imagine this being normal. top - 14:07:56 up 6 days, 2:52, 1 user, load average: 0.97, 0.63, 0.58 Tasks: 117 total, 2 running, 115 sleeping, 0 stopped, 0 zombie Cpu(s): 22.7%us, 26.7%sy, 0.0%ni, 49.5%id, 0.3%wa, 0.2%hi, 0.0%si, 0.7%st Mem: 245792k total, 235308k used, 10484k free, 9248k buffers Swap: 1304648k total, 96604k used, 1208044k free, 44164k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24914 xymon 20 0 7080 1388 1148 R 98.7 0.6 22:59.70 xymond_channel 1 root 20 0 3232 504 316 S 0.0 0.2 0:01.75 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kthreadd 3 root RT 0 0 0 0 S 0.0 0.0 0:00.22 migration/0 4 root 20 0 0 0 0 S 0.0 0.0 0:02.03 ksoftirqd/0 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 6 root RT 0 0 0 0 S 0.0 0.0 0:00.22 migration/1 7 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1 8 root 20 0 0 0 0 S 0.0 0.0 0:01.88 ksoftirqd/1 9 root 20 0 0 0 0 S 0.0 0.0 0:48.12 events/0 10 root 20 0 0 0 0 S 0.0 0.0 0:58.04 events/1 11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuset 12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khelper 13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 netns 14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 async/mgr 15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pm 16 root 20 0 0 0 0 S 0.0 0.0 0:01.54 sync_supers 17 root 20 0 0 0 0 S 0.0 0.0 0:02.81 bdi-default 18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/0 19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/1 20 root 20 0 0 0 0 S 0.0 0.0 0:00.32 kblockd/0 21 root 20 0 0 0 0 S 0.0 0.0 0:00.26 kblockd/1 22 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/0 23 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/1 24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/0 25 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/1 26 root 20 0 0 0 0 S 0.0 0.0 0:00.10 cio 27 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cio_chp 28 root 20 0 0 0 0 S 0.0 0.0 0:01.56 appldata 29 root 20 0 0 0 0 S 0.0 0.0 0:00.38 khungtaskd 30 root 20 0 0 0 0 S 0.0 0.0 0:04.48 kswapd0 31 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd David Diep RHEL 6.2 for IBM zSeries Kernel: 2.6.32-358.23.2.el6.s390x Look out for those in need this winter. When the temperature or wind chill is 32?F or below, the District issues a Hypothermia Alert. For assistance during an Alert, call the Shelter Hotline<http://dhs.dc.gov/page/hypothermia-alert-plan>; at X-XXX-XXX-XXXX or 311.
list Japheth Cleaver
▸
On Mon, December 23, 2013 11:26 am, Diep, David (OCTO-Contractor) wrote:
Hi, I saw a little bit about this back in 8/31/12 (http://lists.xymon.com/archive/2012-August/035381.html), but it was never resolved? Has anyone else seen xymond_channel consuming so much CPU time? I can't imagine this being normal. top - 14:07:56 up 6 days, 2:52, 1 user, load average: 0.97, 0.63, 0.58 Tasks: 117 total, 2 running, 115 sleeping, 0 stopped, 0 zombie Cpu(s): 22.7%us, 26.7%sy, 0.0%ni, 49.5%id, 0.3%wa, 0.2%hi, 0.0%si, 0.7%st Mem: 245792k total, 235308k used, 10484k free, 9248k buffers Swap: 1304648k total, 96604k used, 1208044k free, 44164k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24914 xymon 20 0 7080 1388 1148 R 98.7 0.6 22:59.70 xymond_channel
Hi David,
That much CPU time definitely isn't normal... Were you able to get an
strace or see anything in the logs from around when that was happening.
The only thing that immediately comes to mind might be some sort of
semaphore/IPC issue causing it to spin on incoming channel messages from
xymond.
Regards,
-jc
list David Diep
Hi JC,
I don't see a lot of messages in /var/log/xymon, but I do see some messages in the messages log:
Dec 30 10:00:39 vipservd kernel: type=1400 audit(1388415639.785:228): avc: denied { read } for pid=22630 comm="httpd" name="xymon.html" dev=dm-0 ino=156411 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file
30 10:00:39 vipservd kernel: __ratelimit: 39 callbacks suppressed
Dec 30 10:00:39 vipservd kernel: type=1400 audit(1388415639.768:226): avc: denied { search } for pid=22630 comm="httpd" name="xymon" dev=dm-0 ino=162241 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
Dec 30 10:00:39 vipservd kernel: type=1400 audit(1388415639.785:227): avc: denied { search } for pid=22630 comm="httpd" name="server" dev=dm-0 ino=162242 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir
Dec 30 10:00:39 vipservd kernel: type=1400 audit(1388415639.785:228): avc: denied { read } for pid=22630 comm="httpd" name="xymon.html" dev=dm-0 ino=156411 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file
Dec 30 10:00:39 vipservd kernel: type=1400 audit(1388415639.785:229): avc: denied { open } for pid=22630 comm="httpd" name="xymon.html" dev=dm-0 ino=156411 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file
Would I just execute the strace like so:
strace ./xymon.sh
David Diep
▸
Look out for those in need this winter. When the temperature or wind chill is 32?F or below, the District issues a Hypothermia Alert. For assistance during an Alert, call the Shelter Hotline<http://dhs.dc.gov/page/hypothermia-alert-plan>; at X-XXX-XXX-XXXX or 311. -----Original Message-----
▸
From: J.C. Cleaver [mailto:user-87556346d4af@xymon.invalid]
Sent: Thursday, December 26, 2013 5:11 PM
To: Diep, David (OCTO-Contractor)
Cc: xymon at xymon.com
Subject: Re: [Xymon] xymond_channel consuming a lot of CPU?
On Mon, December 23, 2013 11:26 am, Diep, David (OCTO-Contractor) wrote:
Hi,
I saw a little bit about this back in 8/31/12
(http://lists.xymon.com/archive/2012-August/035381.html), but it was
never resolved? Has anyone else seen xymond_channel consuming so much CPU time?
I can't imagine this being normal.
top - 14:07:56 up 6 days, 2:52, 1 user, load average: 0.97, 0.63, 0.58
Tasks: 117 total, 2 running, 115 sleeping, 0 stopped, 0 zombie
Cpu(s): 22.7%us, 26.7%sy, 0.0%ni, 49.5%id, 0.3%wa, 0.2%hi, 0.0%si,
0.7%st
Mem: 245792k total, 235308k used, 10484k free, 9248k buffers
Swap: 1304648k total, 96604k used, 1208044k free, 44164k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24914 xymon 20 0 7080 1388 1148 R 98.7 0.6 22:59.70
xymond_channel
Hi David, That much CPU time definitely isn't normal... Were you able to get an strace or see anything in the logs from around when that was happening. The only thing that immediately comes to mind might be some sort of semaphore/IPC issue causing it to spin on incoming channel messages from xymond. Regards, -jc
list Jeremy Laidman
On 31 December 2013 02:10, Diep, David (OCTO-Contractor) <user-ca44d68fd6e2@xymon.invalid>wrote:
Would I just execute the strace like so: strace ./xymon.sh
No, find the PID of the process, and specify with "-p", like so: strace -p 24914 J
list Henrik Størner
Den 23.12.2013 20:26, Diep, David (OCTO-Contractor) skrev:
I saw a little bit about this back in 8/31/12
(http://lists.xymon.com/archive/2012-August/035381.html [1]), but it was
▸
never resolved? Has anyone else seen xymond_channel consuming so much
CPU time? I can't imagine this being normal.
It isn't. I know of a couple of weird cases where this has happened, but not enough to pin-point what is causing it. An immediate workaround is to just kill that process - it will restart automatically and should then work as expected. Regards, Henrik Links: [1] http://lists.xymon.com/archive/2012-August/035381.html