PROC matching failure due to column bloat 🔗 link

Peeps

I have both Solaris and Linux servers where a large or long-running process
causes PROC matching to fail.  Here are some examples:


 7701     1 root       Feb 28 S  24  0.0 00:00:00  0.0   572   2692
/sbin/agetty -L 9600 ttyS0 vt102
 7702     1 root       Feb 28 S  23  0.0 00:00:00  0.0   576   2692
/sbin/agetty -L 9600 ttyS1 vt102
 7704     1 named      Feb 28 S  18  2.4 1-08:59:39  4.4 270500 412784
/usr/sbin/named -u named -f
26498  3293 root     12:47:46 S  14  0.0 00:00:00  0.0   468   2676 sleep 180

This is on Linux.  Note the longer-than-a-day TIME column that pushes the
columns after it to the right.

The following is on Solaris 9:

11201 11199  n101649 12:38:54 S  59  0.0        0:00  0.0 1000 1144 vmstat 300 2
11202     1  n101649 12:38:54 S  59  0.0        0:00  0.0  968 1104 sh
-c iostat -dxsrP 300 2 1>/tmp/xymon_iostatdisk.redacted
 3244  2965     root   Feb_16 S  59  0.0        5:20  0.1 7104 18736
/opt/OV/lbin/eaagt/opcle -std
 3245  2965     root   Feb_16 S  59  0.0        1:18  0.1 6376 20960
/opt/OV/lbin/eaagt/opcmona
 3253     1     root   Feb_16 S  59  0.9  1-10:46:45  0.8 58168 59632
/usr/local/sbin/named -f

Solaris "ps" output allows more characters for TIME than Linux.  However in
this case the memory columns (RSS and VSZ) are larger than expected,
pushing a couple of digits over into the process name area.

It seems that Xymon is parsing these based on fixed column sizes, defined
for each OS.  The result of these particular examples is that Xymon fails
to match on the process name.  Instead, I need to use match strings like so:

        PROC "%^(\d* |^)/usr/local/sbin/named(\s*$|\s)" 1 1
"TEXT=/usr/local/sbin/named"

or

        PROC "%^(\d* |^)/usr/sbin/named(\s*$|\s)" 1 1 "TEXT=/usr/sbin/named"

I guess this email is part "am I doing something wrong", part "does anyone
have a better idea", and part feature request (for more awk-like positional
matching).

Cheers
Jeremy

Jeremy,

I would guess it's not matching because what you show begins with a space and not a number or digit.  Could you try simplifying the line to:

PROC "%named" 1 1 "TEXT=/usr/local/sbin/named"

See if that works and then add complexity if there are other processes with the string 'named' in the ps listing.

You may also want to look at the 'Client data' link on the procs alert for this host and then the [ps] section to see how your ps listing is being presented to Xymon as it may not match what you see at the command line depending on what command line ps options you are using.

~David

From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Jeremy Laidman
Sent: Monday, April 23, 2012 23:04
To: xymon at xymon.com
Subject: [Xymon] PROC matching failure due to column bloat

Peeps

I have both Solaris and Linux servers where a large or long-running process causes PROC matching to fail.  Here are some examples:


 7701     1 root       Feb 28 S  24  0.0 00:00:00  0.0   572   2692 /sbin/agetty -L 9600 ttyS0 vt102
 7702     1 root       Feb 28 S  23  0.0 00:00:00  0.0   576   2692 /sbin/agetty -L 9600 ttyS1 vt102
 7704     1 named      Feb 28 S  18  2.4 1-08:59:39  4.4 270500 412784 /usr/sbin/named -u named -f
26498  3293 root     12:47:46 S  14  0.0 00:00:00  0.0   468   2676 sleep 180
This is on Linux.  Note the longer-than-a-day TIME column that pushes the columns after it to the right.

The following is on Solaris 9:

11201 11199  n101649 12:38:54 S  59  0.0        0:00  0.0 1000 1144 vmstat 300 2
11202     1  n101649 12:38:54 S  59  0.0        0:00  0.0  968 1104 sh -c iostat -dxsrP 300 2 1>/tmp/xymon_iostatdisk.redacted
 3244  2965     root   Feb_16 S  59  0.0        5:20  0.1 7104 18736 /opt/OV/lbin/eaagt/opcle -std
 3245  2965     root   Feb_16 S  59  0.0        1:18  0.1 6376 20960 /opt/OV/lbin/eaagt/opcmona
 3253     1     root   Feb_16 S  59  0.9  1-10:46:45  0.8 58168 59632 /usr/local/sbin/named -f
Solaris "ps" output allows more characters for TIME than Linux.  However in this case the memory columns (RSS and VSZ) are larger than expected, pushing a couple of digits over into the process name area.

It seems that Xymon is parsing these based on fixed column sizes, defined for each OS.  The result of these particular examples is that Xymon fails to match on the process name.  Instead, I need to use match strings like so:

        PROC "%^(\d* |^)/usr/local/sbin/named(\s*$|\s)" 1 1 "TEXT=/usr/local/sbin/named"

or

        PROC "%^(\d* |^)/usr/sbin/named(\s*$|\s)" 1 1 "TEXT=/usr/sbin/named"

I guess this email is part "am I doing something wrong", part "does anyone have a better idea", and part feature request (for more awk-like positional matching).

Cheers
Jeremy

Close... AFAIK, it's actually looking for the proper column name in a
given listing (from the first line), and then keying off of that. When the
columns don't line up with the given header, xymond_client examines the
wrong substring. So it's dynamic and static :)


see: xymon-4.3.7/xymond/client/solaris.c:66        
unix_procs_report(hostname, clienttype, os, hinfo, fromline, timestr,
"CMD", "COMMAND", psstr);

and xymond_client.c:958 onward


On the SunOS box I've got, the ps command (xymonclient-sunos.sh) is
providing the following field list, and below that is some of the output
it gets wrong.

I suppose one quick fix if you're getting this a lot might be to manually
change the order of the fields in the client script to "ps -A -o
args,pid,ppid,user,stime,s,pri,pcpu,time,pmem,rss,vsz"


I'm not sure if other processing is going on, but the only drawback might
be slightly odd-looking ps output in your client logs.


HTH,
-jc


-bash-3.2$ ps -A -o pid,ppid,user,stime,s,pri,pcpu,time,pmem,rss,vsz,args
| head -1
  PID  PPID     USER    STIME S PRI %CPU        TIME %MEM  RSS  VSZ COMMAND
-bash-3.2$ ps -A -o pid,ppid,user,stime,s,pri,pcpu,time,pmem,rss,vsz,args
| sort -k8 -r | head
  693   666     root   Mar_28 S  59  0.1  1-12:12:45  0.0 2720 3424
dovecot-auth -w
  160     1     root   Mar_28 S  59  0.2  1-04:06:09  0.1 51552 78848
/usr/sbin/nscd
    3     0     root   Mar_28 S  60  0.1    15:09:32  0.0    0    0 fsflush
  482     1     root   Mar_28 S  59  0.0    10:10:31  0.1 94008 101432
/opt/local/bin/python /usr/local/sbin/denyhosts.py --daemon
--config=/usr/share
    6     0     root   Mar_28 S   0  0.1    08:18:46  0.0    0    0 vmtasks
  461     1     root   Mar_28 S  60  0.0    04:19:34  0.0 1744 3048
/usr/lib/nfs/nfsd
   95     0     root   Mar_28 S  99  0.0    04:02:11  0.0    0    0
zpool-pool
  746   666  dovecot   Mar_28 S  59  0.0    02:33:29  0.0 11376 12584
pop3-login
 1183   666     root   Mar_28 S  59  0.0    02:25:22  0.0 2736 3424
dovecot-auth -w
  666     1     root   Mar_28 S  59  0.0    02:05:38  0.0 2304 3464
/usr/local/sbin/dovecot