Contents of root's crontab in ps list? 🔗 link

So this one has me scratching my head... Noticing that ntpd wasn't running on a secondary server and that xymon didn't notice. Looking at the xymon procs page for the host, I notice that the entire contents of root's crontab is somehow on the ps list! Here's what the offending line in the output looks like. 
19130 19120 562      01:46:38 S  19  0.0 00:00:00  0.0   600   8360 ../bin/xymon 173.166.236.101 status nye,schoolpathways,com.crontab green Thu Feb  5 01:46:39 UTC 2015??# min hour dofmonth month dow?1 * * * * wget --quiet  http://alpha.schoolpathways.com/.hidden/croncheck.php?hostname=`hostname` -O /dev/null?1 1 * * * /root/bin/std/checkyum.php -e?2 3 * * * /root/bin/std/ntpdate.update.sh?1 10 * * * /root/bin/std/fscheck.php check???#SPFS ?1 2 * * * /sis/lr444/bin/scrublogs.php COMMIT?2 3 * * * /sis/lr444/bin/clean_data.php?3 4 * * * /sis/lr444/bin/spfs_clean_temp.sh delete?1 2 * * * /sis/mr-refactor-333/bin/scrublogs.php COMMIT?2 3 * * * /sis/mr-refactor-333/bin/clean_data.php?3 4 * * * /sis/mr-
refactor-333/bin/spfs_clean_temp.sh delete???# zfs ?# */5 * * * * /root/zfs.usage.php?1 1 * * 1 /sbin/zpool scrub spfstank?1 5 * * * /sis/lr444/bin/zfs.snapshot.php nightly?1 10 * * * /sis/mr-
refactor-333/bin/zfs.snapshot.php nightly??# cat crontab files to /tmp/cron_list.log for xymon crontab.sh?0 * * * * /root/bin/std/show_crontab.sh??#XYMON?#?30 * * * * /root/bin/std/smart_chk.sh 2>/dev/null


Yes, that's the entirety of root's crontab, your standard assortment of admin and maintenance scripts... I have no idea how this is even possible, as xymon has no read access to root's cron script. 
I've tried running ~xymonclient/client/bin/xymonclient-linux.sh (as user xymonclient) and it doesn't put out any line like above. Also, I notice that 562 is the "user" even though /etc/passwd has an entry for 562 being xymonclient. 
Environment: CentOS 6
xymon-4.3.7

What else can I provide to help diagnose? 
Ben

Ben

On 5 February 2015 at 13:03, Benjamin Smith <user-bd60c1f964ce@xymon.invalid>
wrote:

# cat crontab files to /tmp/cron_list.log for xymon crontab.sh
0 * * * * /root/bin/std/show_crontab.sh??#XYMON?#?

What does this script (show_crontab.sh) do?

Cheers
Jeremy

The line leads with a standard xymon status header:

../bin/xymon 173.166.236.101 status nye,schoolpathways,com.crontab green
Thu Feb  5 01:46:39 UTC 2015

so you've got a script reporting the crontab contents to the crontab column
in Xymon.  You just "got lucky" in that Xymon's standard client reporting
script happened to grab the "ps -ef" list at the same time the crontab
reporter was running.  I'd bet your crontab reporting script has something
similar to:

   xymon $XYMSRV "status $MACHINE.crontab green `date`

   `cat /tmp/cron_list.log`"

to report the crontab content dumped out to /tmp/cron_list.log by
show_crontab.sh

Ralph Mitchell


On Wed, Feb 4, 2015 at 9:54 PM, Jeremy Laidman <user-71895fb2e44c@xymon.invalid>

wrote:

On 05/02/2015 2:30 PM, "John Palys" <user-6ab3068b00ee@xymon.invalid> wrote:

A simple script. Same script is running on all servers, only some have
the issue reported.

Probably because its only sometimes that the xymon client script runs at
the same time as the cron job. You'll probably find that they drift out of
sync over time, and different servers will be showing this phenomenon at
different times, because the xymon client script runs slightly later each
time (ie not exactly every 5 minutes).

LOGFILE=/tmp/cron_list.log

cat /var/spool/cron/root > /tmp/cron_list.log

snip (why not use $LOGFILE here)

cat /var/spool/cron/cworks >> /tmp/cron_list.log

RESULT=`cat $LOGFILE`


LINE="status $MACHINE.$TEST $COLOR `date`

$RESULT"

$XYMON $XYMSRV "$LINE"

This is the problem. The contents of cron_list.log (ie a copy of the
crontab) is put into $RESULT and in turn into $LINE. Then the text is given
to the xymon command as a parameter, for all to see in ps output.

As soon as you use backticks, you're going to get into trouble. You can't
do anything useful with $RESULT without its contents appearing on a command
line in ps output, should it happen to run at the same time.

Instead, use the alternative form of running xymon, by specifying a
parameter of "@" and feeding the message on stdin.  Like so:

LINE="status $MACHINE.$TEST $COLOR `date`
{ echo "$LINE"; echo; cat $LOGFILE; } | $XYMON $XYMSRV "@"

And get rid of the line that sets $RESULT cos you don't need it anymore.

J
 Jeremy,

A simple script. Same script is running on all servers, only some have the
issue reported.  The xymon ../ext/crontab.sh which sends the output TO the
xymon server is identical to the mis-behaving servers.

#!/bin/bash

LOGFILE=/tmp/cron_list.log

cat /var/spool/cron/root > /tmp/cron_list.log

if [ -f /var/spool/cron/cworks ]; then
echo "================================" >> /tmp/cron_list.log

cat /var/spool/cron/cworks >> /tmp/cron_list.log

else echo ""
fi
#!/bin/sh

#

# Standard XYMON stuff

#

if [ "$XYMONHOME" = "" ]; then

   echo "XYMONHOME is not set... exiting"

   exit 1

fi


if [ ! "$XYMONTMP" ]; then

   . $XYMONHOME/etc/bbdef.sh

fi


#-----------------------

# EDIT Variables here

#-----------------------

TEST=crontab
COLOR=green
LOGFILE=/tmp/cron_list.log
TMPFILE=$XYMONHOME/tmp/$TEST.tmp

RESULT=`cat $LOGFILE`


LINE="status $MACHINE.$TEST $COLOR `date`

$RESULT"

$XYMON $XYMSRV "$LINE"

#rm -f $BBTMP/$TEST.tmp

#rm -f $TMPFILE

On Wed, Feb 4, 2015 at 6:54 PM, Jeremy Laidman <user-71895fb2e44c@xymon.invalid>

wrote:

-- 

John Palys
School Pathways, Inc.
XXX-XXX-XXXX #2010 * Office*