Xymon script question 🔗 link

Hi folks,

My network has about a dozen Xymon clients and one Xymon server, all  running Debian stretch. Yesterday I wrote my first Xymon script for it  (using the example on the 'Xymon Tips and Tricks' page), which checks  if the host it runs on needs a reboot in case the kernel has been  updated. This script runs fine on all of the clients (a new column for  it was automatically added to the 'Current Status' overview), but not  on the server where it is apparently being ignored. There is, of  course, also a Xymon client installed on the server and up until now  it always seemed to work the same as all the other clients.

So, do Xymon client scripts that are run on the server need to be  different, or does something else need to be done before they will  work there?

Also, how can I best go about troubleshooting custom Xymon scripts?  For example, is there a way to run them manually in debug mode?

Thanks,

Jaap

Hi Jaap,

The best way I've found to run them in debug mode is to simply run them by hand and put an echo statement in front of the xymon command in the script, so that the script and all data is sent to stdout.  That lets you see exactly what it's sending to the xymon server.  You will need to run it with xymoncmd to ensure the environment variables are set correctly, like so: xymoncmd ./$YOURTESTNAMEHERE.

I've never run scripts in the client instance on the xymon server itself, I've always run them in the server instance and they've worked exactly like on a client.  I think the client is called differently on a xymon server and may not process the ext scripts the same as on a non-server instance of xymon.  Someone else can correct me if I'm wrong.

I am also interested in your script as I was thinking about something similar. I would appreciate it if you would share your script with me, you may send it to me off-list if you wish.

Thanks!


Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: XXX-XXX-XXXX |  |  Mobile: XXXXXXXXXX | Email: user-9678697f1438@xymon.invalid

-----Original Message-----
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Jaap Winius
Sent: Wednesday, July 5, 2017 8:53 AM
To: xymon at xymon.com
Subject: [Xymon] Xymon script question

Hi folks,

My network has about a dozen Xymon clients and one Xymon server, all
running Debian stretch. Yesterday I wrote my first Xymon script for it
(using the example on the 'Xymon Tips and Tricks' page), which checks
if the host it runs on needs a reboot in case the kernel has been
updated. This script runs fine on all of the clients (a new column for
it was automatically added to the 'Current Status' overview), but not
on the server where it is apparently being ignored. There is, of
course, also a Xymon client installed on the server and up until now
it always seemed to work the same as all the other clients.

So, do Xymon client scripts that are run on the server need to be
different, or does something else need to be done before they will
work there?

Also, how can I best go about troubleshooting custom Xymon scripts?
For example, is there a way to run them manually in debug mode?

Thanks,

Jaap

This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.

Quoting Scot Kreienkamp <user-9678697f1438@xymon.invalid>:

Hi Scot,

Thanks for your reply, I'll try it out in a bit. In the mean time,  here's my script:

==========================
#!/bin/bash

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

COLUMN=Reboot	# Name of the column
COLOR=green	# By default, everything is OK
MSG="Reboot required"

RUN=$(uname -r |awk -F - '{print $1}')	# Version of the running kernel
IRF=$(find /boot -type f -name "initrd.img-$RUN-*") # Find matching  init.rd file
FMD=$(stat -c %Y $IRF)	# Modify date of init.rd file in seconds since 1970
UPT=$(cat /proc/uptime |awk -F \. '{print $1}')	# Seconds of uptime,  rounded down
EPC=$(date +%s)		# Current epoch (seconds since 1970)
DIF=$(($EPC-$FMD))	# Age of init.rd file in seconds

if [ $DIF -gt $UPT ]
then
	MSG="${MSG}:

	No. Already running the latest version of Linux kernel $RUN.
	"
else
	COLOR=yellow

	LFT=$(stat -c %y $IRF |awk -F \. '{print $1}') # Last modified time
	LBT=$(who -b |awk '{print $3" "$4}') # Last boot time

	MSG="${MSG}:

	Yes. A new version of Linux kernel $RUN was installed on $LFT after  the last reboot on $LBT.
	"
fi

# Tell Xymon about it
$XYMON $XYMSRV "status $MACHINE.$COLUMN $COLOR `date`

${MSG}
"

exit 0

==========================

I have the unattended-upgrades package installed on all of the  clients, which are only set to notify me if an error occurs, so this  way I'll know when it's time to reboot any of them.

Is there a Xymon repository somewhere for scripts like this?

Cheers,

Jaap

There is a repository on xymonton.org but not much is contributed to it.  This would need some modification to work on my RH systems but it's a good start.  Thanks for sharing.

Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: XXX-XXX-XXXX |  |  Mobile: XXXXXXXXXX | Email: user-9678697f1438@xymon.invalid
-----Original Message-----

From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Jaap Winius
Sent: Wednesday, July 5, 2017 9:52 AM
To: xymon at xymon.com
Subject: Re: [Xymon] Xymon script question

Quoting Scot Kreienkamp <user-9678697f1438@xymon.invalid>:

Hi Scot,

Thanks for your reply, I'll try it out in a bit. In the mean time,
here's my script:

==========================
#!/bin/bash

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

COLUMN=Reboot   # Name of the column
COLOR=green     # By default, everything is OK
MSG="Reboot required"

RUN=$(uname -r |awk -F - '{print $1}')  # Version of the running kernel
IRF=$(find /boot -type f -name "initrd.img-$RUN-*") # Find matching
init.rd file
FMD=$(stat -c %Y $IRF)  # Modify date of init.rd file in seconds since 1970
UPT=$(cat /proc/uptime |awk -F \. '{print $1}') # Seconds of uptime,
rounded down
EPC=$(date +%s)         # Current epoch (seconds since 1970)
DIF=$(($EPC-$FMD))      # Age of init.rd file in seconds

if [ $DIF -gt $UPT ]
then
        MSG="${MSG}:

        No. Already running the latest version of Linux kernel $RUN.
        "
else
        COLOR=yellow

        LFT=$(stat -c %y $IRF |awk -F \. '{print $1}') # Last modified time
        LBT=$(who -b |awk '{print $3" "$4}') # Last boot time

        MSG="${MSG}:

        Yes. A new version of Linux kernel $RUN was installed on $LFT after
the last reboot on $LBT.
        "
fi

# Tell Xymon about it
$XYMON $XYMSRV "status $MACHINE.$COLUMN $COLOR `date`

${MSG}
"

exit 0

==========================

I have the unattended-upgrades package installed on all of the
clients, which are only set to notify me if an error occurs, so this
way I'll know when it's time to reboot any of them.

Is there a Xymon repository somewhere for scripts like this?

Cheers,

Jaap


This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.

Quoting Scot Kreienkamp <user-9678697f1438@xymon.invalid>:

... like so: xymoncmd ./$YOURTESTNAMEHERE.

Ah, that works. And the script seems to be behaving itself.

I've never run scripts in the client instance on the xymon server  itself, I've always run them in the server instance and they've  worked exactly like on a client.  I think the client is called  differently on a xymon server and may not process the ext scripts  the same as on a non-server instance of xymon. ...

If I place the script in the /usr/lib/xymon/server/ext/ directory as  well, nothing happens. I had earlier modified  /etc/xymon/clientlaunch.cfg as I did on the other client systems, to  let Xymon know about the script, but I guess that isn't the correct  approach on a Xymon server, is it?

Cheers,

Jaap

On my server (RedHat based, so YMMV) I generally put a file in /etc/xymon/tasks.d, one file for each external script.  Same format as for clientlaunch though.

Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: XXX-XXX-XXXX |  |  Mobile: XXXXXXXXXX | Email: user-9678697f1438@xymon.invalid
-----Original Message-----

From: Jaap Winius [mailto:user-16708c3648e1@xymon.invalid]
Sent: Wednesday, July 5, 2017 12:29 PM
To: Scot Kreienkamp <user-9678697f1438@xymon.invalid>
Cc: xymon at xymon.com
Subject: RE: [Xymon] Xymon script question

Quoting Scot Kreienkamp <user-9678697f1438@xymon.invalid>:

Ah, that works. And the script seems to be behaving itself.

If I place the script in the /usr/lib/xymon/server/ext/ directory as
well, nothing happens. I had earlier modified
/etc/xymon/clientlaunch.cfg as I did on the other client systems, to
let Xymon know about the script, but I guess that isn't the correct
approach on a Xymon server, is it?

Cheers,

Jaap

This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.

Quoting Scot Kreienkamp <user-9678697f1438@xymon.invalid>:

Okay, I created a file, called /etc/xymon/tasks.d/reboot, with this content:

==========

[reboot]
	ENVFILE /usr/lib/xymon/server/etc/xymonclient.cfg
	CMD /usr/lib/xymon/server/ext/ufadiff.sh
	LOGFILE /var/log/xymon/reboot.log
	INTERVAL 5m

include /var/run/xymon/clientlaunch-include.cfg

==========

But, still no joy.

Cheers,

Jaap

You shouldn't need the include statement, not sure if that's throwing it off?  Did you restart xymon so it can pick up the new task file, and is it readable by the xymon user?

Anything in the logs or /var/log/xymon/reboot.log?  /var/log/xymon/reboot.log should exist as soon as the xymon server is restarted.

Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: XXX-XXX-XXXX |  |  Mobile: XXXXXXXXXX | Email: user-9678697f1438@xymon.invalid
-----Original Message-----

From: Jaap Winius [mailto:user-16708c3648e1@xymon.invalid]
Sent: Wednesday, July 5, 2017 3:32 PM
To: Scot Kreienkamp <user-9678697f1438@xymon.invalid>
Cc: xymon at xymon.com
Subject: RE: [Xymon] Xymon script question

Quoting Scot Kreienkamp <user-9678697f1438@xymon.invalid>:

Okay, I created a file, called /etc/xymon/tasks.d/reboot, with this content:

==========

[reboot]
        ENVFILE /usr/lib/xymon/server/etc/xymonclient.cfg
        CMD /usr/lib/xymon/server/ext/ufadiff.sh
        LOGFILE /var/log/xymon/reboot.log
        INTERVAL 5m

include /var/run/xymon/clientlaunch-include.cfg

==========

But, still no joy.

Cheers,

Jaap

This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.

Quoting Scot Kreienkamp <user-9678697f1438@xymon.invalid>:

You shouldn't need the include statement, not sure if that's throwing it off?

Okay, I removed that line.

I restarted xymon and /etc/xymon/tasks.d/reboot has 644 permissions.

It still doesn't exit.

Cheers,

Jaap

On 06/07/17 09:16, Jaap Winius wrote:

You should have a xymonlaunch log file, does that report anything related to this task?

PS, on debian I always install the hobbit-plugins package, which includes a check for this already.

-- 
Adam Goryachev
Website Managers
P: +61 2 8304 0000                    user-eaec2ffb4cbc@xymon.invalid
F: +61 2 8304 0001                     www.websitemanagers.com.au

On 06/07/17 09:16, Jaap Winius wrote:

You should have a xymonlaunch log file, does that report anything 
related to this task?

PS, on debian I always install the hobbit-plugins package, which 
includes a check for this already.

-- 
Adam Goryachev
Website Managers
P: +61 2 8304 0000                    user-eaec2ffb4cbc@xymon.invalid
F: +61 2 8304 0001                     www.websitemanagers.com.au


-- 

Adam Goryachev Website Managers www.websitemanagers.com.au
-- 
The information in this e-mail is confidential and may be legally privileged.
It is intended solely for the addressee. Access to this e-mail by anyone else
is unauthorised. If you are not the intended recipient, any disclosure,
copying, distribution or any action taken or omitted to be taken in reliance
on it, is prohibited and may be unlawful. If you have received this message
in error, please notify us immediately. Please also destroy and delete the
message from your computer. Viruses - Any loss/damage incurred by receiving
this email is not the sender's responsibility.

Quoting Jaap Winius <user-16708c3648e1@xymon.invalid>:

There are no doubt many more ways to improve the script I uploaded,  but if you take this line:

   RUN=$(uname -r |awk -F - '{print $1}') # Version of the running kernel

and replace it with

   RUN=$(uname -r |awk -F - '{print $1"-"$2}') # Version of the running kernel

then it will not malfunction in cases where multiple versions of a  kernel are present with the same major and minor revisions, but  different patch numbers.

Cheers,

Jaap

Quoting Adam Goryachev <user-eaec2ffb4cbc@xymon.invalid>:

You should have a xymonlaunch log file, does that report anything  related to this task?

Nope. Nothing.

PS, on debian I always install the hobbit-plugins package, which  includes a check for this already.

This didn't fix the problem for me, although it did at least produce a  new 'libs' column with a warning about upgraded libs linked in running  processes.

But, eventually I did fix the problem on my own. :-)  I got the idea  after reading this message:

   http://lists.xymon.com/archive/2012-October/035800.html

Thus, the solution was to add an extra line to the end of  /etc/xymon/tasks.cfg:

   directory /etc/xymon/tasks.d

I would consider this omission a bug, as without this line everything  placed in /etc/xymon/tasks.d is simply ignored.

Cheers,

Jaap