query-frontend (CLI) for `clientlog`, `xymondboard` and `xymondlog` 🔗 link

Brothers,

currently I’m building a fronted for the various query-commands that can be sent to `xymond`. It is still in it’s early stages but I found myself using it daily already. So others may benefit too.

It’s named `xymonq` (“xymon query”).

Two examples may give a basic idea:

1. Get a list of hosts on page `dc1`:
Traditionally this looks like this:

	$ xymon 127.0.0.1 "xymondboard page=dc1 test=info fields=hostname"

With xymonq

	$ xymonq -P dc1 -L

2. Get disk information this group of hosts:

	$ for my_host in $(xymon 127.0.0.1 "xymondboard page=dc1 test=info fields=hostname"); do
		echo “HOST=$my_host"
		xymon 127.0.0.1 "xymondlog $my_host.disk"
	done

And again the short version with xymonq:

	$ xymonq -q clientlog -P dc1 -s df


More information and download here: http://www.it-eckert.com/software/xymonq/ <http://www.it-eckert.com/software/xymonq/>;

Feedback welcome!

All the best,
Thomas

On Tue, Jun 23, 2015, at 13:57, Thomas Eckert wrote:

This looks really useful. Thanks!!

On Tue, June 23, 2015 12:50 pm, Mark Felder wrote:

On Tue, Jun 23, 2015, at 13:57, Thomas Eckert wrote:

Brothers,

currently I’m building a fronted for the various query-commands that
can
be sent to `xymond`. It is still in it’s early stages but I found
myself
using it daily already. So others may benefit too.

It’s named `xymonq` (“xymon query”).

		echo “HOST=$my_host"

Nifty!

One thing that might be useful is a sort of large-volume mode, optionally
shifting some of the processing from separate TCP xymond queries to a
local grep string, at the expense of network transfer instead.


We often use a perl one-liner (normally in a scriptlet, but on the CL
here) to re-wrap the resulting test message into newlines with the
hostname pre-pended on each line. Since it's a pipe, it's quick, and this
lets us grep lines after the fact:

]$ xymon localhost "xymondboard test=disk fields=hostname,msg" | perl -ne
'm/^(\S+)|/ and $meh=$1 and s/(\\n|<br\/>)/\n$meh: /g and print;' | grep
/boot

server01.example.com|status: /dev/sda1    495844   39330    430914      
9% /boot
server02.example.com|status: /dev/sda1     99150   62675     31355     
67% /boot
server03.example.com|status: /dev/sda1     99150   60628     33402     
65% /boot
server04.example.com|status: /dev/sda1     99150   62856     31174     
67% /boot

etc... (The "<br>" check is so we can use this for HTML content returned
in an HTTP check too.)


-jc

Hi J.C., hi Mark,

thank you for your feedback!

Right, a volume-mode does make sense for larger setups for sure. Some testing shows that the speed improvements are really significant.

Only `xymondboard` is usable for such an optimisation (as it can return data for multiple hosts/tests matching given CRITERIA) I've implemented it as a special case for now:

If no "HOST="-separator is requested (`-S` option) for a xymondboard query "volume-mode" kicks in automatically:

	// slow version w/ one tcp-connection per host:
	root at bb:~# /vagrant/xymonq -q xymondboard -H bb.local -T disk -f hostname,line1,msg -S
	HOST=bb.local
	bb.local|green Fri Jul 17 11:18:03 MSK 2015 - Filesystems ok|status bb,local.disk green Fri Jul 17 11:18:03 MSK 2015 - Filesystems ok\nFilesystem                                             1024-blocks    Used Available Capacity Mounted on\n/dev/sda1                                                    40558164 1614424  36883504       5% /\n/dev/disk/by-uuid/8769c421-b3e7-42b8-979a-28a8add5d0f3    40558164 1614424  36883504       5% /\n

	// volume-mode (no `-S` here):
	root at bb:~# /vagrant/xymonq -q xymondboard -H bb.local -T disk -f hostname,line1,msg
	bb.local:line1:green Fri Jul 17 11:23:04 MSK 2015 - Filesystems ok
	bb.local:msg:status bb,local.disk green Fri Jul 17 11:23:04 MSK 2015 - Filesystems ok
	bb.local:msg:Filesystem                                             1024-blocks    Used Available Capacity Mounted on
	bb.local:msg:/dev/sda1                                                    40558164 1614424  36883504       5% /
	bb.local:msg:/dev/disk/by-uuid/8769c421-b3e7-42b8-979a-28a8add5d0f3    40558164 1614424  36883504       5% /

The post-processing is handled by an external awk-script (`unescape-xymonq.awk`, currently has to be stored in the same directory as `xymonq`) that

- prefixes each line with "[hostname]:[fieldname]:", is the hostname is not printed is is left empty.
- rewrites the escapes in "msg"

The awk-script is tested w/ gawk, mawk and busybox-awk.

A general post-processing implementation might be the best, enabling for something like

	xymonq .... -c unescape-xymonq.awk -c filter2.pl -c filter3.py
	
All filters stored in a cont-able place. These filters can be re-used easily and even shared with others.

The data provided by `xymondlog` could be emulated by `xymondboard fields=line1,msg` and minor post processing. I see no reason to use `xymondlog` at all -- it might be the best to remove this completely from `xymonq`.
I assume that `xymondlog` existed before the more capable `xymondboard` and now serves as a quick "get the status for host.test" from the shell without worrying about the escaping required with `xymondboard`.

For `clientlog` there is no other way to get  the data but using a loop with one tcp-connection to xymond for each host (no other way to using the slow "looping mode" in this case).

The described version is available here: http://www.it-eckert.com/downloads/download/beta/xymonq-HEAD-2015-07-17.tgz

Any feedback much appreciated.

All the best
Thomas