Can custom client-side tests leverage the server-side value in analysis.cfg?

Grant, a few quick responses to your message:

The "DS" means "data source". It's a term used by the RRD (round-robin
database) library (and tools) that Xymon uses for storing historical data.

As a general principle, Xymon clients operate in local mode or central
mode. In (the older) local mode, the clients do all of their own analysis
and create their own data and status messages. In (the more modern) central
mode, the clients do a lot less of that, and instead simply gather data for
the server to parse and alert on.

For your requirement, what I would do is to have the client collect the
queue size (and other data) and include this in the client message and one
of the sections, eg [mailq]. The contents of this could simply be the
output of the "mailq" command. You could create a script in the "sections"
subdirectory, and the Xymon client will automatically run it and include
the output in the [mailq] section of the client message. Then on the
server, run a script (eg from tasks.cfg or something like tasks.d/mailq)
that dumps the client data message for every client, and if a [mailq]
section is found, create a status message.

And optionally turn the mail queue size into a graphable data point (either
create a trends message with the number, or use a colon-separated value
format for the Xymon parser to detect). The Xymon doco has a "how to
setup custom graphs" page with a few different ways to do this type of
thing.

J

On Sat, 14 Oct 2023 at 08:20, Grant Taylor via Xymon <xymon at xymon.com>
wrote:

---------- Forwarded message ----------
From: Grant Taylor <user-86150bc7e4bb@xymon.invalid>
To: xymon at xymon.com
Cc:
Bcc:
Date: Fri, 13 Oct 2023 16:18:08 -0500
Subject: Re: [Xymon] Can custom client-side tests leverage the server-side
value in analysis.cfg?
Hi Conor,

Thank you for your reply.

The overall gist of what I've understood thus far is that it's
/indirectly/ possible to have custom client-side scripts use server-side
values by arranging for a copy of those values to be available (cached)
on the client for local use.  --  Is that an accurate summary?

On 10/13/23 1:20?PM, Conor McCarthy wrote:

Yes, a simple way is to use the "config" command, this lets you
download any .cfg (or arbitrary files, if ALLOWALLCONFIGFILES="TRUE"
but don't do that :) files from the server's etc/ directory.

client/bin/xymon $XYMONSERVER "config analysis.cfg" > tmp/analysis.cfg

If you use "include" or "directory" directives in the .cfg files
Xymon will send you the "flattened" expanded contents.

I'm glad to know that the analysis.cfg will be flattened as we're making
extensive use of includes.

You will need to parse that file to do anything useful though.

Alternatively, fetching the hosts.cfg and using xymongrep
   client/bin/xymon $XYMONSERVER "config hosts.cfg" > tmp/hosts.cfg
   HOSTSCFG=tmp/hosts.cfg client/bin/xymongrep mytag:*
might be simpler for simple configuration data that you can put in a
"tag:" for each host.

I'd rather have the values being checked against -- as opposed to what
tests are run -- in the analysis.cfg file.  It seems to me like having
value entries in an additional file could potentially end up in some
confusion.  Which file do I check CPU load in?  What about mail queue
size?  Type thing.

xymongrep is normally used server-side to loop over multiple hosts,
you will likely need to match just one client name from its output
for a client script.

That's the impression I had gotten.

I feel like the ability to access the config, and thus what other
clients are being monitored, is a little bit of an information leak.
Probably not one that matters.  And it's not like I can do anything
about that.

Alternatively, instead of using analysis.cfg the client-local.cfg
file is logically cleaner (and automatically shipped to clients, see
client/tmp/logfetch.HOSTNAME.cfg).

ACK

This specifies the client-side part of log and file checks, although
it's not documented, you can add arbitrary lines to this file, e.g.
  [myhostname]
  x-my-specialvarA one
  x-my-specialvarB two
  log:/var/log/message:16384
  ignore MARK

Good to know.

Each client will pick this up and store its own "personal" stanza
from the global client-local.cfg, this is automatically refreshed
each client run cycle.

Does the server send the personalized version?  Or does the client
receive the full version and only store it's own personal values?

Any non-documented parameters are explicitly ignored, but best placed
above any of the documented ones to make sure.

Noted.

status: yes (though "clear" might be an option),

Interesting call on the "clear" color.

That has the added benefit of not declaring a green / yellow / read.

data: no.

Noted.

There's also a "modify" message you can use after the fact to
change a status/colour (though take care not to trigger flapping).

Ya.  I had seen modify.  I figured that might be part of how the server
side tweaking of client side colors might happen.

I believe there is some restriction on once a message has been modified,
only the same thing can modify it again.  --  I assume that restriction
is only there until a new status message comes in from the client.

Seems like fetching the hosts.cfg and using xymongrep should work
for that without too much trouble, at least as an exercise, and is
reasonably self-evident.

ACK

hosts.cfg:
  10.1.2.3 mailhost1 #  ssh smtps MAILQ:qwarn=200,qcrit=500

Then in a bash script (light on error checking) for example:
   client/bin/xymon $XYMONSERVER "fetch hosts.cfg" > tmp/hosts.cfg
   HOSTSCFG=tmp/hosts.cfg client/bin/xymongrep "MAILQ:*" |
   while read hostip hostname hostcfg; do
     if [[ "$hostname" == ${MACHINEDOTS} ]]; then
       mailcfg=""
       [[ "$hostcfg" =~ MAILQ:(.*) ]] && mailcfg="${BASH_REMATCH[1]}"
       for var in qwarn qcrit; do
         [[ "$mailcfg" =~ "($var)=([^,;]+) ]] && declare
"cfg_${BASH_REMATCH[1]}"="${BASH_REMATCH[2]}"
       done
       # declare -p ${!cfg_*} ##DEBUG##
       # mailq logic using $cfg_qwarn $cfg_qcrit goes here
       # xymon status submit goes here
     fi
   done

This is some very tight and nice Bash.  I like how you're using test
([[) and parameter expansion.  --  Much to come back to and look at in
more detail later.

Aside:  Did you mean "fetch" or "config" when calling xymon?

I think the biggest gotcha that I have is that some of the systems in my
config are using different host names and relying on CLIENT: to make
things match.  But I think that's a matter of finding and grinding out
the CLIENT: from the ${hostcfg}.

You could dispense with the outer while loop by using
   read hostip hostname hostcfg < <(xymongrep MAILQ:* | grep

"$MACHINEDOTS")

ACK

There is no xymongrep equivalent for client-local.cfg, you could adapt
the above to parse the client cached copy (tmp/logfetch.HOSTNAME.cfg) -
simple enough since the server does not provide the entire client-local
config, just the client-specific matching stanza.

I like the idea that the server only provides the client specific stanza.

But I also like the idea that I can pull the analysis.cfg file using
`${XYMON} ${XYMSRV} "config analysis.cfg"`.

Initial skim of the xymongrep man page make me think that it's intended
for the hosts.cfg file and not going to work with other files.  So I'd
need to parse it.

I know there were some old, old mailq client scripts, there's still
some RRD related residue for parsing their output in Xymon. I've
never used it, but likely this was one:

https://web.archive.org/web/20080718210020/http://www.deadcat.net/viewfile.php?fileid=82

$RESEARCH++

The bf/nmailq-bf.sh script therein simply barfs out a bit of sendmail's
"mailq" output. The point being if you have the data in an RRD then
you can also use "DS" in analysis.cfg to alert based on a GAUGE
type metric.  There's more than one way to skin a (dead)cat :)

I have seen many references to "DS", but I'm still not quite sure what
that means.  I need to read the custom graphing again and work to
understand it.

To scale up the configuration complexity or number of discrete
service checks it would be better to dispense with xymongrep and
distribute your own easy-to-parse configuration file(s), or use the
client-local.cfg approach (my preference).

ACK

The ability to pull config files from the server makes this a bit
easier.  It means that the client can get access to the configuration
data to be able to more accurately set clear / green / yellow / red
colors in reports.

Thank you Conor.  I'll re-read, research, and do some thinking.


--
Grant. . . .
unix || die


---------- Forwarded message ----------
From: Grant Taylor via Xymon <xymon at xymon.com>
To: xymon at xymon.com
Cc:
Bcc:
Date: Fri, 13 Oct 2023 16:18:08 -0500
Subject: Re: [Xymon] Can custom client-side tests leverage the server-side
value in analysis.cfg?