Looking for clarification on Xymon client / server hierarchy. 🔗 link

Hi,

Would someone help me understand the Xymon client / server / proxy hierarchy a little bit better?

My scenario is I have two locations separated by a firewall wherein clients inside can send things out to the larger network, but the xymonnet can't reach in to probe clients in the private LAN.

I had thought that an Xymon proxy might be the answer for this.  --  I did get internal clients to relay updates out through the xymonproxy to the Xymon (display) server.  However xymonnet seems to not utilize the xymonproxy to initiate tests therefrom.

What is the recommended way to have Xymon monitor internal clients that can't be directly reached from the Xymon (display) server?

Aside:  It seems as if the xymonproxy might be for the other way around, to have clients in the wild get messages into a protected Xymon server which can reach out and touch the clients.

Thank you and have a good day.


-- 
Grant. . . .
unix || die

Hi Grant

The xymonnet process needs to be able to send probe packets (eg ping, web
requests, and whatever you're trying to monitor) to the clients. If the
firewall is blocking the probe traffic, then it's not going to work. The
xymon proxy only proxies xymon messages, such as the ones sent by the
xymonnet process to the xymond process when reporting the status of the
probes (success or failure, and round-trip times).

It seems to me that you need a xymonnet process running on the client side
of the firewall. For example, if you can run xymonnet on one of the
clients, then the firewall only needs to allow xymon traffic from the
client to the Xymon server, so that xymonnet can report the status of its
probes.

You can run xymonnet stand-alone, and set environment variables to tell it
where to send its messages. If you already have a xymon client installed on
the client host, you can execute xymonnet from clientlaunch.cfg and it
should then know where to send packets due to the environment that is setup.

The only thing I'm not certain of, is how xymonnet knows which hosts to
probe and what probes to send to them. When xymonnet is running on the
Xymon server, it has access to the hosts.cfg file that's there. When
running elsewhere, I'm not sure. I know that there's a way to fetch the
hosts.cfg contents using xymon messages, so my guess is that xymonnet can
do that too, but might need to be told to do so. And if so, you would only
want that xymonnet instance to probe devices inside the client network, so
you might need to make use of the "NET:" tags in hosts.cfg.

J

On Tue, 17 Oct 2023 at 02:51, Grant Taylor via Xymon <xymon at xymon.com>
wrote:

---------- Forwarded message ----------
From: Grant Taylor <user-86150bc7e4bb@xymon.invalid>
To: xymon at xymon.com
Cc:
Bcc:
Date: Mon, 16 Oct 2023 10:49:42 -0500
Subject: Looking for clarification on Xymon client / server hierarchy.

---------- Forwarded message ----------
From: Grant Taylor via Xymon <xymon at xymon.com>
To: xymon at xymon.com
Cc:
Bcc:
Date: Mon, 16 Oct 2023 10:49:42 -0500
Subject: [Xymon] Looking for clarification on Xymon client / server
hierarchy.

On 10/16/23 7:08?PM, Jeremy Laidman wrote:

Hi Grant

Hi Jeremy,

The xymonnet process needs to be able to send probe packets (eg ping, web requests, and whatever you're trying to monitor) to the clients. If the firewall is blocking the probe traffic, then it's not going to work.

ACK

The xymon proxy only proxies xymon messages, such as the ones sent by the xymonnet process to the xymond process when reporting the status of the probes (success or failure, and round-trip times).

That's what I've deduced.  I'm hoping this (new) thread helps confirm or clarify my deductions.

It seems to me that you need a xymonnet process running on the client side of the firewall. For example, if you can run xymonnet on one of the clients, then the firewall only needs to allow xymon traffic from the client to the Xymon server,?so that xymonnet can report the status of its probes.

ACK

The scenario that I'm working with can be described as a primary Xymon (display) server in one network with a small lab network behind a NATing / SPI firewall.  Clients on the inside side / opposite of the Xymon server are free to send outgoing packets.  It's just that xymonnet running on the Xymon server can't send probes into the clients.

You can run xymonnet stand-alone, and set environment variables to tell it where to send its messages. If you already have a xymon client installed on the client host, you can execute xymonnet from clientlaunch.cfg and it should then know where to send packets due to the environment that is setup.

Oh!  This is promising.

I misinterpreted comments in the tasks.cfg file to mean that xymonnet depended on xymond.  Now it sounds like xymonnet can be satisified by the xymon client.

Running xymonproxy + xymonnet + xymonclient on a system inside of the firewall might do what I'm wanting to do.

The only thing I'm not certain of, is how xymonnet knows which hosts to probe and what probes to send to them. When xymonnet is running on the Xymon server, it has access to the hosts.cfg file that's there. When running elsewhere, I'm not sure. I know that there's a way to fetch the hosts.cfg contents using xymon messages, so my guess is that xymonnet can do that too, but might need to be told to do so.

I currently have a full Xymon (display) server running inside the firewalled network.  But I think that having the full server is complicating things.

I'm guessing that running only the three daemons; xymonclient + xymonproxy + xymonnet inside the firewall, would make my life simpler and wouldn't complicate things with multiple Xymon (display) servers that need to share state.

I'm quite okay with '${XYMON} ${XYMSRV} "config hosts.cfg" > hosts.cfg' on the internal system running xymonnet.

And if so, you would only want that xymonnet instance to probe devices inside the client network, so you might need to make use of the "NET:" tags in hosts.cfg.

I currently have NET: tags and XYMONNETWORK parameters on the systems running xymonnet.

It's working.  But I'm needing to run a xymonproxy on 1984 and distributing messages to xymond on 1985 on localhost and xymond on 1984 on the main Xymon (display) server.

Hence this thread inquiring about a cleaner method of having a topology.

Thank you again Jeremy.


-- 
Grant. . . .
unix || die

On Tue, 17 Oct 2023 at 12:34, Grant Taylor via Xymon <xymon at xymon.com>

wrote:

Indeed. Perhaps more precisely, it can be satisfied by xymonlaunch, which
also runs the main Xymon client script (xymonclient.sh). It's also possible
to run xymonnet from cron, or set it up as a systemd service, or even
trigger its execution by running from a remote ssh session (something like
"ssh client-host /usr/local/bin/xymonnet
--env=/usr/local/etc/xymonnet.conf"). In most cases, running from within a
xymoncmd environment (which is implied when run by xymonlaunch) takes away
the need to explicitly set all of the required options. So you could have a
cron job like so:
*/5 * * * * xymon /usr/lib/xymon/client/bin/xymoncmd
/usr/local/bin/xymonnet >>/var/log/xymon/xymonnet.log 2>&1

But it's worth taking a look at the xymonnet entries in the tasks.cfg file
on the Xymon server. There is a xymonnet-again (I think) task that runs
more frequently but only performs probes for failed tests. In this way when
a fault is detected, the polling rate increases so that you can more
quickly see when a service is restored, and SLA times are more accurate.

Running xymonproxy + xymonnet + xymonclient on a system inside of the

I don't think xymonproxy is required for this. If your clients can talk
directly to your xymon server on TCP/1984, then that's all xymonclient.sh
and xymonnet need to be able to do. I think xymonproxy is only required if
your clients can't connect directly to your server. And it will add delays,
and become an extra point of failure.

If you haven't already, look into msgcache and xymonfetch. These can be
used to bridge networks in a slightly different way to xymonproxy.

I'm quite okay with '${XYMON} ${XYMSRV} "config hosts.cfg" > hosts.cfg'

Makes sense, and should work fine.

I actually like "xymongrep" for its ability to do searches. By default it
looks for hosts.cfg, but if given the --loadhostsfromxymond switch, it'll
connect to the Xymon server and send a "config hosts.cfg" command and
filter for the tags you're interested in. So you could do something like:

xymoncmd xymongrep "NET:behindfirewall" > hosts.cfg

(multiple search strings can be used, and lines that match any of them will
be returned)

This brings up a possible issue for some installations and security
postures. In a multi-network multi-security-level environment, some
consider the "config" command to be a feature that's not worth the
potential risk. For example, if there's a coding error in Xymon, there
might be a way to request the config file /etc/shadow. Or if someone
doesn't realise that hosts.cfg is potentially exposed outside the Xymon
server in this way, they might put a password or private key in the
hosts.cfg file (eg in a tag for a custom extension). An example of this is
that the "devmon" add-on uses the DEVMON: tag to specify SNMP parameters,
including community string, and the operator might not realise that the
community strings are accessible by any Xymon client.

If this is of concern to you, then you should consider ways to mitigate
this. Choosing to not use this feature in your scripts doesn't mean an
attacker can't do so.

I don't know if there's a way to prevent "config" messages from being
honoured by the xymond process. It's possible to have xymond ignore
messages (including config) from IP addresses that aren't in hosts.cfg, so
this will minimise any exposure.

There's a similar feature where you can send a "download" message and
receive a copy of the named file. The download feature can be disabled by
adding "--no-download" to the xymond command line arguments.

Cheers
Jeremy

On 10/16/23 9:51?PM, Jeremy Laidman wrote:

Okay.  That sounds very promising.

I had naively assumed that xymonnet used some sort of IPC to xymond and was thus dependent on it.

Now I get the impression that it's more that xymonnet needs a properly configured environment and that the easiest way to do that is to use Xymon / xymonlaunch to manage that for you.  But it sounds like it's quite possible to crate said environment and call xymonnet directly.

Very good to know.

I ran into a situation this evening where I need to duplicate the firewalled environment behind another host that is currently a stand alone client.

But it's worth taking a look at the xymonnet entries in the tasks.cfg file on the Xymon server. There is a xymonnet-again (I think) task that runs more frequently but only performs probes for failed tests. In this way when a fault is detected, the polling rate increases so that you can more quickly see when a service is restored, and SLA times are more accurate.

Yep.

I suspect that I'll be installing the distro provided xymon-server and tweaking tasks.cfg to run the client, xymonnet, and xymonnet-again.  But not the xymond / display server.

I don't think xymonproxy is required for this. If your clients can talk directly to your xymon server on TCP/1984, then that's all xymonclient.sh and xymonnet need to be able to do.

Good to know.

I think xymonproxy is only required if your clients can't connect directly to your server. And it will add delays, and become an extra point of failure.

Noted.

If you haven't already, look into msgcache and xymonfetch. These can be used to bridge networks in a slightly different way to xymonproxy.

I'm aware of them and I've skimmed (parts of) their manual pages.  I've not yet needed to go to -- what I call -- the poll methodology. Currently, -- ... -- the push methodology has been sufficient to get data from clients to the server.

The biggest hiccup I've had is running xymonnet inside the firewalled network.

Makes sense, and should work fine.

:-)

I actually like "xymongrep" for its ability to do searches. By default it looks for hosts.cfg, but if given the --loadhostsfromxymond?switch, it'll connect to the Xymon server and send a "config hosts.cfg" command and filter for the tags you're interested in. So you could do something like:

Interesting idea.

This brings up a possible issue for some installations and security postures. In a multi-network multi-security-level environment, some consider the "config" command to be a feature that's not worth the potential risk. For example, if there's a coding error in Xymon, there might be a way to request the config file /etc/shadow.

Understood and appreciated.

But if Xymon is running as anything other than root, it's chances of reading the /etc/shadow file are -- dare I say -- exceedingly small.

But I completely get that Xymon coding errors could make it possible to read any file that the user that Xymon is running as can read.

Or if someone doesn't realise that hosts.cfg is potentially exposed outside the Xymon server in this way, they might put a password or private key in the hosts.cfg file (eg in a tag for a custom extension). An example of this is that the "devmon" add-on uses the DEVMON: tag to specify SNMP parameters, including community string, and the operator might not realise that the community strings are accessible by any Xymon client.

Yep.

If this is of concern to you, then you should consider ways to mitigate this. Choosing to not use this feature in your scripts doesn't mean an attacker can't do so.

Yep.

I fully understand that the capability can be used by a bad actor, even if I don't use it myself.

I don't know if there's a way to prevent "config" messages from being honoured by the xymond process. It's possible to have xymond ignore messages (including config) from IP addresses that aren't in hosts.cfg, so this will minimise any exposure.

I should investigate that.

I assume that a firewall also helps here.

There's a similar feature where you can send a "download" message and receive a copy of the named file. The download feature can be disabled by adding "--no-download" to the xymond command line arguments.

Doesn't the download feature also have a more lax mode where it will allow you to download things that aren't config files?  Or am I conflating config and download?

I also wonder where the command comes into play to have clients download updated (tar files) from the central server and install them.


-- 
Grant. . . .
unix || die

On Tue, 17 Oct 2023 at 14:19, Grant Taylor via Xymon <xymon at xymon.com>

wrote:

Correct. The man page for xymonnet should cover the various env vars that
it supports. I think the only ones that might be required are some
combination of (XYMSRV, XYMONSERVERS, XYMSERVERS) as per doco to specify
the display servers, and access to the hosts.cfg file, and XYMONTMP. In
fact if you run xymonnet with the --loadhostsfromxymond switch, it doesn't
even need a local copy of hosts.xfg. Try something like this from a device
in your enclave, and see if the output looks OK, then drop the --no-update
to run it for real:

XYMSERVERS=xymon-server-or-IP XYMONTMP=/tmp
/usr/lib/xymon/server/bin/xymonnet --no-update --loadhostsfromxymond

Optionally, try the --debug and/or --dump switches. You can specify the
FQDN of a target server if you just want to probe one device (eg for
testing).

You can set XYMONNETWORK to something in the NET: tags in hosts.cfg, and
only those devices will be polled. So in hosts.cfg:

10.1.2.3 mywebserver.example.com # NET:enclave

XYMSERVERS=xymon-server-or-IP XYMONTMP=/tmp XYMONNETWORK=enclave
/usr/lib/xymon/server/bin/xymonnet --loadhostsfromxymond

But if Xymon is running as anything other than root, it's chances of

reading the /etc/shadow file are -- dare I say -- exceedingly small.

Yep, agreed. And if that's the only exposure, then we're golden. Might be a
private key that the Xymon user accesses to authenticate for one of its
probes to an HTTPS server with client auth. Might be /var/tmp/ps-dump that
someone is running from their cron, to diagnose some fault, not realising
that someone else's process has "--with-password=M0nkey", because that user
didn't realise that args are visible to all. Might be a core dump that
someone copied to /tmp/, without knowing that they can contain passwords
and private keys.

Of course most possible exploits are really unlikely, and require a bit of
stupidity and a bit of mis-fortune. I'm not trying to suggest likely
exposures, just possible exposures under some circumstances. We can't
mitigate all of them, but we can address the likely ones even if it takes
effort, and we also should address the unlikely ones where it takes very
little effort.

I suppose my point here is, if a client doesn't need to access an arbitrary
file on a server, then it shouldn't be able to. If a client doesn't need to
access any file on a server, then it shouldn't be able to. And a feature
that permits this, and is on by default, and is seldom used or necessary,
and is possibly not known about by the sysadmin, probably should be
addressed, or at least assessed.

Only for some threat models. Every xymon client has to be able to connect
to the xymon server on TCP/1984[*]. A firewall can allow or deny that. But
a firewall can't allow status and data messages while blocking config
messages. I'm not aware of a DPI firewall that can parse Xymon packets.

I actually don't know. I don't use either of these features.

I also wonder where the command comes into play to have clients download

Yes. If my memory serves, the original BigBrother, on which Xymon was
based, was often deployed so that it could upgrade itself. This process
required that the BB server had tarballs of the various clients. The
clients would periodically check their versions with the latest available,
and if there was an updated, the client would fetch it and untar over
itself.

This mechanism was in place when compiling from source code was the only
way to deploy open source. So it seemed like a good idea at the time. And
it has merit. But these days, we're all used to using packages, and we make
good use of the additional features a package manage can bring. So the way
I see the download command is that it's a relic from a previous generation,
that most people won't ever use. Might come in handy, but if there's
another way to do it, probably use the nother way.

J

On 10/17/23 12:37?AM, Jeremy Laidman wrote:

Correct. The man page for xymonnet should cover the various env vars that it supports. I think the only ones that might be required are some combination of (XYMSRV, XYMONSERVERS, XYMSERVERS) as per doco to specify the display servers, and access to the hosts.cfg file, and XYMONTMP. In fact if you run xymonnet with the --loadhostsfromxymond?switch, it doesn't even need a local copy of hosts.xfg. Try something like this from a device in your enclave, and see if the output looks OK, then drop the --no-update to run it for real:

XYMSERVERS=xymon-server-or-IP XYMONTMP=/tmp /usr/lib/xymon/server/bin/xymonnet --no-update --loadhostsfromxymond

I'll give that a try.

I think there is some benefit to leveraging xymonlaunch as in it's somewhat more common way to start / stop a process than modifying crontab.  But that's more outside of Xymon's wheelhouse.

Optionally, try the --debug and/or --dump switches. You can specify the FQDN of a target server if you just want to probe one device (eg for testing).

I need to try this with a couple of systems that seem to keep flapping on TLS protected services when their unencrypted counterparts don't flap.  --  I think I saw a bug about a patch this year about a latency sensitive issue that may be related.

You can set XYMONNETWORK to something in the NET: tags in hosts.cfg, and only those devices will be polled. So in hosts.cfg:

Yep.  I'm doing exactly this.

I've got three separate Xymon installations that I'm helping administer:

  - $DAY_JOB -- has two environments, production in primary DC and DR in a secondary DC.  There is incomplete communications between prod and DR.   Hence the need for xymonnet, and maybe xymonproxy, in DR for things in DR to be tested properly.
  - $PERSONAL -- single site, common configuration.
  - $HELPING_A_FRIEND -- Now three sites; production, office, and friend's house.  Where office and friend's house are basically SOHO NATing routers that port forward SSH to an internal host.  So I'll be looking at xymonclient + xymonnet therein.

10.1.2.3 mywebserver.example.com # NET:enclave

Yep, I've got something like this already.  It's working great.

XYMSERVERS=xymon-server-or-IP XYMONTMP=/tmp XYMONNETWORK=enclave /usr/lib/xymon/server/bin/xymonnet --loadhostsfromxymond

I'll definitely give that a try for manual debugging.

But for normal production, I think a distro package of xymon-server with a trimmed down tasks.cfg to run xymonclient and xymonnet will suffice and be clean with OS init script expectations.

Yep, agreed. And if that's the only exposure, then we're golden. Might be a private key that the Xymon user accesses to authenticate for one of its probes to an HTTPS server with client auth. Might be /var/tmp/ps-dump that someone is running from their cron, to diagnose some fault, not realising that someone else's process has "--with-password=M0nkey", because that user didn't realise that args are visible to all. Might be a core dump that someone copied to /tmp/, without knowing that they can contain passwords and private keys.

All very valid examples of -- what I generally consider -- Oops! type mistakes / unintentional exposures.

Of course most possible exploits are really unlikely, and require a bit of stupidity and a bit of mis-fortune. I'm not trying to suggest likely exposures, just possible exposures under some circumstances. We can't mitigate all of them, but we can address the likely ones even if it takes effort, and we also should address the unlikely ones where it takes very little effort.

ACK

I suppose my point here is, if a client doesn't need to access an arbitrary file on a server, then it shouldn't be able to. If a client doesn't need to access any file on a server, then it shouldn't be able to. And a feature that permits this, and is on by default, and is seldom used or necessary, and is possibly not known about by the sysadmin, probably should be addressed, or at least assessed.

I completely agree.

I've often heard this as "business justification".  Which I have heard a translation of that I like better; "what benefit does doing this provide to the business".  IMHO "justification" can have negative connotation and / or baggage associated.

Only for some threat models. Every xymon client has to be able to connect to the xymon server on TCP/1984[*]. A firewall can allow or deny that. But a firewall can't allow status and data messages while blocking config messages.

ACK

I'm not aware of a DPI firewall that can parse Xymon packets.

I strongly suspect that I could configure an IPTables "string" match to detect requests in normal unencrypted traffic.

But your point is taken and I digress.  ;-)

I actually don't know. I don't use either of these features.

I've used the config support a few times.

I like the idea of having xymonnet `--loadhostsfromxymond`, which I assume needs `config` support.

Yes. If my memory serves, the original BigBrother, on which Xymon was based, was often deployed so that it could upgrade itself. This process required that the BB server had tarballs of the various clients. The clients would periodically check their versions with the latest available, and if there was an updated, the client would fetch it and untar over itself.

I used the option to pull in a tar file using `clientversion` (syntax from memory) once.  I can see using it to distribute additional local files and / or extensions.

We'll see.

This mechanism was in place when compiling from source code was the only way to deploy open source. So it seemed like a good idea at the time. And it has merit. But these days, we're all used to using packages, and we make good use of the additional features a package manage can bring. So the way I see the download command is that it's a relic from a previous generation, that most people won't ever use. Might come in handy, but if there's another way to do it, probably use the nother way.

Eh ... package managers have their place.  But they don't, and IMHO, shouldn't cover everything.  Config and / or extensions probably don't go in custom distro packages.  --  Re-using a distro distribution system for custom configuration packages may make sense.  Arguably using a configuration management system makes the most sense, but they aren't always available.  It's a balancing act of what works in each use case with the fewest side effects / exposures.


-- 
Grant. . . .
unix || die