PORTS and STATE syntax 🔗 link

I'm not successful filtering on the connection state associated with a port.
None of the syntax variations I have tried have been successful.
If I remove the STATE specifier, matches are found.

There are multiple hosts connecting to the same port:

ESTAB      0      0              10.160.8.130:61617         10.160.8.132:57765
ESTAB      0      0              10.160.8.130:61617         10.160.8.132:57766
ESTAB      0      0              10.160.8.130:61617         10.160.8.132:57768
ESTAB      0      0              10.160.8.130:61617         10.160.8.133:45096
ESTAB      0      0              10.160.8.130:61617         10.160.8.133:45104
ESTAB      0      0              10.160.8.130:61617         10.160.8.133:45107
ESTAB      0      0              10.160.8.130:61617          130.118.4.2:36141
ESTAB      0      0              10.160.8.130:61617          130.118.4.2:36150
ESTAB      0      0              10.160.8.130:61617          130.118.4.2:36151
ESTAB      0      0              10.160.8.130:61617         136.177.16.3:34320
ESTAB      0      0              10.160.8.130:61617         136.177.16.3:34321
ESTAB      0      0              10.160.8.130:61617         136.177.16.3:34324
ESTAB      0      0              10.160.8.130:61617       137.227.240.32:50726
ESTAB      0      0              10.160.8.130:61617       137.227.240.32:50727
ESTAB      0      0              10.160.8.130:61617       137.227.240.32:50729
LISTEN     0      0                         *:61617                    *:*

I've set up several port monitoring specifications, but none of them
match the state (the first example where no state is specified
succeeds):

PORT LOCAL=%[:](61617) REMOTE=%10.160.8.132   MIN=3 MAX=3 COLOR=yellow
TEXT=ActiveMQ-DHCP
PORT LOCAL=%[:](61617) REMOTE=%10.160.8.133   STATE=ESTABLISHED MIN=3
MAX=3 COLOR=yellow TEXT=ActiveMQ-nsp.er
PORT LOCAL=%[:](61617) REMOTE=%136.177.16.3   STATE=ESTAB MIN=3 MAX=3
COLOR=yellow TEXT=ActiveMQ-ns.cr
PORT LOCAL=%[:](61617) REMOTE=%137.227.240.32 STATE=%ESTAB MIN=3 MAX=3
COLOR=yellow TEXT=ActiveMQ-ns.er
PORT LOCAL=%[:](61617) REMOTE=%130.118.4.2    STATE=%ESTAB* MIN=3
MAX=3 COLOR=yellow TEXT=ActiveMQ-ns.wr

Note: On this server netstat does not exist and ss is being used,.


Observation: Discovering the syntax for REMOTE was trial and error.
Specifying the IP address alone did not work, and I found no examples
for the type of filtering above.

-- 
                                         -- David Boldt
                                            <user-945c7be1b8e4@xymon.invalid>


   "Discovery consists of seeing what everybody has seen and thinking
what nobody has thought."
    --Albert Szent-Gyorgyi (1893 - 1986)

David

On Thu, Mar 10, 2016 at 1:15 AM Boldt, David <user-945c7be1b8e4@xymon.invalid> wrote:

I couple of things to note, but first some explanation of how things work.
The STATE and PORT rules are matched against the text in the [ports]
section of the client data, which is generally the output of "netstat -na"
in one form or another, but including only TCP sockets.  The output of this
command, and hence the required processing varies between operating
systems, but in most cases the local IP and port is in column 3 (numbered
from 0), the remote is in column 4 and the state is in column 5.  (The
exceptions to this are Solaris, using columns 0, 1 and 6 respectively; and
Windows which uses columns 1, 2 and 3.)

The output of "ss" is most definitely different to any "netstat" that I've
seen.  In particular, the socket state is on column 0 - before either of
the local or report addresses/ports.  It might be helpful to know what OS
you're using, that doesn't have a netstat command.

None of the syntax variations I have tried have been successful.

It's just a coincidence that the column numbers for local and remote are
the same for netstat and ss.  Had this not been the case, you'd have had
more trouble getting matches even with STATE unspecified.

The normal "netstat" output uses the string "ESTABLISHED".  But "ss" uses
"ESTAB".  If [ports] shows ESTAB then that's what you need to match.

PORT LOCAL=%[:](61617) REMOTE=%136.177.16.3   STATE=ESTAB MIN=3 MAX=3
COLOR=yellow TEXT=ActiveMQ-ns.cr

Would work, if ESTAB was in the correct column.

PORT LOCAL=%[:](61617) REMOTE=%137.227.240.32 STATE=%ESTAB MIN=3 MAX=3

I think this will match only when the state starts and ends with "ESTAB"
and so it'd probably work, provided the state was in the first column.

COLOR=yellow TEXT=ActiveMQ-ns.er
PORT LOCAL=%[:](61617) REMOTE=%130.118.4.2    STATE=%ESTAB* MIN=3
MAX=3 COLOR=yellow TEXT=ActiveMQ-ns.wr

The "%" marks the string that follows as a regular expression.  The
asterisk in a regular expression means to match zero or more of the
previous character.  So this would match "ESTAB" as well as "ESTA" and
"ESTABB" and "ESTABBBBBB" but it wouldn't match "ESTABLISHED".

Note: On this server netstat does not exist and ss is being used,.

Yep, this is your problem.  I'm assuming that you edited the
xymonclient-<os>.sh script and replaced "netstat -na" with "ss"?

Observation: Discovering the syntax for REMOTE was trial and error.

The analysis.cfg file has several examples for the use of LOCAL.  The
REMOTE usage is the same.

So, what to do?  The simplest thing would be to adjust the output of "ss"
so that the columns match up.  Simply replicating the state column would
suffice.  Something like so:

ss -nt | awk '{ print $0 $1 }'

Cheers
Jeremy

On Mon, Mar 14, 2016 at 7:02 AM, Jeremy Laidman

<user-71895fb2e44c@xymon.invalid> wrote:

Sapphire OS, a Linux derivative of unknown provenance, customized for
use in IPAM/DHCP/DNS appliances sold by BT Diamond.

I should have taken the documentation at its word, but was led astray
by Internet examples which didn't make a lot of sense to me as regex
expressions. Simplifying, now:.

PORT LOCAL=%:61617 REMOTE=%10.160.8.132   STATE=ESTAB MIN=3 MAX=3
COLOR=yellow TEXT=ActiveMQ-DHCP
PORT LOCAL=%:61617 REMOTE=%10.160.8.133   STATE=ESTAB MIN=3 MAX=3

COLOR=yellow TEXT=ActiveMQ-nsp.er
PORT LOCAL=%:61617 REMOTE=%136.177.16.3   STATE=ESTAB MIN=3 MAX=3
COLOR=yellow TEXT=ActiveMQ-ns.cr
PORT LOCAL=%:61617 REMOTE=%137.227.240.32 STATE=ESTAB MIN=3 MAX=3
COLOR=yellow TEXT=ActiveMQ-ns.er
PORT LOCAL=%:61617 REMOTE=%130.118.4.2    STATE=ESTAB MIN=3 MAX=3
COLOR=yellow TEXT=ActiveMQ-ns.wr

and working

Yep, this is your problem.  I'm assuming that you edited the xymonclient-<os>.sh script and replaced "netstat -na" with "ss"?

yes

ah, but none of the LOCAL examples specifies an IP address, only port
distinctions.

Thanks for your thorough descriptions of the underlying process,and
the example conversion.

-- 
                                         -- David Boldt
                                            <user-945c7be1b8e4@xymon.invalid>


   "... you cannot reason a person out of a position that the person
has not been reasoned into."
    --Washington Post editorial comment on political positions

On Wed, Mar 16, 2016 at 10:00 AM Boldt, David <user-945c7be1b8e4@xymon.invalid> wrote:

I'm somewhat familiar with the BT Diamond product, but not running on an
appliance.  It so annoys me when vendors do that.  I cannot fathom why they
think it's useful to anyone when they nobble an OS - I guess they call it
"hardening".  I suppose you're lucky that you can even get an agent
installed and running on there.

I should have taken the documentation at its word, but was led astray

Yep, that should work (providing the STATE matches).  Technically, a
regular expression should be anchored at the beginning (with "^") and often
at the end (with "$"), and special characters like "*" and "." should be
escaped (with "\"), so as to ensure you don't get an accidental match.  For
example, your above line would match the IP address "210.160.8.132", and
I'm confident you wouldn't want it to.  If you need to match a whole string
exactly, don't use the percent, something like "REMOTE=10.160.8.132".
Otherwise, it's safest to anchor and escape; so to match a /24 subnet, you
could do "REMOTE=%^10\.168\.8\."

A problem I once encountered is with a PROC test where I was monitoring the
presence of the "named" process for BIND DNS, so I was matching "named" and
requiring exactly one instance.  An operator would occasionally run
something like "tail /var/log/named" and this process string would also
match, alerting that there were two named processes instead of one.  Using
"%named" was just as bad.  I then switched to using a regular expression of
"%^/usr/sbin/named", which improved things somewhat, but commands like
"/usr/sbin/named-checkconf" also matched and caused alerts.  After a few
iterations, my PROC line is now:

        PROC "%^/usr/sbin/named(\s*$|\s)" 1 1 "TEXT=/usr/sbin/named"

This matches only when the command starts with /usr/sbin/named and either
ends there (optionally with trailing spaces) or has a space (presumably
followed by command-line arguments).

Anyhow, the moral of the story is, regular expressions should use adequate
anchoring and escaping.

and working

I'm puzzled why this works if the STATE isn't in column 5, as is the case
for the "ss" command.  Unless you also modified the "ss" command to create
column 5, as I suggested.

Except for this one:

#             Example: Check that ONLY ports 22, 80 and 443 are open for
incoming connections:
#                 PORT STATE=LISTEN LOCAL=%0.0.0.0[.:].*
EXLOCAL=%[.:](22|80|443)$ MAX=0 "TEXT=Bad listeners"

Although this perhaps isn't the best example for what you were trying to
do.  (Not to mention that it didn't escape the dots, nor anchor.  However
in this example the matching happens to be unambiguous for the possible
strings - it would be unusual to see any other IP address matching the
regexp in the example.)

Nevertheless, it looks like you have a solution.  It's always nice to see
Xymon working with appliances - despite the best efforts of the vendors.

Cheers
Jeremy

On Tue, Mar 15, 2016 at 7:58 PM, Jeremy Laidman

<user-71895fb2e44c@xymon.invalid> wrote:

It's a testament to xymon that I was able to extract a tar file into
an arbitrary directory and have it run.

Very close to your recommendation, modified slightly to pick up LISTEN
states, xymonclient-linux.sh using this for [ports]:

ss -ant | awk '{ print $0 $1 }'

indeed.

Thanks once again for all of your help.

-- 
                                         -- David Boldt
                                            <user-945c7be1b8e4@xymon.invalid>


   "... you cannot reason a person out of a position that the person
has not been reasoned into."
    --Washington Post editorial comment on political positions