Xymon Mailing List Archive search

Disk space test ignored

11 messages in this thread

list Scot Kreienkamp · Tue, 5 Apr 2016 19:14:40 +0000 · 
Hi everyone,

I recently upgraded from 4.3.21 to the Terabithia RPM's for 4.3.27.  I copied this config file verbatim (except for server name) from my old server to my new server:


HOST=innocent.in.hq
        PORT "LOCAL=%([.:]80)$" MIN=1  TRACK=WWW "TEXT=80-WWW"
        PORT "LOCAL=%([.:]55000)$" MIN=0 color=red TRACK=RPC_Clients "TEXT=55000-RPC Client Access Sessions"
        SVC MSExchangeADTopology status=started
        SVC MSExchangeIS status=started
        SVC MSExchangeMailboxAssistants status=started
        SVC MSExchangeRPC status=started
        SVC MSExchangeSA status=started
        DISK J 15728640U 10485760U
        DISK L 15728640U 10485760U
        DISK M 15728640U 10485760U
        DISK N 15728640U 10485760U
        DISK S 15728640U 10485760U
        DISK T 15728640U 10485760U
        DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE


However, the disk tests are now alerting on 95% full disk instead of on 10 gigs remaining space:
M (95% used) has reached the PANIC level (95%)

To further troubleshoot I commented out all the disk lines and added this entry:
DISK * 15728640U 10485760U

That didn't make any difference, the disk is still alerting for 95% full.  It is parsing and applying the configuration because the port and SVC information is showing up in the web, so I can't understand why the disk test isn't working.


This is a key feature for us as we have many servers with multi-terabyte drives, so alerting at percentages doesn't work nearly as well.  The client has not been touched, only the server.

Anyone have any ideas?

Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162  | * XXX-XXX-XXXX | | * 7349151444 | *  user-9678697f1438@xymon.invalid<mailto:%7BE-mail%7D>
www<http://www.la-z-boy.com/>.la-z-boy.com<http://www.la-z-boy.com/>; | facebook.<https://www.facebook.com/lazboy>com<https://www.facebook.com/lazboy>/<https://www.facebook.com/lazboy>lazboy<http://facebook.com/lazboy>; | twitter.com/lazboy<https://twitter.com/lazboy>; | youtube.com/<https://www.youtube.com/user/lazboy>lazboy<https://www.youtube.com/user/lazboy>;

[cid:lzbVertical_hres.jpg]


This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
Attachments (1)
list Japheth Cleaver · Tue, 5 Apr 2016 19:45:51 -0700 ·
▸ quoted from Scot Kreienkamp

On Tue, April 5, 2016 12:14 pm, Scot Kreienkamp wrote:
Hi everyone,

I recently upgraded from 4.3.21 to the Terabithia RPM's for 4.3.27.  I
copied this config file verbatim (except for server name) from my old
server to my new server:


HOST=innocent.in.hq
        PORT "LOCAL=%([.:]80)$" MIN=1  TRACK=WWW "TEXT=80-WWW"
        PORT "LOCAL=%([.:]55000)$" MIN=0 color=red TRACK=RPC_Clients
"TEXT=55000-RPC Client Access Sessions"
        SVC MSExchangeADTopology status=started
        SVC MSExchangeIS status=started
        SVC MSExchangeMailboxAssistants status=started
        SVC MSExchangeRPC status=started
        SVC MSExchangeSA status=started
        DISK J 15728640U 10485760U
        DISK L 15728640U 10485760U
        DISK M 15728640U 10485760U
        DISK N 15728640U 10485760U
        DISK S 15728640U 10485760U
        DISK T 15728640U 10485760U
        DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE


However, the disk tests are now alerting on 95% full disk instead of on 10
gigs remaining space:
M (95% used) has reached the PANIC level (95%)

To further troubleshoot I commented out all the disk lines and added this
entry:
DISK * 15728640U 10485760U

That didn't make any difference, the disk is still alerting for 95% full.
It is parsing and applying the configuration because the port and SVC
information is showing up in the web, so I can't understand why the disk
test isn't working.


This is a key feature for us as we have many servers with multi-terabyte
drives, so alerting at percentages doesn't work nearly as well.  The
client has not been touched, only the server.

Anyone have any ideas?

Some possibilities:

Did you wait for xymond_client to reload analysis.cfg after the change?
Alternatively, did you remove/reorder the DISK sections or comment them
out in place? It's possible the catchall entry wasn't applied and what was
actually alerting was the default DISK section.

Can you run xymond_client --dump-config after the change to see what's
happening?


-jc
list Scot Kreienkamp · Wed, 6 Apr 2016 03:08:52 +0000 · 

Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: XXX-XXX-XXXX |  |  Mobile: XXXXXXXXXX | Email: user-9678697f1438@xymon.invalid
▸ quoted from Japheth Cleaver
-----Original Message-----
From: J.C. Cleaver [mailto:user-87556346d4af@xymon.invalid]
Sent: Tuesday, April 05, 2016 10:46 PM
To: Scot Kreienkamp
Cc: xymon at xymon.com
Subject: Re: [Xymon] Disk space test ignored


On Tue, April 5, 2016 12:14 pm, Scot Kreienkamp wrote:
Hi everyone,

I recently upgraded from 4.3.21 to the Terabithia RPM's for 4.3.27.  I
copied this config file verbatim (except for server name) from my old
server to my new server:


HOST=innocent.in.hq
        PORT "LOCAL=%([.:]80)$" MIN=1  TRACK=WWW "TEXT=80-WWW"
        PORT "LOCAL=%([.:]55000)$" MIN=0 color=red TRACK=RPC_Clients
"TEXT=55000-RPC Client Access Sessions"
        SVC MSExchangeADTopology status=started
        SVC MSExchangeIS status=started
        SVC MSExchangeMailboxAssistants status=started
        SVC MSExchangeRPC status=started
        SVC MSExchangeSA status=started
        DISK J 15728640U 10485760U
        DISK L 15728640U 10485760U
        DISK M 15728640U 10485760U
        DISK N 15728640U 10485760U
        DISK S 15728640U 10485760U
        DISK T 15728640U 10485760U
        DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE


However, the disk tests are now alerting on 95% full disk instead of on 10
gigs remaining space:
M (95% used) has reached the PANIC level (95%)

To further troubleshoot I commented out all the disk lines and added this
entry:
DISK * 15728640U 10485760U

That didn't make any difference, the disk is still alerting for 95% full.
It is parsing and applying the configuration because the port and SVC
information is showing up in the web, so I can't understand why the disk
test isn't working.


This is a key feature for us as we have many servers with multi-terabyte
drives, so alerting at percentages doesn't work nearly as well.  The
client has not been touched, only the server.

Anyone have any ideas?

Some possibilities:

Did you wait for xymond_client to reload analysis.cfg after the change?
Alternatively, did you remove/reorder the DISK sections or comment them
out in place? It's possible the catchall entry wasn't applied and what was
actually alerting was the default DISK section.

Can you run xymond_client --dump-config after the change to see what's
happening?


-jc

I commented them out in place and put the catchall entry directly above it.


I ran the dump and grep'd for the server name.  Here are the matching lines:

PORT local=%([.:]80)$ min=1 color=red TRACK=WWW HOST=innocent2.in.hq,innocent.in.hq TEXT=80-WWW (line: 468)
PORT local=%([.:]55000)$ min=0 color=red TRACK=RPC_Clients HOST=innocent2.in.hq,innocent.in.hq TEXT=55000-RPC Client Access Sessions (line: 469)
SVC MSExchangeADTopology status=started color=red HOST=innocent2.in.hq,innocent.in.hq (line: 470)
SVC MSExchangeIS status=started color=red HOST=innocent2.in.hq,innocent.in.hq (line: 471)
SVC MSExchangeMailboxAssistants status=started color=red HOST=innocent2.in.hq,innocent.in.hq (line: 472)
SVC MSExchangeRPC status=started color=red HOST=innocent2.in.hq,innocent.in.hq (line: 473)
SVC MSExchangeSA status=started color=red HOST=innocent2.in.hq,innocent.in.hq (line: 474)
DISK * 15728640U 10485760U 0 -1 red HOST=innocent2.in.hq,innocent.in.hq (line: 475)
DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE HOST=innocent2.in.hq,innocent.in.hq (line: 482)


It appears to be understanding the config correctly, but it's still alerting on the percentage:
M (96% used) has reached the PANIC level (95%)
Filesystem    1K-blocks         Used        Avail  Capacity  Mounted    Label      Summary(Total\Avail GB)
M            2115137532   2032580452     82557080        96% /FIXED/M:\ Ret_Mail   2017.15\78.73


Thanks JC!
▸ quoted from Scot Kreienkamp


This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
list Japheth Cleaver · Tue, 5 Apr 2016 21:07:06 -0700 ·
▸ quoted from Scot Kreienkamp

On Tue, April 5, 2016 8:08 pm, Scot Kreienkamp wrote:

I ran the dump and grep'd for the server name.  Here are the matching
lines:

PORT local=%([.:]80)$ min=1 color=red TRACK=WWW
HOST=innocent2.in.hq,innocent.in.hq TEXT=80-WWW (line: 468)
PORT local=%([.:]55000)$ min=0 color=red TRACK=RPC_Clients
HOST=innocent2.in.hq,innocent.in.hq TEXT=55000-RPC Client Access Sessions
(line: 469)
SVC MSExchangeADTopology status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 470)
SVC MSExchangeIS status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 471)
SVC MSExchangeMailboxAssistants status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 472)
SVC MSExchangeRPC status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 473)
SVC MSExchangeSA status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 474)
DISK * 15728640U 10485760U 0 -1 red HOST=innocent2.in.hq,innocent.in.hq
(line: 475)
DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE HOST=innocent2.in.hq,innocent.in.hq
(line: 482)


It appears to be understanding the config correctly, but it's still
alerting on the percentage:
M (96% used) has reached the PANIC level (95%)
Filesystem    1K-blocks         Used        Avail  Capacity  Mounted
Label      Summary(Total\Avail GB)
M            2115137532   2032580452     82557080        96% /FIXED/M:\
Ret_Mail   2017.15\78.73


It still might be part of the DEFAULT entry, though. Is the "host" entry
listed literally as:

HOST=innocent2.in.hq,innocent.in.hq

...in the config? IIRC, that needs to be a regex. Comma-separated is only
used when specifying colors (which aren't evaluated textually).


HTH,
-jc
list Scot Kreienkamp · Wed, 6 Apr 2016 10:55:01 +0000 ·
▸ signature

Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: XXX-XXX-XXXX |  |  Mobile: XXXXXXXXXX | Email: user-9678697f1438@xymon.invalid
-----Original Message-----
▸ quoted from Japheth Cleaver
From: J.C. Cleaver [mailto:user-87556346d4af@xymon.invalid]
Sent: Wednesday, April 06, 2016 12:07 AM
To: Scot Kreienkamp <user-9678697f1438@xymon.invalid>
Cc: Xymon Mailing List <xymon at xymon.com>
Subject: RE: [Xymon] Disk space test ignored


On Tue, April 5, 2016 8:08 pm, Scot Kreienkamp wrote:

I ran the dump and grep'd for the server name.  Here are the matching
lines:

PORT local=%([.:]80)$ min=1 color=red TRACK=WWW
HOST=innocent2.in.hq,innocent.in.hq TEXT=80-WWW (line: 468)
PORT local=%([.:]55000)$ min=0 color=red TRACK=RPC_Clients
HOST=innocent2.in.hq,innocent.in.hq TEXT=55000-RPC Client Access
Sessions
(line: 469)
SVC MSExchangeADTopology status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 470)
SVC MSExchangeIS status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 471)
SVC MSExchangeMailboxAssistants status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 472)
SVC MSExchangeRPC status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 473)
SVC MSExchangeSA status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 474)
DISK * 15728640U 10485760U 0 -1 red HOST=innocent2.in.hq,innocent.in.hq
(line: 475)
DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE
HOST=innocent2.in.hq,innocent.in.hq
(line: 482)


It appears to be understanding the config correctly, but it's still
alerting on the percentage:
M (96% used) has reached the PANIC level (95%)
Filesystem    1K-blocks         Used        Avail  Capacity  Mounted
Label      Summary(Total\Avail GB)
M            2115137532   2032580452     82557080        96% /FIXED/M:\
Ret_Mail   2017.15\78.73


It still might be part of the DEFAULT entry, though. Is the "host" entry
listed literally as:

HOST=innocent2.in.hq,innocent.in.hq

...in the config? IIRC, that needs to be a regex. Comma-separated is only
used when specifying colors (which aren't evaluated textually).


HTH,
-jc

If it's not parsing comma separated HOST= lines then the man pages are wrong.  I'll separate it out and see if it makes any difference.

Here's the section from the analysis.cfg man page:
HOST=targetstring Rule matching a host by the hostname. "targetstring" is either a comma-separated list of hostnames (from the hosts.cfg file), "*" to indicate "all hosts", or a Perl-compatible regular expression. E.g. "HOST=dns.foo.com,www.foo.com" identifies two specific hosts; "HOST=%www.*.foo.com EXHOST=www-test.foo.com" matches all hosts with a name beginning with "www", except the "www-test" host.
▸ quoted from Scot Kreienkamp


This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
list Scot Kreienkamp · Wed, 6 Apr 2016 17:26:34 +0000 ·
▸ signature
Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: XXX-XXX-XXXX |  |  Mobile: XXXXXXXXXX | Email: user-9678697f1438@xymon.invalid
-----Original Message-----
▸ quoted from Scot Kreienkamp
From: Scot Kreienkamp
Sent: Wednesday, April 06, 2016 6:55 AM
To: J.C. Cleaver
Cc: Xymon Mailing List
Subject: RE: [Xymon] Disk space test ignored


-----Original Message-----
From: J.C. Cleaver [mailto:user-87556346d4af@xymon.invalid]
Sent: Wednesday, April 06, 2016 12:07 AM
To: Scot Kreienkamp <user-9678697f1438@xymon.invalid>
Cc: Xymon Mailing List <xymon at xymon.com>
Subject: RE: [Xymon] Disk space test ignored


On Tue, April 5, 2016 8:08 pm, Scot Kreienkamp wrote:

I ran the dump and grep'd for the server name.  Here are the matching
lines:

PORT local=%([.:]80)$ min=1 color=red TRACK=WWW
HOST=innocent2.in.hq,innocent.in.hq TEXT=80-WWW (line: 468)
PORT local=%([.:]55000)$ min=0 color=red TRACK=RPC_Clients
HOST=innocent2.in.hq,innocent.in.hq TEXT=55000-RPC Client Access
Sessions
(line: 469)
SVC MSExchangeADTopology status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 470)
SVC MSExchangeIS status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 471)
SVC MSExchangeMailboxAssistants status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 472)
SVC MSExchangeRPC status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 473)
SVC MSExchangeSA status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 474)
DISK * 15728640U 10485760U 0 -1 red
HOST=innocent2.in.hq,innocent.in.hq
(line: 475)
DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE
HOST=innocent2.in.hq,innocent.in.hq
(line: 482)


It appears to be understanding the config correctly, but it's still
alerting on the percentage:
M (96% used) has reached the PANIC level (95%)
Filesystem    1K-blocks         Used        Avail  Capacity  Mounted
Label      Summary(Total\Avail GB)
M            2115137532   2032580452     82557080        96% /FIXED/M:\
Ret_Mail   2017.15\78.73


It still might be part of the DEFAULT entry, though. Is the "host" entry
listed literally as:

HOST=innocent2.in.hq,innocent.in.hq

...in the config? IIRC, that needs to be a regex. Comma-separated is only
used when specifying colors (which aren't evaluated textually).


HTH,
-jc

If it's not parsing comma separated HOST= lines then the man pages are
wrong.  I'll separate it out and see if it makes any difference.

Here's the section from the analysis.cfg man page:
HOST=targetstring Rule matching a host by the hostname. "targetstring" is
either a comma-separated list of hostnames (from the hosts.cfg file), "*" to
indicate "all hosts", or a Perl-compatible regular expression. E.g.
"HOST=dns.foo.com,www.foo.com" identifies two specific hosts;
"HOST=%www.*.foo.com EXHOST=www-test.foo.com" matches all hosts
with a name beginning with "www", except the "www-test" host.

It's separated out with only that hostname on a HOST= line by itself.  No change in behavior.  I waited 4 hours between when I made the change and when I checked the results as I was out all morning, so it has definitely taken affect by now.

PORT local=%([.:]80)$ min=1 color=red TRACK=WWW HOST=innocent.in.hq TEXT=80-WWW (line: 468)
PORT local=%([.:]55000)$ min=0 color=red TRACK=RPC_Clients HOST=innocent.in.hq TEXT=55000-RPC Client Access Sessions (line: 469)
SVC MSExchangeADTopology status=started color=red HOST=innocent.in.hq (line: 470)
SVC MSExchangeIS status=started color=red HOST=innocent.in.hq (line: 471)
SVC MSExchangeMailboxAssistants status=started color=red HOST=innocent.in.hq (line: 472)
SVC MSExchangeRPC status=started color=red HOST=innocent.in.hq (line: 473)
SVC MSExchangeSA status=started color=red HOST=innocent.in.hq (line: 474)
DISK * 15728640U 10485760U 0 -1 red HOST=innocent.in.hq (line: 475)
DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE HOST=innocent.in.hq (line: 482)

Any ideas?
▸ quoted from Scot Kreienkamp


This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
list David Boyer · Wed, 6 Apr 2016 14:30:51 -0400 · 
JC,
    Would we be able to see the GROUP= when running the xymond_client
--dump-config?
I need to send out emails to different groups and I was trying to use the
GROUP= and it's not
working.  I posted this about a week ago and got no hits.

Dave

On Tue, Apr 5, 2016 at 10:45 PM, J.C. Cleaver <user-87556346d4af@xymon.invalid>
▸ quoted from Scot Kreienkamp
wrote:

On Tue, April 5, 2016 12:14 pm, Scot Kreienkamp wrote:
Hi everyone,

I recently upgraded from 4.3.21 to the Terabithia RPM's for 4.3.27.  I
copied this config file verbatim (except for server name) from my old
server to my new server:


HOST=innocent.in.hq
        PORT "LOCAL=%([.:]80)$" MIN=1  TRACK=WWW "TEXT=80-WWW"
        PORT "LOCAL=%([.:]55000)$" MIN=0 color=red TRACK=RPC_Clients
"TEXT=55000-RPC Client Access Sessions"
        SVC MSExchangeADTopology status=started
        SVC MSExchangeIS status=started
        SVC MSExchangeMailboxAssistants status=started
        SVC MSExchangeRPC status=started
        SVC MSExchangeSA status=started
        DISK J 15728640U 10485760U
        DISK L 15728640U 10485760U
        DISK M 15728640U 10485760U
        DISK N 15728640U 10485760U
        DISK S 15728640U 10485760U
        DISK T 15728640U 10485760U
        DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE


However, the disk tests are now alerting on 95% full disk instead of on
10
gigs remaining space:
M (95% used) has reached the PANIC level (95%)

To further troubleshoot I commented out all the disk lines and added this
entry:
DISK * 15728640U 10485760U

That didn't make any difference, the disk is still alerting for 95% full.
It is parsing and applying the configuration because the port and SVC
information is showing up in the web, so I can't understand why the disk
test isn't working.


This is a key feature for us as we have many servers with multi-terabyte
drives, so alerting at percentages doesn't work nearly as well.  The
client has not been touched, only the server.

Anyone have any ideas?

Some possibilities:

Did you wait for xymond_client to reload analysis.cfg after the change?
Alternatively, did you remove/reorder the DISK sections or comment them
out in place? It's possible the catchall entry wasn't applied and what was
actually alerting was the default DISK section.

Can you run xymond_client --dump-config after the change to see what's
happening?


-jc
list Scot Kreienkamp · Wed, 6 Apr 2016 18:38:15 +0000 · 
Please do not thread-hijack.
▸ signature


Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 | Office: XXX-XXX-XXXX | | Mobile: XXXXXXXXXX | Email: user-9678697f1438@xymon.invalid
▸ quoted from David Boyer
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of David Boyer
Sent: Wednesday, April 06, 2016 2:31 PM
To: J.C. Cleaver <user-87556346d4af@xymon.invalid>
Cc: Scot Kreienkamp <user-9678697f1438@xymon.invalid>; xymon at xymon.com
Subject: Re: [Xymon] Disk space test ignored

JC,
    Would we be able to see the GROUP= when running the xymond_client --dump-config?
I need to send out emails to different groups and I was trying to use the GROUP= and it's not
working.  I posted this about a week ago and got no hits.
Dave

On Tue, Apr 5, 2016 at 10:45 PM, J.C. Cleaver <user-87556346d4af@xymon.invalid<mailto:user-87556346d4af@xymon.invalid>> wrote:


On Tue, April 5, 2016 12:14 pm, Scot Kreienkamp wrote:
Hi everyone,

I recently upgraded from 4.3.21 to the Terabithia RPM's for 4.3.27.  I
copied this config file verbatim (except for server name) from my old
server to my new server:


HOST=innocent.in.hq
        PORT "LOCAL=%([.:]80)$" MIN=1  TRACK=WWW "TEXT=80-WWW"
        PORT "LOCAL=%([.:]55000)$" MIN=0 color=red TRACK=RPC_Clients
"TEXT=55000-RPC Client Access Sessions"
        SVC MSExchangeADTopology status=started
        SVC MSExchangeIS status=started
        SVC MSExchangeMailboxAssistants status=started
        SVC MSExchangeRPC status=started
        SVC MSExchangeSA status=started
        DISK J 15728640U 10485760U
        DISK L 15728640U 10485760U
        DISK M 15728640U 10485760U
        DISK N 15728640U 10485760U
        DISK S 15728640U 10485760U
        DISK T 15728640U 10485760U
        DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE


However, the disk tests are now alerting on 95% full disk instead of on 10
gigs remaining space:
M (95% used) has reached the PANIC level (95%)

To further troubleshoot I commented out all the disk lines and added this
entry:
DISK * 15728640U 10485760U

That didn't make any difference, the disk is still alerting for 95% full.
It is parsing and applying the configuration because the port and SVC
information is showing up in the web, so I can't understand why the disk
test isn't working.


This is a key feature for us as we have many servers with multi-terabyte
drives, so alerting at percentages doesn't work nearly as well.  The
client has not been touched, only the server.

Anyone have any ideas?

Some possibilities:

Did you wait for xymond_client to reload analysis.cfg after the change?
Alternatively, did you remove/reorder the DISK sections or comment them
out in place? It's possible the catchall entry wasn't applied and what was
actually alerting was the default DISK section.

Can you run xymond_client --dump-config after the change to see what's
happening?


-jc


This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
list Japheth Cleaver · Wed, 6 Apr 2016 15:13:41 -0700 ·
▸ quoted from Scot Kreienkamp

On Wed, April 6, 2016 10:26 am, Scot Kreienkamp wrote:
On Tue, April 5, 2016 8:08 pm, Scot Kreienkamp wrote:

I ran the dump and grep'd for the server name.  Here are the
matching
lines:

PORT local=%([.:]80)$ min=1 color=red TRACK=WWW
HOST=innocent2.in.hq,innocent.in.hq TEXT=80-WWW (line: 468)
PORT local=%([.:]55000)$ min=0 color=red TRACK=RPC_Clients
HOST=innocent2.in.hq,innocent.in.hq TEXT=55000-RPC Client Access
Sessions
(line: 469)
SVC MSExchangeADTopology status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 470)
SVC MSExchangeIS status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 471)
SVC MSExchangeMailboxAssistants status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 472)
SVC MSExchangeRPC status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 473)
SVC MSExchangeSA status=started color=red
HOST=innocent2.in.hq,innocent.in.hq (line: 474)
DISK * 15728640U 10485760U 0 -1 red
HOST=innocent2.in.hq,innocent.in.hq
(line: 475)
DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE
HOST=innocent2.in.hq,innocent.in.hq
(line: 482)


It appears to be understanding the config correctly, but it's still
alerting on the percentage:
M (96% used) has reached the PANIC level (95%)
Filesystem    1K-blocks         Used        Avail  Capacity
Mounted
Label      Summary(Total\Avail GB)
M            2115137532   2032580452     82557080        96%
/FIXED/M:\
Ret_Mail   2017.15\78.73


It still might be part of the DEFAULT entry, though. Is the "host"
entry
listed literally as:

HOST=innocent2.in.hq,innocent.in.hq

...in the config? IIRC, that needs to be a regex. Comma-separated is
only
used when specifying colors (which aren't evaluated textually).


HTH,
-jc

If it's not parsing comma separated HOST= lines then the man pages are
wrong.  I'll separate it out and see if it makes any difference.
Yikes. That might be a documentation bug. Pretty sure COLOR is all that
will be eval'd like that.
▸ quoted from Scot Kreienkamp


Here's the section from the analysis.cfg man page:
HOST=targetstring Rule matching a host by the hostname. "targetstring"
is
either a comma-separated list of hostnames (from the hosts.cfg file),
"*" to
indicate "all hosts", or a Perl-compatible regular expression. E.g.
"HOST=dns.foo.com,www.foo.com" identifies two specific hosts;
"HOST=%www.*.foo.com EXHOST=www-test.foo.com" matches all hosts
with a name beginning with "www", except the "www-test" host.

It's separated out with only that hostname on a HOST= line by itself.  No
change in behavior.  I waited 4 hours between when I made the change and
when I checked the results as I was out all morning, so it has definitely
taken affect by now.

PORT local=%([.:]80)$ min=1 color=red TRACK=WWW HOST=innocent.in.hq
TEXT=80-WWW (line: 468)
PORT local=%([.:]55000)$ min=0 color=red TRACK=RPC_Clients
HOST=innocent.in.hq TEXT=55000-RPC Client Access Sessions (line: 469)
SVC MSExchangeADTopology status=started color=red HOST=innocent.in.hq
(line: 470)
SVC MSExchangeIS status=started color=red HOST=innocent.in.hq (line: 471)
SVC MSExchangeMailboxAssistants status=started color=red
HOST=innocent.in.hq (line: 472)
SVC MSExchangeRPC status=started color=red HOST=innocent.in.hq (line:
473)
SVC MSExchangeSA status=started color=red HOST=innocent.in.hq (line: 474)
DISK * 15728640U 10485760U 0 -1 red HOST=innocent.in.hq (line: 475)
DISK %^(1|2|3|4|5|6|7|8|9|0).* IGNORE HOST=innocent.in.hq (line: 482)

Any ideas?


Can you grep for DISK on this instead of the host? The config shown
*looks* correct to me, which makes me think that it's a different rule
being applied still.

When you run with --debug enabled on xymond_client, is there any output on
the 'df' evaluation for this host?

-jc
list Scot Kreienkamp · Fri, 8 Apr 2016 14:23:54 +0000 ·
▸ quoted from Japheth Cleaver
Can you grep for DISK on this instead of the host? The config shown
*looks* correct to me, which makes me think that it's a different rule
being applied still.

When you run with --debug enabled on xymond_client, is there any output
on
the 'df' evaluation for this host?

-jc

JC, you're a miracle worker.  So what I finally found out was that a class definition was placed in a hosts.d subfile with the rest of the hosts configs in another hosts.d subfile, and due to alphabetical order it was being loaded first.  When I renamed the files so the class file was being loaded last the test started using the proper config.

Thanks!
▸ quoted from Scot Kreienkamp

Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: XXX-XXX-XXXX |  |  Mobile: XXXXXXXXXX | Email: user-9678697f1438@xymon.invalid

This message is intended only for the individual or entity to which it is addressed.  It may contain privileged, confidential information which is exempt from disclosure under applicable laws.  If you are not the intended recipient, you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information.  If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
list Japheth Cleaver · Fri, 8 Apr 2016 12:53:06 -0700 ·
▸ quoted from Scot Kreienkamp

On Fri, April 8, 2016 7:23 am, Scot Kreienkamp wrote:
Can you grep for DISK on this instead of the host? The config shown
*looks* correct to me, which makes me think that it's a different rule
being applied still.

When you run with --debug enabled on xymond_client, is there any output
on
the 'df' evaluation for this host?

-jc

JC, you're a miracle worker.  So what I finally found out was that a class
definition was placed in a hosts.d subfile with the rest of the hosts
configs in another hosts.d subfile, and due to alphabetical order it was
being loaded first.  When I renamed the files so the class file was being
loaded last the test started using the proper config.

Thanks!

No problem :)

This is definitely potentially confusing though. I wonder if it might be
possible a generation time to warn on overlapping rules like this.

-jc