Xymon Mailing List Archive search

xymonnet alarm

8 messages in this thread

list Foster Patch · Thu, 1 Sep 2016 19:34:15 +0000 · 
Hello,

A change to how our DNS is configured has caused the DNS test on xymonnet to reach around 50 seconds. Combined with the other tests, the total time exceeds the time limit (60). Because of this, xymonnet has been alarming for a very long time. Is there any way I can adjust the time limit to account for this? (Bring it up to 80 instead of 60 for example) Here is our xymonnet section of tasks.cfg:

[xymonnet]
               ENVFILE /usr/lib/xymon/server/etc/xymonserver.cfg
               NEEDS xymond
               CMD xymonnet --report --ping --checkresponse --dns-timeout=5
               LOGFILE $XYMONSERVERLOGS/xymonnet.log
               INTERVAL 1m

Any advice would be appreciated.

Thanks,

Foster Patch
Server Technician
list Japheth Cleaver · Thu, 1 Sep 2016 14:38:36 -0700 ·
▸ quoted from Foster Patch
On 9/1/2016 12:34 PM, Foster Patch wrote:
Hello,

A change to how our DNS is configured has caused the DNS test on xymonnet to reach around 50 seconds. Combined with the other tests, the total time exceeds the time limit (60). Because of this, xymonnet has been alarming for a very long time. Is there any way I can adjust the time limit to account for this? (Bring it up to 80 instead of 60 for example) Here is our xymonnet section of tasks.cfg:

[xymonnet]

               ENVFILE /usr/lib/xymon/server/etc/xymonserver.cfg

               NEEDS xymond

               CMD xymonnet --report --ping --checkresponse --dns-timeout=5

               LOGFILE $XYMONSERVERLOGS/xymonnet.log

               INTERVAL 1m

Any advice would be appreciated.

Thanks,

Foster Patch

Server Technician

Hi Foster,

The warning time passed in by xymonnet can be adjusted with the --timelimit= option given in the CMD above. By default, it's equivalent to the INTERVAL runtime (assuming xymonnet is run with one, which it usually is), since it's there to let you know that the testing interval isn't being met. You can also adjust that upwards to match the average time things are taking.

Unfortunately, DNS lookups-as-tests are handled differently from the other tests, and timeouts or network problems can increase xymonnet's runtime linearly with failures. Assuming the ones you're testing are the only ones you need to test (i.e., remove dead hosts), reducing '--dns-timeout' as low as you're comfortable with is the only other real tweak here. Enabling/disabling the test changes whether xymon considers an outage to be an alertable issue, but doesn't prevent xymonnet from checking (and waiting on) that service in the first place.

If you have a large number of these with different SLAs, or a huge number of DNS servers to test, there are advanced ways of setting up xymonnet runtimes to isolate these effects. Basically, you'd set up a separate instance of xymonnet just for doing DNS tests, while excluding them from your normal install.


HTH,
-jc
list Foster Patch · Thu, 8 Sep 2016 18:51:36 +0000 · 
Hello,

I added what you recommended, but the alarm still exists. Here is the tasks.cfg section:
[xymonnet]
               ENVFILE /usr/lib/xymon/server/etc/xymonserver.cfg
               NEEDS xymond
               CMD xymonnet --report --ping --checkresponse --dns-timeout=5 --timelimit=75
               LOGFILE $XYMONSERVERLOGS/xymonnet.log
               INTERVAL 1m

The xymonnet alarm now says under Error output:
Unknown option –timelimit=75 – try –help

Any suggestions?

Foster
▸ quoted from Japheth Cleaver


From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Japheth Cleaver
Sent: Thursday, September 1, 2016 5:39 PM
To: xymon at xymon.com
Subject: Re: [Xymon] xymonnet alarm

On 9/1/2016 12:34 PM, Foster Patch wrote:
Hello,

A change to how our DNS is configured has caused the DNS test on xymonnet to reach around 50 seconds. Combined with the other tests, the total time exceeds the time limit (60). Because of this, xymonnet has been alarming for a very long time. Is there any way I can adjust the time limit to account for this? (Bring it up to 80 instead of 60 for example) Here is our xymonnet section of tasks.cfg:

[xymonnet]
               ENVFILE /usr/lib/xymon/server/etc/xymonserver.cfg
               NEEDS xymond
               CMD xymonnet --report --ping --checkresponse --dns-timeout=5
               LOGFILE $XYMONSERVERLOGS/xymonnet.log
               INTERVAL 1m

Any advice would be appreciated.

Thanks,

Foster Patch
Server Technician

Hi Foster,

The warning time passed in by xymonnet can be adjusted with the --timelimit= option given in the CMD above. By default, it's equivalent to the INTERVAL runtime (assuming xymonnet is run with one, which it usually is), since it's there to let you know that the testing interval isn't being met. You can also adjust that upwards to match the average time things are taking.

Unfortunately, DNS lookups-as-tests are handled differently from the other tests, and timeouts or network problems can increase xymonnet's runtime linearly with failures. Assuming the ones you're testing are the only ones you need to test (i.e., remove dead hosts), reducing '--dns-timeout' as low as you're comfortable with is the only other real tweak here. Enabling/disabling the test changes whether xymon considers an outage to be an alertable issue, but doesn't prevent xymonnet from checking (and waiting on) that service in the first place.

If you have a large number of these with different SLAs, or a huge number of DNS servers to test, there are advanced ways of setting up xymonnet runtimes to isolate these effects. Basically, you'd set up a separate instance of xymonnet just for doing DNS tests, while excluding them from your normal install.


HTH,
-jc
list Japheth Cleaver · Fri, 9 Sep 2016 20:11:15 -0700 ·
▸ quoted from Foster Patch
On Thu, September 8, 2016 11:51 am, Foster Patch wrote:
Hello,

I added what you recommended, but the alarm still exists. Here is the
tasks.cfg section:
[xymonnet]
               ENVFILE /usr/lib/xymon/server/etc/xymonserver.cfg
               NEEDS xymond
               CMD xymonnet --report --ping --checkresponse
--dns-timeout=5 --timelimit=75
               LOGFILE $XYMONSERVERLOGS/xymonnet.log
               INTERVAL 1m

The xymonnet alarm now says under Error output:

Unknown option –timelimit=75 – try –help

Any suggestions?

Foster

Hi Foster,

Unfortunately, I can confirm that this is a bug in how that option is
processed. As a workaround, you can lie to xymonnet about what interval
it's been told to run (from xymonlaunch) at with:

CMD env TASKSLEEP=75 xymonnet --report --ping --checkresponse
--dns-timeout=5

This should result in what you want.

The following patch should fix it for you, and this will be in the next
version.


HTH,
-jc
list Japheth Cleaver · Fri, 9 Sep 2016 20:29:45 -0700 · 
Patch now included.

-jc
▸ quoted from Japheth Cleaver


On Fri, September 9, 2016 8:11 pm, J.C. Cleaver wrote:
On Thu, September 8, 2016 11:51 am, Foster Patch wrote:
Hello,

I added what you recommended, but the alarm still exists. Here is the
tasks.cfg section:
[xymonnet]
               ENVFILE /usr/lib/xymon/server/etc/xymonserver.cfg
               NEEDS xymond
               CMD xymonnet --report --ping --checkresponse
--dns-timeout=5 --timelimit=75
               LOGFILE $XYMONSERVERLOGS/xymonnet.log
               INTERVAL 1m

The xymonnet alarm now says under Error output:

Unknown option –timelimit=75 – try –help
▸ quoted from Japheth Cleaver

Any suggestions?

Foster

Hi Foster,

Unfortunately, I can confirm that this is a bug in how that option is
processed. As a workaround, you can lie to xymonnet about what interval
it's been told to run (from xymonlaunch) at with:

CMD env TASKSLEEP=75 xymonnet --report --ping --checkresponse
--dns-timeout=5

This should result in what you want.

The following patch should fix it for you, and this will be in the next
version.


HTH,
-jc
list Foster Patch · Sat, 10 Sep 2016 03:32:29 +0000 · 
Hello, 

I am unsure of how to apply patches. Is this something that would changes the structure of the server? I would need to approve this with my superior if so.

-Foster
▸ quoted from Japheth Cleaver

-----Original Message-----
From: J.C. Cleaver [mailto:user-87556346d4af@xymon.invalid] 
Sent: Friday, September 9, 2016 11:30 PM
To: Foster Patch <user-830dc635f051@xymon.invalid>
Cc: Xymon Mailing List <xymon at xymon.com>
Subject: Re: [Xymon] xymonnet alarm (with patch)

Patch now included.

-jc


On Fri, September 9, 2016 8:11 pm, J.C. Cleaver wrote:
On Thu, September 8, 2016 11:51 am, Foster Patch wrote:
Hello,

I added what you recommended, but the alarm still exists. Here is the 
tasks.cfg section:
[xymonnet]
               ENVFILE /usr/lib/xymon/server/etc/xymonserver.cfg
               NEEDS xymond
               CMD xymonnet --report --ping --checkresponse
--dns-timeout=5 --timelimit=75
               LOGFILE $XYMONSERVERLOGS/xymonnet.log
               INTERVAL 1m

The xymonnet alarm now says under Error output:
Unknown option –timelimit=75 – try –help

Any suggestions?

Foster

Hi Foster,

Unfortunately, I can confirm that this is a bug in how that option is 
processed. As a workaround, you can lie to xymonnet about what 
interval it's been told to run (from xymonlaunch) at with:

CMD env TASKSLEEP=75 xymonnet --report --ping --checkresponse
--dns-timeout=5

This should result in what you want.

The following patch should fix it for you, and this will be in the 
next version.


HTH,
-jc
list Foster Patch · Sat, 10 Sep 2016 03:37:49 +0000 · 
Should I replace this:

CMD xymonnet --report --ping --checkresponse --timelimit=75 --dns-timeout=5

With this 

CMD xymonnet --report --ping --checkresponse --dns-timeout=5 --timelimit=75


Or should I add an additional line to what you put in?
▸ quoted from Foster Patch


-----Original Message-----
From: J.C. Cleaver [mailto:user-87556346d4af@xymon.invalid] 
Sent: Friday, September 9, 2016 11:30 PM
To: Foster Patch <user-830dc635f051@xymon.invalid>
Cc: Xymon Mailing List <xymon at xymon.com>
Subject: Re: [Xymon] xymonnet alarm (with patch)

Patch now included.

-jc


On Fri, September 9, 2016 8:11 pm, J.C. Cleaver wrote:
On Thu, September 8, 2016 11:51 am, Foster Patch wrote:
Hello,

I added what you recommended, but the alarm still exists. Here is the 
tasks.cfg section:
[xymonnet]
               ENVFILE /usr/lib/xymon/server/etc/xymonserver.cfg
               NEEDS xymond
               CMD xymonnet --report --ping --checkresponse
--dns-timeout=5 --timelimit=75
               LOGFILE $XYMONSERVERLOGS/xymonnet.log
               INTERVAL 1m

The xymonnet alarm now says under Error output:
Unknown option –timelimit=75 – try –help

Any suggestions?

Foster

Hi Foster,

Unfortunately, I can confirm that this is a bug in how that option is 
processed. As a workaround, you can lie to xymonnet about what 
interval it's been told to run (from xymonlaunch) at with:

CMD env TASKSLEEP=75 xymonnet --report --ping --checkresponse
--dns-timeout=5

This should result in what you want.

The following patch should fix it for you, and this will be in the 
next version.


HTH,
-jc
list Japheth Cleaver · Sat, 10 Sep 2016 07:28:29 -0700 ·
▸ quoted from Foster Patch
On 9/9/2016 8:37 PM, Foster Patch wrote:
Should I replace this:

CMD xymonnet --report --ping --checkresponse --timelimit=75 --dns-timeout=5

With this

CMD xymonnet --report --ping --checkresponse --dns-timeout=5 --timelimit=75


Or should I add an additional line to what you put in?
You should change your existing line to:
CMD env TASKSLEEP=75 xymonnet --report --ping --checkresponse --dns-timeout=5
That should immediately take effect for you. You can leave the 
"--timelimit=75" at the end, but you'll get a warning to the error log 
on each run.

-jc
▸ quoted from Foster Patch


-----Original Message-----
From: J.C. Cleaver [mailto:user-87556346d4af@xymon.invalid]
Sent: Friday, September 9, 2016 11:30 PM
To: Foster Patch <user-830dc635f051@xymon.invalid>
Cc: Xymon Mailing List <xymon at xymon.com>
Subject: Re: [Xymon] xymonnet alarm (with patch)

Patch now included.

-jc


On Fri, September 9, 2016 8:11 pm, J.C. Cleaver wrote:
On Thu, September 8, 2016 11:51 am, Foster Patch wrote:
Hello,

I added what you recommended, but the alarm still exists. Here is the
tasks.cfg section:
[xymonnet]
                ENVFILE /usr/lib/xymon/server/etc/xymonserver.cfg
                NEEDS xymond
                CMD xymonnet --report --ping --checkresponse
--dns-timeout=5 --timelimit=75
                LOGFILE $XYMONSERVERLOGS/xymonnet.log
                INTERVAL 1m

The xymonnet alarm now says under Error output:
Unknown option –timelimit=75 – try –help

Any suggestions?

Foster
Hi Foster,

Unfortunately, I can confirm that this is a bug in how that option is
processed. As a workaround, you can lie to xymonnet about what
interval it's been told to run (from xymonlaunch) at with:

CMD env TASKSLEEP=75 xymonnet --report --ping --checkresponse
--dns-timeout=5

This should result in what you want.

The following patch should fix it for you, and this will be in the
next version.


HTH,
-jc