Xymon Mailing List Archive search

acknowledgements does not survive xymon restart

list Norbert Kriegenburg
Fri, 2 Nov 2018 12:43:45 +0100
Message-Id: <user-ad880bfc17bf@xymon.invalid>

Hi Thomas,

thx for your suggestions, but unfortunately this did not catch the issue.
The restart runs without messages, nothing suspicious in the logs.
But after restart and after some minutes all acks are gone.
Also the ack table on bottom of the nongreen page is empty.

My alert.chk file is always up-to-date, a SIGUSR1 does not change anything.
But it is quite large (5,1MB) currently due to the lot of alerts (my access
to one of the DMZ is blocked creating a lot of conn/ssh/rdp alerts).

I wrote a script to mass-ack such alerts, otherwise the noise would be
unmanageable.
This acks all the red conns/ssh/rdp for this DMZ, and i can see the acks in
the nongreen page.
Until next restart...

Btw: In difference what Erik wrote: at least new CLASS settings in
analysis.cfg need a xymon.sh restart to take effect (just checked).

Norbert


From:	Thomas Eckert <user-2a86d6cd6326@xymon.invalid>
To:	xymon <xymon at xymon.com>
Cc:	Norbert Kriegenburg <user-501bbe9c5409@xymon.invalid>
Date:	11/02/2018 10:45 AM
Subject:	Re: [Xymon] acknowledgements does not survive xymon restart


Hi Norbert,

as I can not remember that I encountered this I just tested this (in a mini
setup, xymon 4.3.28 on Debian9) and my 2 acks survived the xymon-restart.
As your environment is fairly large this could be size/load problem.

Random ideas:
- is the restart “clean” or does something crash during the stop-phase
(logfiles)?
- you could try is to manually force writing a checkpoint-file by sending
SIGUSR1 to xymond before restart.
- if you have a redundant/multi-server setup: Is there any chance that an
other xymon-server is propagating incomplete state (`xymond_distribute` not
enabled)?

All the best
Thomas
      On 01 Nov 2018, at 13:16, Norbert Kriegenburg <
      user-501bbe9c5409@xymon.invalid> wrote:


      Don't get me wrong: i don't do frequent restarts usually, but from
      time to time i need a restart, or the whole server must be restarted
      bc. of patches.
      And as i constantly have to add new checks with new NCV definitions
      the TEST2RRD and SPLITNCV settings in xymonserver.cfg changes, this
      needs also a restart.

      Because we have such a huge numer of servers a lot of departments use
      Xymon regularly (luckily), and use the ack mechanism to organize
      their work (add ticket number to an alert, do some evaluation reports
      and so on).
      To have >100 alerts ack'ed is normal situation.
      And it creates a lot of extra work to restore this.
      In old BB times the acks always survived downtimes and restarts, but
      now there are no more ack files stored in the acks dir, so i thought
      it would be restored from the info in the alert.chk file, what is not
      the case.

      Norbert


      <graycol.gif>EDSchminke---11/01/2018 12:58:35 PM---What config
      changes are you making that requires such frequent restarts? Changes
      to hosts.cfg, alert

      From: user-15513f33c451@xymon.invalid
      To: xymon at xymon.com
      Cc: user-501bbe9c5409@xymon.invalid
      Date: 11/01/2018 12:58 PM
      Subject: Re: [Xymon] acknowledgements does not survive xymon restart


      What config changes are you making that requires such frequent
      restarts?

      Changes to hosts.cfg, alerts.cfg, analysis.cfg, client-local.cfg;
      these are
      files that get changed most frequently in my environment.  None of
      which
      require a restart to pick up the changes.  I think the only one that
      *does*
      need a restart, would be xymonserver.cfg, and the only component that
      would
      need to be restarted would be xymxm-multiack.sh -t conn -c clear -a
      rdp -d 144000 -r "FW blocked" -i de152911
ond, not the full stack.

      Erik D. Schminke | Associate Systems Programmer
      Hormel Foods Corporation | One Hormel Place | Austin, MN XXXXX
      Phone: (XXX) XXX-XXXX
      user-15513f33c451@xymon.invalid | www.hormelfoods.com