/var/lib/xymon/rrd folders growing larger than anticipated

list Japheth Cleaver
Fri, 20 Sep 2019 17:21:52 -0700
Message-Id: <user-27b05638250a@xymon.invalid>

On Fri, September 20, 2019 5:42 am, Eliza Danyi wrote:

Morning all,

We are currently running Xymon 4.3.28-1.el7.terabithia (on CentOS 7.x) and
I have been struggling with this since we switched to the RPMs from an
older version of xymon built from source about a year ago. We've gone from
a few GB on the old server to now almost 40GB just for this directory.

I posed the question a little while back to the group and thought I had
found the cause in the rrddefinitions.cfg file under the line:

RRA:HWPREDICT:8064:0.1:0.0035:2016

I tried a few different things but ended up deciding to comment out this
line to see if it would slow the growth, but it does not appear to be
working. Is there any reason these files would be increasing in number or
size? Is there something I can do with rrdtool to clean up data? Is it
normal that an rrd directory for a single host is upwards of 60M on the
newer versions of Xymon?

Here is an example of the files in a folder for a host:

Hi,

RRD files by themselves do not change size (unless intentionally
manipulated), however new RRAs would have been added in I believe when it
was first run.

The ones you're seeing below, though, are a result of two things; one
easier to fix than the other. Systemd creates "/run/user/xxxx" partitions
on user login, and if the client is running when a new UID is logged in,
you'll get a new RRD file dropped in. This was changed in 4.3.29+ I
believe, but you should be able to exclude them by adding this in to your
analysis.cfg file:

CLASS=linux
        # Ignore some usually uninteresting tmpfs mounts.
        DISK    /dev IGNORE
        DISK    /lib/init/rw IGNORE
        DISK    /sys/fs/cgroup IGNORE
        DISK    %^/run/user/\d+$ IGNORE
        INODE   /dev IGNORE
        INODE   /lib/init/rw IGNORE
        INODE   /sys/fs/cgroup IGNORE
        INODE   %^/run/user/\d+$ IGNORE

After that's in there, you should be able to confirm that the /run/user/*
ones aren't being updated any more and you can purge them.

The second issue, however, is with your network interfaces:
ifstat.veth08022d9.rrd, etc... It looks like something is giving you
highly variable transient interface names?

Xymon does not currently have a mechanism to exclude interfaces by name
like this, since they just look like new things to track. Are these
changing during known-events that might be able to trigger a cleanup?

As a workaround, you should be able to purge ones that have not been
updated after a certain length of time.


HTH,
-jc

[root at xymon]# du -h --max-depth=1 /var/lib/xymon/rrd/server.example/*
216K    /var/lib/xymon/rrd/server.example/clock.rrd
424K    /var/lib/xymon/rrd/server.example/disk,boot.rrd
424K    /var/lib/xymon/rrd/server.example/disk,dev,shm.rrd
424K    /var/lib/xymon/rrd/server.example/disk,elasticsearch.rrd
424K    /var/lib/xymon/rrd/server.example/disk,home.rrd
424K    /var/lib/xymon/rrd/server.example/disk,root.rrd
424K    /var/lib/xymon/rrd/server.example/disk,run.rrd
424K    /var/lib/xymon/rrd/server.example/disk,run,user,0.rrd
424K    /var/lib/xymon/rrd/server.example/disk,run,user,1000.rrd
424K    /var/lib/xymon/rrd/server.example/disk,run,user,1001.rrd
424K    /var/lib/xymon/rrd/server.example/disk,run,user,1002.rrd
424K    /var/lib/xymon/rrd/server.example/disk,run,user,1003.rrd
424K    /var/lib/xymon/rrd/server.example/disk,run,user,1004.rrd
424K    /var/lib/xymon/rrd/server.example/disk,run,user,1014.rrd
424K    /var/lib/xymon/rrd/server.example/disk,run,user,1016.rrd
424K    /var/lib/xymon/rrd/server.example/disk,sys,fs,cgroup.rrd
424K    /var/lib/xymon/rrd/server.example/disk,tmpfs.rrd
424K    /var/lib/xymon/rrd/server.example/disk,tmp.rrd
424K    /var/lib/xymon/rrd/server.example/disk,var.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.docker0.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.eno16780032.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth0506ddf.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth08022d9.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth081fb4b.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth0828eba.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth0ed858f.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth164c573.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth1740a33.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth189a45e.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth197189c.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth19d7cca.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth1dfb69f.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth2090f52.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth2282b2c.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth258f7aa.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth273f639.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth28aa403.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth29630d9.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth2c50b73.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth2ccd63f.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth38ba0c9.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth38c413a.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth3a47851.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth3c5e33e.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth3cb1195.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth3e20beb.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth42dcce0.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth4311e0a.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth461d830.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth4855b73.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth5074706.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth51e84f6.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth525c545.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth569cd53.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth59a3d88.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth5c4439e.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth5f09582.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth5fa1dd2.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth67d3cbf.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth67dbb70.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth682379a.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth684bcf3.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth69ab431.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth6ad9701.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth6c80cac.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth6ce4f39.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth6e24fbf.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth6e59721.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth6fb0766.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth71f97e2.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth741d0d5.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth7c752f1.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth7f93b1f.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth8386dfb.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth8e3b4e4.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth8f68e08.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth9063983.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth91bc3c6.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth96c1fac.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth9a73f58.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.veth9ed52d1.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vetha09504c.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vetha184ef3.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vetha410e0c.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vetha460481.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vetha5a56d6.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vetha6cec13.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethad8eda4.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethad8faa1.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethaffd7df.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethb3c04b7.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethb426b52.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethb52aae4.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethb6937d3.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethba647c0.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethbd846e1.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethc01c9dd.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethc81bb06.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethc8ea13a.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethcc1a6d6.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethcc51afb.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethd01eb33.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethd105c62.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethd1b1789.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethd676e0b.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethd947e9e.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethdcb0028.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethed73659.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethf05d463.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethf4659bf.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethf84f9fc.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethf8efbee.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethfb569a5.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethfb5988e.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethfeec0dd.rrd
424K    /var/lib/xymon/rrd/server.example/ifstat.vethffc6c75.rrd
424K    /var/lib/xymon/rrd/server.example/inode,boot.rrd
424K    /var/lib/xymon/rrd/server.example/inode,dev,shm.rrd
424K    /var/lib/xymon/rrd/server.example/inode,elasticsearch.rrd
424K    /var/lib/xymon/rrd/server.example/inode,home.rrd
424K    /var/lib/xymon/rrd/server.example/inode,root.rrd
424K    /var/lib/xymon/rrd/server.example/inode,run.rrd
424K    /var/lib/xymon/rrd/server.example/inode,run,user,0.rrd
424K    /var/lib/xymon/rrd/server.example/inode,run,user,1000.rrd
424K    /var/lib/xymon/rrd/server.example/inode,run,user,1001.rrd
424K    /var/lib/xymon/rrd/server.example/inode,run,user,1002.rrd
424K    /var/lib/xymon/rrd/server.example/inode,run,user,1003.rrd
424K    /var/lib/xymon/rrd/server.example/inode,run,user,1004.rrd
424K    /var/lib/xymon/rrd/server.example/inode,run,user,1014.rrd
424K    /var/lib/xymon/rrd/server.example/inode,run,user,1016.rrd
424K    /var/lib/xymon/rrd/server.example/inode,sys,fs,cgroup.rrd
424K    /var/lib/xymon/rrd/server.example/inode,tmpfs.rrd
424K    /var/lib/xymon/rrd/server.example/inode,tmp.rrd
424K    /var/lib/xymon/rrd/server.example/inode,var.rrd
216K    /var/lib/xymon/rrd/server.example/la.rrd
216K    /var/lib/xymon/rrd/server.example/memory.actual.rrd
216K    /var/lib/xymon/rrd/server.example/memory.real.rrd
216K    /var/lib/xymon/rrd/server.example/memory.swap.rrd
3.3M    /var/lib/xymon/rrd/server.example/netstat.rrd
216K    /var/lib/xymon/rrd/server.example/procs.rrd
216K    /var/lib/xymon/rrd/server.example/tcp.conn.rrd
216K    /var/lib/xymon/rrd/server.example/users.rrd
3.5M    /var/lib/xymon/rrd/server.example/vmstat.rrd


Do these look like reasonable file sizes/number of files compared to
everyone else? How much space should we reasonably expect to use for this
given we have roughly 1k hosts being monitored (mix of Windows and Linux).

Thanks,
*Eliza Danyi*
Operating Systems Analyst
VCU Computer Center
Phone: XXX-XXX-XXXX
E-mail: user-d7b2efa5cd13@xymon.invalid

*Don't be a phishing victim -- VCU and other reputable organizations will
never use email to request that you reply with your password, social
security number or confidential personal information.  For more details,
visit **https://phishing.vcu.edu/* <https://phishing.vcu.edu/>;